Γ Ciao, I'm Giannis Zorufakis with a quick message to friends, comrades, colleagues in the United States and beyond regarding the general election, the presidential election in America. How do I see the prospect of a very thin, marginal victory by Jorht Biden as the worst possible outcome? And let me explain that. I am deeply happy that Trump is not going to be in the White House. But on the one hand, we have a resurgence, not a diminution of Trumpism and of the old right, and of the neo-proto-fascist, racist, Republican right wing. They feel strengthened by the outcome, they got millions more votes, and also this sense that they have, which is of course completely false, but nevertheless it is shared amongst them of having been robbed of the White House by the liberal establishment. This augurs ill for all of us because of the radicalization of this movement that combines racism and a complete embracing of everything which is misanthropic, xenophobic and particularly regressive. Now let's look at our side of politics, progressive politics. The one good thing, the silver lining for us supporters of Bernie Sanders after Sanders was yet again impeded by the establishment within the Democratic Party from becoming a genuine opposition to Donald Trump from winning the denomination of the Democratic Party. The silver lining was that at least there was some collaboration between the Sanders team and the Biden team that produced a green new deal of sorts, not what I would have liked, but an interesting nevertheless economic policy to be implemented if Biden and Harris won the presidency and also the Senate. The fact that Biden is not going to control the Senate and the fact that he has not had this landslide that he was banking on means that now a president that was originally unwilling to implement a genuinely progressive green new deal now has been liberated from the obligation to do so as a result of not controlling the Senate. So in contrast to the Trumpist right which is going to be united, radicalized and also strengthened and reinforced by the fact that they are going to be in opposition, not to mention the fact that the next two years are going to see a great depression descending upon the United States which is going to energize their opposition. In contrast to that, think of our side. We will have spent the last few months arguing amongst our own progressives that always saw Biden quite correctly as representative of big business that nevertheless they should hold their nose and vote for Biden because of the green new deal. Now there will be no green new deal. We will have in other words have helped put in the White House a president who is acting simply on behalf of big business if he is acting at all given the paralysis of the administration in view of the fact that it's not controlling the Senate. Had Trump won the White House, I would have been mourning of course, but at least Democrats would be facing two or four years of radicalization, of rethinking and reorganizing. But now the next two years will be a period during which only the Trumpists are going to be organizing while we progressives are going to have to be defending a president who does not want to be in the same camp as us against big business. As for the rest of the world, I very much fear the neocons that sided with Biden and who will be returning to the Pentagon and the State Department. I hope those fears are proven false. I'm pleased that the Bolsonaro's modus lapenselvinens and the rest of the nationalists, xenophobic international, are going to lose their great support in the White House. But at the same time we have let us not forget the radical center back in control of the White House and the Pentagon and they have a very sorry record of interventions, of military adventures. I hope that they do not jeopardize the only good thing that Trump did on the international stage, which is, in fact, not to start a new war. With greetings from Greece and the sincere hope that Joe Biden is going to prove me wrong, this is Yanis Varoufakis on behalf of DN25. Κάπη DM, let's continue the struggle. We have a lot to do.


Okay, very good morning. It is Tuesday the 16th of November and quite a few things for me to get you up to speed on talking about The overnight performance of the Asia pack indices the MSCI Asia pack Index was up for the fourth consecutive day We had of course the virtual meeting between President Xi and US President Joe Biden last night So I'll get you up to speed in terms of what they discussed, but overall a generally Positive development the meeting happening for longer than was anticipated in terms of their discussions We're also going to talk about US politics the infrastructure bill passed in the US What's the next hurdles there with the bill back better bill still to be? Tackled and then we're going to talk about the Fed nomination There's some comments out of a Senate banking committee member that are worth noting also an informed source about what the odds are of a power Renomination and then we're going to talk a little bit about UK Covid the performance of booster shots Also going to look at Tesla Elon Musk dumping some more shares yesterday or becoming quite normal fare now Although Tesla shares moving into technically a bear market territory giving yesterday's move and then we're going to look at what to expect From the day ahead. We've got US retail sales We've also got some earnings reports coming out of large brick-and-mortar retailers like Walmart and Home Depot as well today So that's what's on the agenda first things first though a bit of a flavor for the Overall asset class mix and we've got a bit of a divergence here in their major currency pairs here is euro dollar which euro dollar Tells a bit of a macro divergence story from a top-level macro perspective Here's euro dollar on a daily Continuation chart to give you a bit of an idea here just squeezing this in so from what you can see here from where My video feed is going right. This is 2017 18 2019 2020 2021 So this is looking at much higher time frame chart and as you can see here the 115 area as per these ellipses was quite a key inflection point for price both as resistance in 2019 2020 and also on the acceleration of the breakdown in price here That we've had more recently and we've we've kind of run lower quite quickly here down to that next key technical level We're just below 114 at the moment So we've got euro weakness and at the same time in European equities and now starting to see further out performance This is looking at the DAX future now on the daily looking at the multiple month price reaction And after the low that we saw in October which actually was a respect of the range Which you can see here going back if I just put an ellipse around these price points here from the high that we had Back in March the low in May We came back to test that in October before this latest move higher that we've had and and here we are back up at these Record higher levels and one of the things here to be aware of is that you know what we're seeing from a very top level global Perspective is the Federal Reserve certainly getting tapering underway Talk now about the timing of when the next rate hike is going to come When is it going to be 2022 2023 so on and so forth But at the same time the Bank of England looks pretty much set that they're going to be hiking rates as well when it gets Even this year at the end of the end of the December meeting and all the meanwhile the ECB are tackling what is a quite difficult and challenging situation with COVID at the moment. I mean we looked at it yesterday, but let me just See if I can bring that up again And here we are here is the the kind of COVID Performance at the moment if I zoom this in in to give you the last three months This is what COVID looks like as new Confirmed daily cases per one million people and as you can see here Germany France all seeing quite aggressive tick up ticks, you know, if we're looking at Austria be off the charts To the upside the Netherlands also quite challenging situation at the moment as well So what we're seeing is the the ECB are a long way from any type of real policy action And there's quite a large distance lagging the likes of the Federal Reserve and the Bank of England And hence the reason why you're getting a bit of a divergent play where that's an environment where a lower rate Here is going to be more supportive for equities weaker for the dollar in comparison to the strengthening dollar Reckoning euro that is to the strengthening dollar, which we've had of late obviously having moved to a 50 plus week high last week So that's the general flavor for the euro sterling as I said our performance So euro dollars down 10 pips Cable is actually up about 30 pips and you can see here a bit of a breakout from the relative Consolidation that we're seeing from yesterday afternoon's trade and the reason for that is that a lot of eyes have been on the unemployment data that was that came out this morning and the reason for that is it kicks off a trifecta of UK data of significance for this week jobs CPI and retail sales CPI tomorrow and retail sales on Friday and the reason why the jobs number was so key is because a Lot of the rationale behind why the Bank of England did not pull the trigger on a right hike on the November 4th Meeting as much to the dismay of markets was because of the fact that they wanted to be equipped With the impact of what the end of furlough might have meant for the job situation and in actuality the ILO Unemployment rate for September has come out this morning at 4.3% and that's actually below the expected 4.4 And the average weekly earnings for September was actually stronger than expected So lower unemployment rate and a higher wages both bullish signals for sterling And it's just managed to break out of that that consolidation from yesterday in that range And also the R1 and the futures market for a quick run up towards the R the R2 here So one would say that probably now Economists already have the belief that the Bank of England are going to hike rates in December Just given the current state of play We also had comments out of Bailey yesterday who was saying he's very concerned about inflation So still very much dropping the hints towards a rate hike is pending So the jobs report is probably enough to seal the deal for a deck BoE rate hike Particularly as well that Bailey will probably want to shake off the kind of any notions of him being an unreliable Boyfriend or not delivering on his promises. He won't want that to happen again So I would say for the moment all things remaining equal Obviously we'll wait for the inflation number, but inflation is already kind of net the predetermined kind of Metrics i.e. it's heading higher It was really the jobs market that was lagging that this data now really kind of cements that idea for the market that To the rate hike is somewhat inevitable now for the Bank of England All right, let's move on there and talk about a couple of news stories And I'm gonna start off with what happened overnight So President Joe Biden and the Chinese leader Xi Jinping spoke for the need for cooperation in their first face-to-face summit Virtually of course which went on longer than expected it went on for more than three hours Even though there was no major breakthroughs not that you would be expecting any in these types of talks at the moment This is just about rekindling the as Xi would say the old friend vibes Rather than anything definitive and any type of trade agreements or anything like that so What Xi said was that he was happy obviously to see Biden that both sides face multiple challenges together while they must increase communication and cooperation Jilo had hoped that Biden would return US policy toward China back to rational and a pragmatic path But he did go on to say that he warned China would safeguard its sovereignty security and Development interests and obviously this is talking about the thorny issue of Taiwan of course And so all in all The meeting went went pretty good Nothing kind of concrete coming out of it But opening the the channels again to resume dialogue is a positive step forward from where we were perhaps just a few weeks ago When they were posturing and flexing military muscles and sailing the fleet through Contested waters and so on and so forth. That's not to say that those things will stop anytime soon But all in all a fairly positive outcome as I said the MSCI Asia Pact shares Was up for a full straight session overnight Equity index futures US are pretty flat this morning. We had a completely square finish unchanged across all major three indices yesterday in the US So going into the North American handover It's it's pretty pretty neutral to a Worst case scenario averted I would say on the flip side WTI crew is also about 50 cents and training an 81 hand all this morning Elsewhere the other thing of course that happened last night confirmation Biden has signed into law his $1 trillion infrastructure bill at a ceremony yesterday Biden now has a lot of work to do to restore then given how dragged out These these these talks have been to the passage of these bills obviously highly Contested between the Democrats and Republicans in this divided Congress at the moment So now he's going to hit the road and he's traveling to several crucial swing states to basically sell this infrastructure bill to the American public and of course the rationale there being that he's got half an eye on the midterms which will be happening kind of this time next year with both Control of both chambers of Congress is going to be up for grabs of course and we've seen some Republican gains in the likes of Virginia and so on and so forth and the falling approval rating of Biden So hopefully he can use this passage to kind of galvanize galvanize a little bit of the momentum back in his his favor Democrats are increasingly rarity of the electoral chances obviously after that as I said disappointing statewide elections So other than Virginia, New Jersey's being the other one that happened earlier this month Meanwhile back in Washington. Don't forget that you know, it's far from the job is done for Biden Lawmakers in his own party will this week continue their wrangling over the president's second and much larger proposal of that 1.75 trillion which is the bill back better bill that package opposed by Republicans has been held up by months of Disagreement of course between progressives and moderates even within the own Democratic Party To give you a bit of flavor of where those talks are at the moment a handful of moderate Democrats in the House are withholding Support for the bill into an independent cost assessment is conducted by the CBO the congressional budget office They said on Monday did they expect to publish those estimates by the end of the working week I this week And so it's hard to see any real movement on that at this point in time particularly with Biden hidden the roadshow to kind of push out the The passage of this infrastructure bill. So that's where we're at at the moment Before I move on to the other stories, if you're not already signed up to it, don't forget you can Just simply put in your email here to get hold of my daily Market newsletter I put out at the end of every European trading day. So around 5 30 This will hit your inbox completely free It's where I kind of digest or break down deconstruct one of the major topics day by day That have happened in markets and so Yeah, feel free to to shoot to shoot your email in there I'll drop the link in the video as well if you want to get that email in your inbox Otherwise moving on just having a look then elsewhere some fed comments to be aware of This is feds barking and the reason why I'm mentioning barking is he is a voter At least for now in the fmc and he said it may take several months at least for the federal reserve to understand if high Inflation and labor shortages are offshoots of a pandemic that will eventually ease or reflect more durable changes in the economy So giving a bit of a timeline there and and and I'd say a fairly measured comment Which isn't unsurprising barking fairly centralist in his policy views But gives a degree of kind of support then to the overall mantra of what the Fed has been under the power Stewardship which is fairly just towing the line and and just seeing how the economy evolves and not being too phased by market movements And it's fairly in keeping with that The other thing that came out was of course We're still awaiting whether or not power himself is going to be reappointed as the Fed chair And the latest here has come out of a senate banking chairman Sherrod brown who said he was told by the white house officials to expect quote an imminent announcement about president Joe Biden's pick for the federal reserve charlie gasparino from fox business news and Some of you might not be aware of who he is. He's a fairly well recognized journalist He's very well informed during the financial crisis So he's got a bit of a rep and his source is generally a fairly high quality And he noted that wall street odds of a power reappointment are at just 50 50 So it's still a bit of a toss up. Obviously if it's not power then it's brainard But as we've been talking about before brainard, if anything is even even more One step to the left of being dovish if you like So if she was to get in you might see a potential or some equity bid dollar weakness yield lower type move, but i'd say that um, the actual division between the two isn't really that far Different and actually Powell has seen his odds to decline over time So it wouldn't be the most shocking event ever if he didn't get reappointed My bet for what it's worth is still that powell It gets the nod In time and if that is the case then yeah, you wouldn't really expect any immediate reaction just more continuity which will be of some Relief to markets overall and would likely further cement then the directional trend of generally Equities and and so on that we've had of late All right elsewhere We've had some comments that um boris johnson. He spoke yesterday He said he left the door open to another coronavirus lockdown this winter Warning that people must get their coveted 19 vaccinations and booster doses to avoid fresh restrictions Now one of the things that we're seeing at the moment is this Which is yesterday? You would have seen that all over 40s in the uk will be offered a third vaccine after advice from government scientists And one of the things here is that new data in england show that boosters do not really top up immunity They elevate protect protection well and above the peak of what a two dosage regime would mean if you're taking the covet vaccine And actually as you can see here, then these numbers start to become you know really high Three doses cuts the risk of infection by over 93 percent according to a uk health secretary agency Not only this though that it's kind of A strategy where we're trying to squeeze both ends of the age spectrum demographically so whilst we're now Given a third booster shot To the vulnerable on the elderly and we start to work that down now to be inclusive of the all over 40s We also are bringing up the bottom end by it came out yesterday That the joint committee in vaccination immunization have said that 16 17 year olds initially offered a single dose Should now receive a second dose And at the moment, obviously this is all Supply permitting But this would obviously be quite positive science going into the winter And what i'd say is that boris johnson is saying these types of things I mean bloomberg have really made a bit of a mountain out of a molehill here Chris witty the chief medical officer has said that the nhs is very close to that kind of point of which Starts to become problematic because of the seasonal changes in the flu meaning that more pressure naturally happens During this time of year on the nhs and so any after that outbreak of more covid patients is going to be problematic And sure in theory that would make sense if we then start to see a renewed wave But equally scientists have said, you know, if you go back to remember that covid chart Although we've had a bit of an uptick after that decline since late october Death rates overall hospitalization still remain fairly level in terms of the uk And some would say that actually this peak that we're starting to see now in germany france austria and others The pattern has been that that's a lagged covid case rate pattern from what we had already seen through early october And this has generally been the kind of status quo that we've had since the beginning of the onset of the pandemic back in early 2020 And so what's happening in europe right now? Some would suggest doesn't necessarily mean that then that's going to impact england. It's almost the other way around England had this outbreak earlier and now it's hitting mainland europe kind of like when we had the kent alpha variation Obviously it said to have originated from the uk and ken It really saw an aggressive outbreak here given the higher degree of transmissibility And then it moved to mainland europe which caused some issues. So We shall see so for me. This is more political management coming out the pm And I think it's a sensible move to do but at the moment it's looking good on the booster situation All things left aside the undeveloped world It's still very much struggling to even receive first level Vaccine vaccines, but that's a discussion for a different session All right elsewhere Morgan Stanley just our point point out case, you know euro any of you are interviewing at ms at the moment Doing assessment centers things like that their head equity u.s strategist wilson Has basically come out and you know this absolutely fair play to the to this guy I think there's a picture of him here Mike wilson and he basically is quite a bear on wall street and he said earlier in the year That equities were going to sell off aggressively his 2021 target. He issued at the beginning of this year He said we were going to fall to 3,900 and we didn't get anywhere near there The market just kept going up now his target for the next 12 months is a pullback down to 4,400 And he said that while profits are projected to extend their expansion He wants that growth slow down and withdraw fed stimulus more likely pressure valuations for the rationale while we'll come off the why we will come off this peak But the point being here and what I liked about this guy is that look he said he was wrong If only Andrew Bailey could put his hands up and say, you know what I got it wrong My communication could be better and will be improved going forward and all would be forgiven. I mean, what's quite Just frustrating to watch from the bank of england governor is he's insistent that he keeps blaming traders in the markets Uh, and I think he's slightly forgetting here is that as much as that's true You can't be at the beck and call of traders and markets at the end of the day Your greatest weapon as a central bank is your forward guidance and loss of trust and credibility Is devastating to your ability to manage an economy and he's doing a pretty good job at the moment of Making sure that it's very difficult to really Take anything that he says with any type of credibility at this moment and that that Reestablishing of trust is going to take some time But you know just like wilson did you know what got it wrong? This is what I see going forward and this is why I see that view And you know credit to him, but yeah Just to keep in mind and the ms much more on the bearish side Of the forecast spectrum. The other thing was tesla Shares at the moment did break a thousand bucks, but they they managed to close just above that yesterday But basically what's happened is elon musk offloaded another 934 000 shares on monday So just under a billion dollars worth according to regulatory fighting Filings that got unveiled that adds to the six point nine billion He already dumped last week to meet his 10 threshold. Here's a bit of a visual graphic for you This is the kind of the amount that he sold in millions And the days of which he sold them. So it was november 9th last week when he really Dumped the biggest the biggest load if you'd like But one of the things here is even though tesla's share price has fallen, you know 20% or so since he's Started this process. He's still got a heck of a long way to go Potentially which is going to be super interesting to see where tesla shares really end up Musk would need to sell some 17 million Shares more than double the 7.3 million He's already offloaded in recent days if he's to commit to that poll that he did about The 10 percent amount that he was talking about selling so Yeah, watch out for tesla and obviously we talk about tesla and its volatility that it sees but obviously that does have a direct Translation because of its market cap into the major indices like the nasdaq for example, so definitely keep an eye on that going forward All right all the s&p I should say Having a look then at the calendar for today a couple things we're looking out for We've already had the jobs data as mentioned the morning fairly quiet And actually the main focal point of today is going to be on us retail sales You also get industrial production cap utilization later with the nh hb housing market index Going to start with retail sales though. It's on track to increase The month-to-month readings is better at 1.1 percent from previous 0.7 Many americans getting their Christmas shopping done early There's also lots of risks around you know supply constraints meaning of out of stock Situation the closer we get to a holiday So people generally the behavior likely to be this year that people in front load those purchases Even earlier we've got things like black friday and so on coming up as well Not far off to consider in next month's retail sales report coupled with apparent robust demand for halloween items The early holiday rush has probably given sales a bit of an autumn boost albeit fairly moderate As I mentioned we are also on the flip side going to get Delights of walmart and home depot announced their quarterly results today as far as home Walmart is concerned analysts on both adjusted earnings per share and revenue is expected to rise But basically a slower rate than in recent quarters And the reason therefore is that they were they performed very well during the initial part of the pandemic Concerts a bit of a moderation from those elevated levels Investors will also be watching for walmart's what we call the u.s. Comparable Sales growth x fuel number that metric measures the rate of growth generated by the company's existing stores and clubs in the U.S. Including e-commerce sales and analysts expect comparable sales to rise year on year with growth Accelerating from the pace set in recent quarters, but flat to the year ago quarter. Just as a reference point And then Yeah, that is pretty much it So other than that you've got the api limitries after market as usual from the speaker perspective christine lagarde There's no text expected, but just so you're aware just after 4 p.m. London time Feds bark in speaks again alongside fellow voting, but more hawkish-minded member bostic They're going to speak speaking on racism and the economy at 5 p.m. And then mary daly a voter speaks later on just before the wall street close But that is it going to leave it there let you guys get on with the day and Any questions at all feel free to drop me a comment. Otherwise, I'll see you tomorrow. Thanks very much


Thank you. Today's topic is scaling ironic and by scaling we don't mean climbing, although you could possibly climb ironic in terms of learning knowledge and so on and so forth, but I'll just ramble. Actually, before I really dive into this, I do want to thank everyone, every large operator that ever talked to the ironic community and expressed frustration or feedback and has really guided us a long way. CERN, Yahoo, even the folks in the video have provided us feedback and a lot of what we've been able to take action upon and really improve some of the user experience. So a little bit of information that we're going to talk about today is the architecture roughly from 10,000 feet. This is a lot of theory, how it is supposed to work. I'll talk about a couple different options are available. And really this is up to however you wish to deploy your environment. And I'll talk about some common pitfalls and lesser known details. After that, I'll talk about some solutions. These solutions are mostly for ironic, and I'll talk about inspector. Inspector is a special topic in this case because it is different and has a different use case. So first the architecture from 10,000 feet. Before I dive into this, I found this quote from Terry Pratchett, which I thought was very interesting. If you don't know where you come from, then you don't know where you are. And if you don't know where you are, then you don't know where you're going. And if you don't know where you're going, then you're probably wrong. And that kind of feels very appropriate in this case, because in order to administer and run a large infrastructure deployment, you really have to understand the lower level details. These are things that tend to be forgotten, or knowledge that's actually being lost in our society, because, you know, technology is becoming so advanced, but there's still the lower level infrastructure and that's kind of the reason why we're all here actually. So at the highest level. Ironic has an API service which you can any number of it uses a message bus and a database. And then you have an ironic conductor, and you have any number of conductors. And the theory at a high level is that the database for the API is read only only the conductor writes to the database. API sends commands or requests to the conductor over the message bus, and the conductor replies over the message bus. And there's two different options you want to use rabbit MQ as the message bus through also messaging, or one can use JSON RPC. Ironic is very much agnostic to the two. In order to have explicit support, the real driver behind using JSON RPC was just to eliminate rabbit MQ as a component to enable smaller deployments. Another probably an important aspect of this is ironic doesn't store any persistent data on the message bus. It's merely a pass through communications channel so JSON RPC works very well for this as well. Again, message buses for transactional information requests and responses. The database is a persistent data store. If that database has the details state information, everything that guides how the software works. So largely when we look at distribution work requests can be distributed across the API's. And you can have depending on how you set up your environment in essence, any number of API's talking to any number of conductors. There's probably some scalability limits here that Arnie's probably hit himself. That being said, the way it works is when you request something that has to go to conductor, it chooses the most appropriate conductor based upon the hash ring of the note and when I say the hash ring of the node, we distribute machines by a consistent hash ring that is a calculation of mapping the node name to the actual physical hardware. The reason for this is not the case like creating new nodes, it gets sent to whatever conductor can handle the request, but generally, this is how requests get distributed so your API may get a request. It will send it along to whatever conductors with us appropriate or the conductor responsible for managing the node. The conductors themselves have that consistent hash ring. They assign nodes to that consistent via that consistent hash ring. They are essentially responsible for all the managing of state of that node and that interaction. So that's the theory. Now, I'm going to dive in some common issues and pitfalls and lesser known details about how all this works. API requests. Specifically, in wallaby, we added a bunch of role based access control logic. We measured the performance and it was atrocious. Far worse than we had previously. And so we spent some time during the Zena cycle to actually clean up this performance, these bottlenecks and the optimization of code and how we do this request processing. And managed to increase performance quite a bit. And we backboard these patches. But before Zena. Ironic only removed unrequested data from a request. So if you can imagine, you're packing a box full of information from the database, you're sending it to the web server, the web server takes this giant box of data, starts unpacking it and throws out 90% of the data. So that was the thing we were running into quite a bit. So, at this point, now what we do is we only return the requested data and the essential fields to the API. And then return that response based upon what is requested. So if you say, give me the no detail for all the machines. Yes, it's going to be slow. Under the hood, it's doing thousands of data conversions. So we're a list of 100 machines. So it's very resource intensive to do a detailed list. If you do a column based list, or say I need these three fields, it's much faster. This was previously kind of in place but not, it really wasn't performant and actually generate more database queries when you did it. Which was kind of actually surprising when we dug into how the back and sequel interaction worked. So for example, an image provided by CERN. This is their maximum response times after the upgraded work before, during and after I believe. And you can kind of see the amount of time that was spent for API request processing on average dropped substantially. My benchmarks locally ran about 5.5.7 times faster. Aren't you posted something much, much larger, but it really depends on what you're doing. Go ahead. It's about 10 times I measured about 10 times and you can also see like in the on the left side of the picture there's even a structure, which shows the pagination, for instance, so you can see like how we get notes and chunks by, I think 1000. So there's all kinds of structure, the structure is still there but it's now like in the right hand side is like much more condensed, but it's roughly a factor of 8 to 10 I would say it's faster. Excellent. That makes me feel really good as someone spent a lot of time on this. So where this has a huge impact though is the Nova to ironic synchronization. And the way at a high level that this works is Nova maintains or Nova compute processes maintain a cache of nodes. They basically constantly repole or semi constantly repole and update that cache with new entries as they appear. However, this is a lot of overhead. Where this these patches really make a huge difference though is in getting that data because a huge amount of that time that was being sent spent by that Nova compute was just retrieval from ironic. There is another bottleneck inside Nova that I won't really talk about. But long story short, it's data manipulation processing that causes some bottlenecks, which is why one might want from many Nova computes. We can talk about that little, a little bit more about that in a minute. But the bottom line is the Nova compute process does do a heavy lift with all this data. Additionally, one thing we did find is some operators were having to we're finding their need to restart Nova computes to address issues in their environment due to race conditions and the actual list processing because it could take so long for this list to be processed. And then we found that we were doing unnecessary with attachments or virtual interfaces, and we've actually stripped that out of the interaction so now the compute process should actually start much faster I know particular operator that's on this call that actually was the inspiration for me fixing that because I wanted to you'll meet for coffee one morning or something like that. And they were having to restart and it was taking atrocious amount of time. I felt really bad about it. That operator may have played up the impact of that event slightly. Okay. So, before we really move on, I think it's important that we kind of understand the process model of ironic. Every conductor launches actions with what's called a task. It's not a green thread. Or periodic tasks. And these are things like is this task done, or can I resume this task or is the computer done, or has the state changed or rate configuration finished or bias configuration finished. All of these things are tasks. And by default, the conductor can do 100 tasks. And a huge consumer of this is power synchronization. And this is actually one of the huge bottlenecks that can exist in environments. Internally also ironic has only two reserved green threads we've not found this necessary to extend but largely this is just used for once reserved for basically the database heartbeat saying I'm working I'm alive. The other is reserved for heartbeat processing from agents, but most of the conductor time is actually spent orchestrating actions, hence the name conductor. So, I mentioned power sync and power sync is an core feature of ironic. It's an extremely useful feature of ironic. We found hyperscalers that operate at scales of 40 to 250,000 nodes or so tend to have to turn this off. And largely because it is an aggressive process that updates the power status. So it can be reflected to users in relatively shorter order. Obviously you don't want to have machine moves power and have all the guys say, yes it's alive it's working everything's fine. It's not ideal, especially if you have other processes that may automatically kick off or launch new instances. You kind of get the idea at that point. So a common setting that's used is to actually change the interval to make it longer. We did make improvements to this back in March 2021. And I believe we changed the threading model a little bit around this, but largely this is still a very process intensive operation that just runs in the background. And I think that falls every five minutes or 10 minutes. So, one of the things that actually turned found was that we were doing extra work. Surprise surprise. We like to be very thorough.  And we can see that this actually had a huge impact in the performance of ironic in terms of the amount of database queries that were occurring. Because you know, if you're just doing a list of nodes or you're just working on one task, you don't necessarily need the ports or the drivers or any other related information. So obviously we cut down number queries, the more concurrency that can occur in an environment. These patches worked out. We're back forward all the way through Stein, which is quite old at this point. But it should be a huge performance improvement. And this is probably an addition to the API performance improvements is just another reason operators really should try and run more up to date versions. It's actually very difficult resource intensive for us to get patches down to, say, Stein. So, the other issue of PowerSync is IPI. Ironic uses the operating system provided IPI tool binary to talk to each machine individually. This has a huge benefit of abstracting a number of details away from Ironic in terms of retry handling ciphers. These are but these are still things sometimes we need to set or know about pass into IPI tool to address certain cases. But the bottom line is this process is resource intensive to launch because it's native C binary. Python's execution of other binaries is also CPU intensive. So it's CPU intensive to launch it. It's CPU intensive to operate it briefly. And if you start doing the math with default settings, if you have a cluster of 250 nodes, or one conductor of 250 nodes, it launches or it will try and launch 15,000 IPI tool commands an hour, which is a enormous workload. And if you have a cluster of 1000 machines, this can easily reach a million IPI calls, which seems insane, but it is extremely resource intensive. A path forward to prevent this resource intensive use is to use Redfish. This is a native client that caches sessions that tries to be a bit or much more efficient, and it's not causing a system binary to be executed every single time which is causing libraries to be loaded and processed, and so on and so forth. So if you can use Redfish, I would highly advise it for scaling any ironic clusters, just because of the amount of performance gain that you'll have just based upon not using IPI tool. Another reason not to use IPI tool is it's incredibly insecure. And when I say insecure, I mean exceptionally insecure. For those that aren't aware. If you earn most cases, and in what the specification says, if you fail authentication, it sends you the encrypted hash of the password. So that you can try and figure out what your password was basically and it's just awful. So please don't use IPI if you can avoid it. So going back to periodic tasks, another strategy that is often used by operators is to disable a needed periodic tasks. So if you're not doing things like grade configuration or firmware updates, these have periodic tasks wrapped around them, you largely to complete state and move this conductor to the next task. These queries have been optimized to try and minimize load minimize creation of tasks, but still additional load. It's still additional database queries sorting through nodes. So if you don't absolutely need it and you need to scale, consider evaluating periodic tasks that exist. And usually you can see these in the ironic conductor configuration by looking at anything named with an interval. Not everything is an interval. So some explicit hard pauses for tasks are intervals as well. But generally everything named interval is a API task timer or used in that calculation. So, if you set those values to zero generally. That means during the task. Also, one thing to note is even if you don't use a driver in an environment. If it's enabled, it's para tasks will still run. It's part of the driver design model. And it's kind of handy, but at the same time, if you're running many machines, and you have 2% extra, 2% extra database queries in your environment that are unnecessary work, you might want to disable them. So here's some solutions to talk about. And I know I've been talking about solutions and how to kind of navigate these things, but largely those are pitfalls and aspects that are somewhat known but not commonly talked about. So one of the things when we talk about scaling ironic is just add more conductors. The hashering which I mentioned earlier will distribute nodes across the hashering. So, you can increase the number of nodes and increase number of conductors and that will scale. However, there is a general happy place where the operators tend to find an ideal conductor to machine ratio. And this is largely for operators running IPMI, or a large number of IPMI nodes, because of that computational overhead of launching processes. So I think the operators have have provided feedback of somewhere between 250 and 500. I would say 250 is no longer really the happy place I would say in another two years with improvements and processors that might be 750. But again, this depending, it all depends on the workload, your drivers, everything you're doing. I will say that some operators have achieved tens of thousands of nodes per conductor. These operators are the ones that tend to turn off power synchronization though, which, as I mentioned is the most resource intensive operation. So, as I mentioned earlier, you can scale the number of API's you can spell scale number of conductors. But at some point you start running into some logistical issues. Database is one aspect where you might run into problems. We have not tested in the ironic community but an option may exist. And the community would probably more than happy to help explore this. The also DB service or not service library does allow for what are called secondary databases, or read only databases to be loaded in terms of URL paths to additional read replicas. That allows read queries to go to one database, or any number of databases, which could be behind a load balancer, which could allow the service to scale even more. So, when adding more conductors, you do want to sequence your restarts. And this is largely because there is a hatching calculation, there are database updates that have to occur, especially if you do a full complete cold cold start where you shut everything down. That tends to cause a huge spike in database activity. Loading all the nodes, starting all the tasks. And Nova starting to sync. And you can just imagine how this kind of goes and this is a graph provided by turn, where you can kind of see the running environment.  And then they started everything up and you can see the queries per second, actually just go through the roof. I believe the, the next section until about 2pm UTC was when they were trying to figure out what was going on. Right. And I believe the next section was when they started to introduce a sequencing into it. Right. Go ahead. So basically like, I mean, as you said, it's absolutely correct. So it's basically shows like on the very left hand side before the upgrade, then everything was shut down everything was started at the same time.  I ran with that patch before so you can see on the right on the left hand side so I was like, totally confused while all of a sudden the database goes goes crazy I realized at around two. And then the last step is around like seven when I sequence out the restart so basically the secret the restarts of the conductors are staged and smeared out rather than all synchronized which would all hit the database at the same time so this is the different sequence you see so in the end, you look at the very right, you see it's like lower database activity and it's more, it's less spiky than on the very left, but yeah, we had to go through multiple stages here. How many conductors were here and how much interval did you do the sequencing we how long did you delay before starting the conductors. So for around 8000 physical notes, at the time they had around 8000 physical notes, it's 25 conductors, and the start is like a random number between the stage restart is between 30 and 90 seconds. So basically, basically I go, yeah, it's like this. I basically go one by one and then leave 30 to 90 seconds in between the start of each. So it takes like 40 minutes or so to restart it. Okay, great. Does that affect issues that with pastures either on the Nova side or the ironic side. The Nova hashrang is independent of the ironic hashrang. So it has no insight into if conductors are alive or not. It trusts the API service will get the request to the destination endpoint. Right, and then we have conductor groups which map the notes directly to a conductor. Okay. Yeah, thanks. I think the next slide is actually conductor groups. Yes, it is conductor groups. So conductor groups is a concept that we created to kind of represent physical realities. There are scale limitations to environments. There are geographic limitations where you don't need or want a conductor in say New York, managing machines in London. You don't need data center row. See talking days in a row a. So the idea is to use conductor groups where you delineate the hashrang further by stating to the two ironic. This isn't in conductor group, say a conductor group a will only have nodes assigned into that hashrang based upon the actual configuration of the notes. So if you have two conductors for conductor group a, then all of your other nodes will end up on other other conductors. So this is kind of useful for those physical realities and preventing these sorts of things but it can actually be used and also query directly from Nova compute to limit the scope and view of the Nova compute process as well. So kind of an idea where this ends up heading is you almost start splitting the infrastructure. So if you have a distinct set of Nova computes, you can have one set of ironic APS or multiple sets of ironic APS, they can run with different configuration, and they can show the same database, or same configuration and show the same database. And then you can run multiple groups of conductors depending on your needs and your load, you may have a group that does not need power synchronization as often that might just be doing long term batch processing. So if you have conductors for instances that you want to keep a state, the state information as up to date as possible so you can provide your users with input or feedback so they know oh my machine's off or oh I need to do something on my machine or so on so forth. And one of the useful things is you can run these conductors with different configurations. They'll have a configuration saying I'm in this conductor group, but you can say, here's your intervals, here's your timers, and here's your boot default boot settings or your drivers even. So you have a lot of power, and probably going back to the hashrang drivers are also delineators in the hashrang. It's just conductor groups are before drivers in the hashrang calculation. So this is a slide of CERN's layout, and you can kind of see what Arnie was talking about in terms of they have NOVA configs with conductor group configuration assigned to ironic conductors, in essence, with specific conductor configuration, so that those that pull of machines is or that those interactions are limited to that environment. So you can say, oh, I want to run all these processes on one machine, or I need to run it in this physical area or a center A, a center B, and kind of the sky's the limit in terms of what you end up with and how you want to work with this. Right, if I may add one one example, one additional example of what you just described. It's like you can, she said this on the previous slide as well, is that you can like have different configurations and this is also what we have here and what I also tried to like show a little bit in the picture so if you go to the conductor group, and which is the leading group you could have like more conductors because these is maybe the group where new nodes are added so there's more activity in the beginning because you have inspection and you have cleaning and have deployment and then it gets more calm for the for the conductor so one conductors maybe good enough. Another thing that we do is like in this leading group where we add new nodes, we have fast track enabled so in the leading, well, it's not exactly as we have another group which is group zero, if you like, where we have fast track enabled so the nodes go through inspection cleaning and so on benchmarking burn in without rebooting. And there is no Nova conductor, or sorry no Nova compute for this group at all. And we have multiple nodes in there. So basically, the team that is enrolling you have where it uses this group in order to like, it's like a waiting room for ironic, and they do all their stuff like benchmarking burning there. There's no computer that is interfering in any way there's fast track and once they're done, I move them to the corresponding conductor group, whether than deployed. So these conductor groups are really in addition to scaling of a lot of possibility to do additional configuration that suits your, your workflows. We introduced them mostly for scaling because of Nova compute because of the time it takes for the resource tracker to find all these notes this is why we chunk it up into smaller pieces. We discovered that we can do a lot more with these so it's really a very powerful concept. There are some pitfalls to this powerful concept specifically if you have a node already in Nova and change the conductor group. Specifically the Nova compute code doesn't understand this or handle this very well. There's some discussion in patches, their work in progress to try and remedy this situation. It's actually a couple different interaction issues with the design of Nova and how it tracks resources. So, hopefully, sometime in the near future we'll have this largely resolved and the patch that we can back and forth to older branches to try and make this seamless and just be a thing. So this is, this is even an issue if you don't when you don't have an instance on the note. So, with an instance I would not have dare to like reshuffle things between conductor groups, but I thought without an instance, what does Nova care, but it does care, and I screwed it up big time so there's a big warning yeah. So that's, that's one of the, one of the issues. While we're wanting to fix this issue is because Nova thinks that oh, this is a physical compute node that runs BMS not bare metal, and thinks, Oh, this is no longer in the cluster it's gone, I delete the record that record might have appeared elsewhere. It doesn't know it, or the one that issues the delete doesn't know it, and it's just is what it is right now. Hopefully it'll be fixed soon. I'm not an ironic inspector, because this is a different topic. Why would you want to scale ironic inspectors probably good question because ironic inspectors usages largely intended for when you add nodes or enroll them originally. So, if you want to add a bunch of nodes at one time, it might be useful. Another case is active node introspection. And there are some important things to keep in mind with ironic inspector. It has the ability to have a scaling model similar to ironic. However, most operators only launched as a single process. We have mixed messages, if you can actually run it with load balancers, and it works or not. And I'll talk about this a little bit more than I slide x two slides actually. The issues in inspector largely heard that it has some internal state. Which is very much not exactly load balancer friendly. Often inspector ends up being configured without a real database, which compounds this problem. The internal caching code has been improved so that you don't necessarily run into this issue. However, if you're running a much older version you probably don't have the patch to fix this issue, and it might not work. But if you're using load balancers, depending on what you're using, you may require a bug fix. In short, inspector is not really ready to scale, but it was never really tended or designed to do so. So, back to load balancers. We recently discovered a bug in event lit of all places. The long story, as short as possible, is that event lit made some bad decisions or bad decisions not the right word. They read the RFC one way, and they read the next paragraph. And in essence they thought all empty replies are treated one way. In reality, empty replies are treated a little bit differently if you have a return code of 204, which is what inspector will use in two cases. This was causing some versions of HA proxy to actually fail the request processing completely. So we found a patchy kind of handles this and masks it, but sometimes not. And this is kind of where we're getting some mixed signaling. We're not entirely sure. So we have created a patch to try and fix this. I've had feedback saying both it works and not doesn't work. So I'm kind of hoping we'll figure this one out soon. But the event lit maintainers were quite responsive when we point out the RFC and the actual what the code is supposed to do. So, I'm hopeful that a future version of that then will be fixed. So now it's time for questions and comments and everything else that makes us so valuable makes great for everyone and ourselves. Thank you, Julia for this great overview of all the work that has gone into scaling ironic. We have questions. Speak up comments user experiences. At what point did you find me at what point did you find that tuning the power sink actually became necessary for performance inner in your environments. Number of nodes or processes or what was your trigger to realize hey, by turning down the power sink I can try and does. If somebody is encountering these types of performance issues what would they be looking for in order to say hey, turning this down or turning this up or off would potentially help and improve my environment. My feeling would be task exhaustion would be some a solid sign of this issue. High database load, high CP load on the conductor for the most part. This be like looking for the your task has outlasted process by 27 seconds in the log be a trigger that somebody would use to look for these things. I don't believe we actually have for that in terms of that's more on the Nova side that does that but I'm just yeah, but there's like the priority task when it runs into itself that's at least the time. So if it hasn't finished, like I think the default as Julia said was 60 seconds I don't know that change now to 300 seconds or something. Because at some point, the task was running into itself and did not finish actually. And this was at the couple of thousand notes already, it ran into itself this is when we like increased it. And there's also like parameters that were introduced, I know two or three years ago where you could like have these power spin calls and the IPM calls in parallel. And we also play with this a little bit but this is, you know, the chorus also pointed out this with IPMI there's all kinds of debase that are not always under full control this. It's very unpredictable but I think for us what it was was at some point the power soon was running into itself and then we increased the interval. In our particular case we ended up seeing folks on the CLI end up with six out of six tries. And, you know, please retry your operation again, and it was literally because the API was swamped with power checks. It's not the API that gets swamp with the power checks. It's a conductor threads, running out of conductive threads because it's waiting. Absolutely. Yes, thank you for correcting me. So that's, that's a solid indicator of time to scale my deployment quite a bit when you start running the CLI command start returning six out of six retries failed. Talking about like, like scaling the deployment, do we have numbers actually of, for instance, number of API nodes per and number of nodes, because what the way we have deployed it is basically we have like all in one ironic controllers that always have an API a conductor and an inspector and I just scale them out horizontally. By for other components, for instance, for Nova, we basically split the API from the conductor, and we have depending on the needs more or less API nodes than conductor nodes. So do we have any like, rule of thumb, like how many API nodes you would need for like a deployment with Nova per, I don't know, end nodes or something. I think we do. And largely this is kind of an artifact of launch, the process launch model. If you do a native launch the API. I believe it will launch workers based on the number of CPUs you have. So say 90 CPUs. I think you used to actually try and launch 90 workers or something absurd like that. I think that's been toned down by default, but another common launch pattern is actually use WS GI. Say if you with Apache, you may run into a situation where your configuration is, oh, I'm only well, let me clarify. Your configuration might be such that you say, I only want to concurrent request to be processed or accessible, which is really not ideal for an API surface. And this is workers and threads count thread counts, basically in the US GI app workers are useful in case you have more the processes die however that's not really been a reported issue. The threads are really where it's kind of as useful as much as workers based on number of processors and the amount of workload that you have in your environment and that's going to depend on what you have and how you're working with it and the scale which you're running. And we have a hard and fast number because there are, there's really three or four different ways you could launch the API and run it and configure it and have it working in an environment. And there's also going to be additional overhead, say if you're terminating SSL on the on the end process versus terminating SSL load balance or, or something like Apache, because running all that trans all that insulation through additional layers does add overhead, however, say if it's patchy it's much closer to see and the CPU and more optimized interactions. So, I wish we had numbers, and I would love to talk about their experiences here. It's more like, I mean, as I said a couple of times, I'm more because I pack everything into one controller and then like add controllers as I see fit, but I never made use of the fact that I could actually split API is and conductors and the inspector actually, I just like replicate them. And then it seems to be working okay so I was just wondering here. Inspector is, is probably the weakest point in all of ironic in terms of scaling and operation but active node inspection and inspecting number of nodes for not really original use cases for inspector they're, they're bolted on or made logical sense of, oh I need to update this one machines data or I need to add these 10 machines. Right. Once mileage will vary or kilometers. Just just to add on this for those who wonder what active introspection is. So active introspection is when you run on a deployed instance, the inspector in order to update inventory data that the inspector would provide. I was done this because there was at the original inspection, when we run it there was a bug in the version of LS hardware that we used, and it reported zero Ram and that went into our inventory system and then these nodes that actually went on notice the nodes went into production. And at some point later someone noticed like hey all these nodes don't have Ram and our inventory there's something wrong so we re ran introspection why a container that we built and updated that information and this is like, when we needed to do. introspection at scale. Well, at some scale with 200 nodes but still, we may run this on a couple of thousand nodes in order to update some other data that is missing, where this came in, came in very handy that we had this feature of active introspection. Joey, sorry, go ahead. I was gonna say you deployed that as a container was the was the underlying operating system on those nodes capable of supporting that container natively or did you have to install all of that support my concern is going out and installing a bunch of stuff on users machines. And so I was lucky the the set of notes that I was needing to do this part of our batch cluster and they had all the infrastructure already there. So all I needed to do is basically run one line Docker command which holds the container executes it. And then with the parameters that were actually provided that you can pass to the active introspection, which is the the interval where you can say okay run only once. I think if you emit the interval or something it runs only once for the container exits the containers cleaned up the command exits, and you're all done okay the image is still cash there but okay apart from this everything is cleaned up to very, very neat way of doing this updating the introspection data afterwards, after the fact actually. And since we moved from, or the first half of our notes has not been inspected with all the hardware details so there's this extra hardware collector where you get actually more hardware information, like much more detailed and we only enable this after we had enrolled a couple of thousand notes, and we want to have this detailed information we will run a campaign and run this on all the other 4,000 nodes but yeah, we may have to introduce or install some additional tools we have to see our goals that. Okay. More questions or comments. I have one more comment. I think one very nice takeaway from from this work and this session is also that like operator feedback actually works. So many of the things that we have seen here is actually we're triggered by operators saying like hey, this is top can we do something about this, and then like upstream, if they find the time or if the problem is big enough, like go along and actually fix things and you saw some of the massive improvements in the graphs that Julia shared and that's just like, like an awesome collaboration or feedback loop between operations and like how the code is improved. I find this quite amazing and you saw multiple instances of this in the presentation today. It's really cool. It is the most powerful interaction open source to get to have a feedback loop as working operating and gives us the requirements and experiences that we can take back and use this context, moving forward. And I really do want to thank all the operators that have expressed frustration feedback constructively that have really helped us move this forward because it's really amazing the scale some of the operators have reached. There are obviously lots of config changes some operators had to do even some code change themselves. But getting the stuff back into the community getting this information out there is kind of vital in terms of making easier because people can run ironic with one machine. Or one machine deploy another or they can run it to deploy thousands of thousands and thousands of machines. It just really depends on how they want to run. Okay. Well, thank you everyone. Thanks again and thanks everyone for joining and see you next time round.


accredited accounts payable. So we expensed something that we didn't pay for in that sense. And therefore, we're recording an expense that needs to be backed out. So we have an expense that brought down support accounting instruction by clicking the link below giving you a free month membership to all of the content on our website broken out by category further broken out by course, each course then organized in a logical reasonable fashion, making it much more easy to find what you need than can be done on a YouTube page. We also include added resources, such as Excel practice problems, PDF files and more like QuickBooks backup files when applicable. So once again, click the link below for a free month membership to our website and all the content on it. This number, and we need to take it back out of that number. And that's why basically if accounts payable goes up, then we're actually going to increase the net income. On the other hand, if accounts payable goes down, what happened then we paid more cash, meaning we credited cash and we debited accounts payable. So we paid the cash out at a later date. And if we paid the cash out, then we need to recognize the expense when we paid the cash. And therefore, if accounts payable goes down, we are going to decrease the net income. So again, you want to mull that over in your head, but and the same is going to be true with all other liabilities, basically. So the same is true for all assets will be doing this, all liabilities will be doing that. It's good to think through the AR and the AP just for practice and then apply that same rule to the rest of the accounts. So we're just going to say negative of the 84 to 50 and enter. And there we can see that it's bringing down the balance here. And that's because it went from the 102 down to 17. So it went so it went down, meaning it's going to decrease the flows or the net income. So now we have the net cash flows from operating is this number here. And I'm going to highlight this and say we found a home for it. All right, now we have the short term notes payable short term notes payable. Where's it going to go? Is it going to go from flows from operations flows from investing flows from financing? And note that if we if we think about, you know, what happens or what types of things affect the short term note payable, a note payable, if we got a note payable, we debited cash and we credited note payable, no income statement accounts, therefore not going to be on the statement of operations is not going to be on the cash flow from operations. So is it going to be investing or is it going to be financing? And in this case, if we're talking about if we're talking about loans, we're talking about ways to finance the business, that's going to be a financing activity. So we're going to put it down here in the financing activities. Now in the description, I'm going to put cash borrow from short term note. Why? Because the note went up. And I would assume that that's because we borrowed more money. So we're going to say that the more money was borrowed in that case, again, that there's not going to be a lot of transactions in a note. Therefore, we can go back and kind of check that and see if that's the case by just looking at the at the general ledger looking at the activity, looking at the agreement that came up with those numbers. So I'm going to put a negative of that 5000 and enter. And that of course is going to increase the cash flow statement because we received cash in that case. All right, next one, we got the long term note. Once again, it went from here up to there. And it's a note that and the note is not going to be on the cash flow from operations. It's also going to be of course the financing activity down here. Now, I'm also going to put cash borrowed here and I'm going to put a negative of this number. We're assuming that it went up again that we borrowed that once again, I'm going to go back to the terms of this note. I'm going to highlight these both of these and say look at that the terms of that note and see if that is what actually happened. But for now, I'm going to say that we found a home for this number. We're going to go back and look at it at a later time. Okay, so then we have the common stock. So common stock is going to be the stocks that were basically issued for the most part. And a lot of times they're not going to change from period to period because the common stocks won't be issued. Stock will be trade often traded oftentimes if it's a publicly traded company. But the stock issuances do not happen all the time. In this case, we went from 200 to 215. Therefore, it looks like we issued more common stocks. So question being, is it going to be in the cash flow from operating cash flow from investing or finance? And if you think about the common stocks, what's going to happen, we're going to debit cash when you get cash from investors, stockholders, and we're going to credit the common stock, nothing on the income statement in that case. Therefore, it's going to be financing or investing and it's going to be part of financing. If we issue more common stock, we're generally trying to generate money financing for the company. So we are going to assume we've received cash because of course the common stock went up. Why did it go up? Because we got cash for it. We're going to put a negative of the 15 that'll flip the sign and that'll increase the cash flow, which makes sense because the cash flow would be going up from the financing that we received. Then we're going to have the paid in capital. Now the paying capital is a little bit tricky because you might be thinking, well, where does the paid in capital come from? Now remember when we issue common stock, if the stock is issued at par, meaning we have this set par rate out there, then we're not going to actually sell it for the par value. We're going to sell it for whatever we can get. So anything that we get over the par value is going to have to put into this additional paid in capital. So note that the additional paid in capital is basically part of the common stock, meaning we have this many outstanding at par, but we really sold it for this plus this or the two 45. So that's part of the issuance here. So I'm going to include that in this number. So I'm going to double click on it. I'm going to go to the end of it. And I'm going to put a negative of this number. And the formulas might get a little confusing for you there. I mean, if you think about it, obviously if we highlight those two, it's a 45,000. We want to make it a positive number. We set a negative plus the negative. And that will give us a flip the sign of both of them. You can also do it this way, negative SUM of this plus this. So I want to take that 45 and flip the sign, make it a positive number because the cash flow should be increasing. All right. So we're going to add these up now. So the cash flows from from operating notes, we found a home for all these. So this is good. We're going to add them up and we're going to see if the change then now adds up to the 61 nine, which it should if we found a home for everything and we got everything going kind of the right way here. So we already added up the cash flow from operating. That's all of this. Then we've got the cash flow from investing. So I'm going to highlight this and sum these up. And of course, we we pull this out to the side. Generally, that's how we often format our worksheets. I'm going to highlight these two. I know nothing is here yet. Probably something is going to go there when we start breaking things out. And so there we have that. And then we got cash flows from financing activity. I'm going to go down to the bottom here, going to equal the sum of these highlight these. And once again, I know I have a blank cell down here because probably there's something else that we're going to have to break out down there. And I'm going to put enter. And so now we have these are our subtotals operating investing financing, then we're going to sum up those three numbers. And that should give us the net change in this case, an increase in cash. So we're going to say this is the sum of the operating plus the investing and the financing given us the 619 that should match 619 up here. Note that it has that has to work. So if we found a home for all of these, we have it going the correct way in terms of a plus and minus format, then the difference has to be that it has to be it has to work out. So if it doesn't, then just, you know, go through here and check each one of these items until you find which one is not going the right way. And it will figure out if done properly, it has to work. Okay, so then what we're going to do is we want to end off with the cash flow at the end of the year, because we want to be able to tie out the financial statements and tie it out to the cash flow beaming. We want it the end number to be this number. So what we're going to do is we're going to take the change and then we're going to we're going to add to it what the cash was equals the cash at the beginning enter. And that means that if equals this is the change 619 plus the cash that we started with 61 550, then the ending cash would be the 123 450, which should tie out to the cash here. So that's what's going to the cash flow ending balance will always tie out to the balance sheet cash amount. All right, so now we have this entire thing, but and it works at all times out everything where you know everything is the way it should be. However, we have these accounts here that remember we're saying those could have some issues. And we want to go into each one of those individual and say, okay, is that number right? Is it do we have to adjust that number? And then we'll adjust them systematically so that we can know if we're out of bounds where we're out of bounds when we got out of bounds. So most problems will actually give us some more information. If it's a book problem, they often give a detailed information or more information like like this laid out like this in real life, obviously, we would go back and look at the actual transactions in the general ledger. So so I give us some more information here. So let's take a look at the net income. So we found that if we obviously we're going to check net income to the income statement and see if it ties out, we did that before we said the 104.5 does not tie out to the net income on the income statement, which is 158.1. So we know something's wrong. There's something's going to have to change. We're going to have to adjust this number somehow. Now, why would that happen? Why would the change in the retained earnings not just be net income? And again, if we went to the GL to the retained earnings general ledger account, we would see exactly what happened when we'd say, oh, we closed out the 104.5. But we also had probably something else called dividends in this case. So dividends is probably the other thing that could happen. We wouldn't have to guess that in real life. We wouldn't be looking at, you know, just a random set of numbers. We'd go look at the T account or the general ledger. And we would see, okay, there's something else in there. That's something being dividends. If we think about the, I'm going to think about the journal entry now, and then think about where we could adjust. I'm going to adjust this number to agree with net income in such a way that it will be in balance. So you can see I have a similar kind of setup here. We've got the number here. We're going to put a change here. We're going to end with the balance here being that plus that. And therefore, if we have one change going one way and the other change going the other way, we will remain in balance. So let's see. First, let's analyze what a depreciation entry would look like. If we, if we paid out dividends, then what would happen is cash would be paid out, cash would go down.  It's going to be in our added information. We would see the dividends. In this case, dividends declared and paid 53,600. That means that cash went down. I'm going to scroll over to our worksheet. If we were going to do the journal and treat related to that, going to put a credit of 53,600 cash went down. We're going to debit something for 53,600. And that debit is going to go to retained earnings. So that's obviously that retained earnings is what is throwing us off there. That's what's making the net income not work. So now let's think of that. Well, if that's the case, then what should we increase or decrease on our cash flow statement? And of course, we know that net income needs to increase. Notice, I'm not talking about debits and credit here, but same kind of idea. We're going to say that, well, net income is going to go up by that 53,600. And something else is going to have to happen. I know that this needs to go up by 53,600, because that makes my net income be what we want. So if I look at that, now this ties out. If I go over here and say, okay, that's the 158.1, that's what had to happen. But what's the other side going to go to? We are now out of balance if we scroll down here. We know that this number needs to be the 123,450. It is not now. So we're going to have a decrease somewhere in relation to that. And that decrease is going to be called cash paid for dividends. And where's that decrease going to go? Well, it's going to be a, it's going to be a financing activity in this case. That's going to be in this blank spot that we left down here conveniently for this number. So I'm going to put the cash paid for dividends and I'm going to put it over here into the change column. So it's over in the change column, so it doesn't throw off this number down here. So we're going to say the cash paid for dividends negative 53600. And now we're back in balance. So that's the idea. We broke this number out. Okay, now this one is fixed. I'm going to unhighlight that. We found a home. It's really this number and we fixed it in such a way that we're still in balance and we can move on to the next one and say, okay, how about this depreciation? Is it out of whack? Let's check it on the income statement and let's go through the same process. So the way we'll do that is we'll say, okay, depreciation, that's on the income statement that 15 750 should tie out to the income statement. Let's go down to the income statement here and we look at depreciation and doesn't tie out right. And we go, there's something else in there. So obviously in real life, what would we do? We would go to the general ledger account for depreciation and see what else is in there. Something else would be posted to it other than just depreciation expense. Obviously here we're going to look at the problem and see what else is there. If we take a guess at it, what else could be affecting depreciation besides the normal journal entry of debiting depreciation, crediting cumulative depreciation? And the answer would be that we would sell something. What if we sold something or disposed of something that could affect the cumulative depreciation? And so if we look at our information here, we said we sold equipment so that we have equipment sold for 26 50 51,000 cost and it had accumulated depreciation on the books at 22 850. So let's take a look at what that journal entry would look like, then think about how we can back those numbers into our cash flow statement. So I'm going to, I'm going to see rebuild this journal entry and then think about how that would affect our cash flow statement. So let's go over here into our journal entry area. And we if we purchased equipment, if we sold equipment, I should say we sold it for cash, they said cash would go up by the 26 50. This is the journal entry that would happen and equipment is going to go off the books. Equipment has a debit balance. We're going to make it go down by doing the opposite thing to it. Equipment is going to be a credit for how much we bought it for, which was 51,000. So that's going off the books. We're now selling it. And then we have to get rid of the accumulated depreciation related to the equipment. So remember the equipment was on their at cost at 51,000,