Hello everyone and welcome back to Conversations with Tyler today. I'm here with Sebastian Malaby who is the author of numerous excellent books on economics politics and current affairs He is a senior fellow with the council on foreign relations a Contributing columnist to a Washington Post and most importantly He has a new excellent and very well reviewed book out called the power law venture capital and the making of the new future Sebastian welcome Great to be with you Tyler Do the observed high returns to venture capital funds constitute a counter example to the theory of efficient markets Yes Why does it continue? Why doesn't capital just flow into the sector and bid down the returns? Well, maybe that's what we've been seeing Just in the last three years or so with enormous amounts of money coming in during the pandemic and Quite possibly there will be a correction And You're asking I guess a more subtle question about whether on a sustained basis money would flow in And bid away the returns and I think the point here is that any time you're dealing with Alternative asset managers whether it's hedge funds or venture capital It's the skill that you need and the money may be Limitless, but the skills and the connections instead of convey and what have you that is not Unlimited and so skill can continue to generate good returns So when that underlying mental model the excess returns can stay more or less forever, right more capital coming in won't bid down the returns much I Think that's right. I mean, you know, the returns have been pretty good at least for the better Venture capital funds and here, you know, we may want to come back to that caveat, but you know going back to the Beginning of the story in the 60s with Arthur Rock the pioneer of venture capital right through to today You do see good returns. I mean Sequoia, which I regard as the top venture capital partnership and Silicon Valley has generated returns of About 12 times Investors money This century so you you know, that's that's 12 X. That's 1200% it's pretty good. So why isn't that skill replicable if the returns to replicating it are so high Well, you know, I think there's a sort of Inefficient markets thinking There is a sort of notion that you can assume a way skill and that if skill is good You could just make more of it and that would mean that you would compete the returns down I think some of this skill is so far off in the tail of the distribution that it's quite hard to Replicate it. I mean a really good venture capitalist combines technical knowledge of What he or she is investing in whether that's by technology or computer science Plus business feeling plus the sort of skill of networking with people and putting teams together And a phenomenal energy, you know, because you've got to be out there Getting up in the morning for one breakfast with one potential person You might invest in and then doing 14 cups of coffee before you go to bed and being that wired you still sleep You know, I it's it is tough to find enough people like that and you know Maybe we're going to discover that the last four or five years of boom have sucked in so much talent That it will be completed away That could be true, but so far it hasn't happened What do you think of the view that in recent years? There's been a huge consumer retail tech boom Basically fang stocks, right? And when that is over it might be over now The excess returns to VC will go away if you look at venture capital for biotech Which has been hammered lately as we're speaking here late January 2022 And maybe venture capital is a limited model for one period of time and otherwise it just does okay true or false false I Say that because in a cyclical sense you might be right, but I think there's a deep structural Shift, which is really important and that is that Intangible capital has become more and more important in our economy and The nature of intangible capital is that it's hard to measure it in financial reports And so to understand whether a particular software investment for example is worth a huge amount or really nothing You kind of need to understand what that Software development within the company is doing and so you need to be hands-on You need to have the technical skills to evaluate that software project And and and that's you know, the more that intangible capital Rises as a share of new GDP creation the more this venture style hands-on Expert investing is going to be valuable Your explanation if I understand it to me seems to suggest that venture capital for biotech won't work very well So you're portraying it as something that's very very hard to do a very limited skill So you're going to be wrong a lot of times so that means the times you're right The product has to be scalable very rapidly, but in biotech there's regulators right you often need a sales force It's not scalable in the way that say LinkedIn or or Netflix are scalable So doesn't that mean VC will just stay limited to a very small area those things that are super rapidly scalable Or if you think it's pretty easy to pick winners, then you have to think the rents get exhausted Yeah, well, I mean This is a version of a actually a wider debate which goes beyond biotech Which is sort of the claim that venture capital is really only good for software projects that it you know Software can be scaled Very very fast. There are network effects once you get product market fit and You don't need much capital the you know the marginal cost of Serving one more customer is pretty much zero once you've built the Google search engine and So people will argue look, you know, that's all that venture capitalists do that is for sure Historically inaccurate. I mean there's a long history of venture investing which was more about hardware Then software, you know, it was about Apple computer It was about you net building the pipes of the internet. It was about compact Computers it was about semiconductor firms and so forth So we know the software version of this argument that venture capital only does, you know cheap to scale software that that's just wrong now biotech I agree is a tougher example, it's more regulated and Historically, you know, there have been cases early on like Genentech Which went public in 1980 with the first artificial Insulin huge huge venture return Really set kind of Perkins on the way to dominance over the next two decades in Silicon Valley in terms of venture investing But that's an outlier and it's true that healthcare investing has been tougher and less profitable and a lot of companies have withdrawn from that but I think the past isn't necessarily a guide to the future and the fun thing about venture is Is that you have to be watching to see where Basic science is generating innovation that can be commercialized and the thing that didn't necessarily do that for a long time might be the thing that does it in the future and now you've had You know, a bunch of innovations from gene sequencing to CRISPR, which makes it faster and Easier to develop new drugs. Those drugs can have huge consequences. I mean Moderna the coronavirus vaccine from Moderna is a good example of this and I think that it would be wrong to rule out The idea that even a pretty capital intensive regulation intensive sector like biotech You know, I think it might still work But what then in your mental model does limit the size of venture capital because as a percentage of entire capital flows It's pretty tiny, right much smaller than private equity What at the margin makes venture capital not work? You're right. It's small and the paradox is that the impact is big. I mean just a couple of numbers on that fewer than 1% of Companies that get formed every year receive venture capital backing But if you look at the year since 1995 Half of all the companies that go public got venture backing and three quarters of the market cap from those companies derived from venture backed companies, so Tiny share get the money less than 1% but three quarters of the market cap as a result So, I mean, that's the first point just because it's small doesn't mean it has low impact I think at the limit To answer your question directly There are things which are either so capital intensive like building a new semiconductor fab, you know, where you're really in the billions From from the get-go Where that's you know, just venture doesn't do that. It's it's about why not cap capital is not that scarce, right? Yeah, I mean ventures Advantage maybe helps to answer why sometimes it has a disadvantage So one of the advantages that is the idea of stage by stage financing So you give the company a bit of money and then if it fails early it fails cheaply because you haven't given it a huge check right from early on and Something which is going to definitely take an enormous amount of money up front Like a new silicon doctor fab semiconductor fab is it's not the right I mean just this good in terms of comparative advantage venture conceivably could raise that money and could conceivably go finance that It just wouldn't be the natural sweet spot and it would be better to leave that to you know TSMC or Samsung or some established maker of semiconductors Now as you know sequoia has changed its rules so that it can now hold investments for much longer term periods than had been the case If you extend this to its logical conclusion You could imagine a future 10 15 years from now where hedge funds VC firms a lot of other financial intermediaries Are all blended mixes of doing varying combinations of similar things 10 or 15 years from now What do you think will be the unique feature of venture capital or do you think everything will be a blend? That's a great question. I mean I would say that you know the useful definition of venture capital is that it is an early stage Venture an adventure in fact And when you get to investing in companies which are more than about 500 million dollars in market cap That's a different thing. That's growth equity. You know, I didn't call that venture anymore And the reason I picked 500 million is as a number is that when Amazon went public in the late 90s It's public market cap when it IPO it was was between 400 and 500 million Um, and now what's happened is that the IPO point has been delayed because you've got this ability to raise late stage growth capital and More and more checks are being written 100 million 200 million 300 million Into companies that are worth 1 billion 2 billion 10 billion and You know that is being grafted into traditional venture Partnerships like sequoia capital. So I write a lot about sequoia and you know, one of the amazing things is how much franchise risk They've been willing to take they were a traditional early stage investment shop in 2000 And then they grafted on this growth equity business. They grafted on a hedge fund business They built an endowment fund on top of that and now as you say, they've got permanent capital So they're going to end up with multiple business lines a bit like a golden sacks has multiple business lines But I think the venture capital portion of sequoia's business Will remain the early stage part Even after the pandemic it's striking to me how much venture capital Remains concentrated in the bay area The major deals are mostly done there. What's your mental model for that? Well, I think the sort of the lesson of silicon valley Is that agglomeration effects or pastoring effects Are actually even bigger than economics as traditionally explained In other words economics, um, and I you know just reading when I was writing my book about You know, um, so economic geography and that whole literature And the traditional story about why A cluster is a productive thing Is that you get better matching between The skills of workers and the needs of companies when you have a very deep labor market So if you're talking about you know coding And a company wants to hire a particular database engineer in a particular kind of database software In silicon valley, there will be the precise type of engineer that you want Whereas if you're in a less deep pool of labor You won't find that and the same goes the same argument goes for suppliers if you want to Have a supplier that provides a particular kind of bespoke semiconductor You're more likely to find that That that supplier locally and be able to sort of go visit them if you're in silicon valley than elsewhere That would be why the firms the startups are clustered Why are the venture capital deals clustered and furthermore in a funny way they're anti clustered like they're not all in new york Which is our major financial center by a long ways san francisco is not that right? So it's a mix of anti clustering and then some extreme clustering for this Set of deals that have some particular features Yeah, well, I mean, I think that shows you why Venture capital is fundamentally different to the other kinds of capital that are going on on the east coast and in greenwich You know new york and greenwich and so forth. I mean, it's just a totally different thing I always like to say You know the the kind of post-war Archetypical financial companies on the east coast the archetypes were prudential and fidelity And their names tell you a lot about their attitude to risk, right? They were about stewarding capital and not losing it This kind of high risk power law return Um, you know grand slam business that emerged in venture capital in the west coast is utterly different So it's not surprising that we would have an anti cluster a different center For that kind of financing, but I think it clusters because This is a business where people syndicate into each other's deals Right, so one venture capitalist will lead the series a round Financing a company and then another one will lead the series b and another one the series c Um, often a single round like the series c will have multiple investors in it And it's just economical for the entrepreneurs to be able to visit a whole bunch of different vcs And pitch to 10 of them without having to get on a plane and fly around Um, so I think there are clustering effects And also just sort of you know that the circulation of ideas People and money within the clusters, which is something that venture capitalists facilitate Uh works better when it's concentrated geographically And that's why you know at least until the last 10 years Um when perhaps we've gotten to the point with zoom Uh and remote work and so forth where this is less true than it used to be But I think the the agglomeration effects were really very very powerful and venture capital Needed to be clustered to be most effective and that's why you got the silicon valley dominance When is venture capital more effective versus when is angel investing more effective? Angel investing wasn't really a factor until the mid-late 1990s Um once it became a factor and and it that happened because Just venture capital, I mean, sorry silicon valley had reached a point of maturity Where there were enough Rich exited entrepreneurs who had made money starting their own companies and then wanted to turn around and Fund some younger people who looked a bit like them That's what happened with google google could raise, you know, a million dollars in 1998 before going to any venture capitalists Because angels have become a thing. One of them was jeff bezos of amazon. In other words, andy bestial time Who had funded started sun micro system sometime before? So Once you get angels, then I think entrepreneurs are going to want to use them because they're kind of friendly Um, they're typically people who know a lot technically and that you relate to you look up to when you're a founder And so I think angels are useful For the first round of money the first amount of mentoring And then when you get a bit more Further along in your company, you want the series a which means more money a more formal company structuring A more sort of tougher clear a sense of what your next milestone is that you have to reach. Otherwise, you'll be shut down Then you go to the proper vc True or false good CEOs make good vcs Not always true. So as a as a generalization, I would say false. I mean The premise of andreese and horror it's um, the venture partnership set up in Uh, 2009 was that to be a good venture capitalist.  Um, and andreese and horitz has now given up the rule That to become a partner at the company You have to have been a former founder of a company yourself because it just isn't the best way necessarily It's one way you can choose Um, a good venture capitalist, but it's not the only way by any means Why has mike more it's been so good at vc? Wow, that's a great question. I think he was, um, willing to do Two things, um When it came to company founders He had the ability to enlarge their sense of themselves of of that he could in two ways Of that he he could intuit What they were doing he could get on people's wavelength and not only understand them Sort of understand them better than they understood themselves and see more potential in their project than even they saw When they looked at it. So, you know, the first example of this is is yahoo Uh, where, you know, jerry yang Um, and david filo were, you know in their, um, sort of porta cabin on the stanford campus And they thought they were building a hobby type thing and they were sort of proud of yahoo as a directory of the emerging internet But, you know More it showed up listened understood and said, you know, this is the new apple You are going to make something with a quirky name apple was quirky yahoo is quirky You're going to have a brand and you're going to be the face of a new phase in tech history So he enlarged their sense of themselves And he was just great at delivering that kind of cool to greatness Speech, you know, he sat down at paypal with one of the founders of paypal Who was resisting the idea of a merger with the elon musk Rival which was called x.com And he said to him, listen, you know, if you do this merger, I will never sell stock in the company And you will build something that makes history in the valley and that Sort of call to greatness, you know, was inspiring. So that's one thing that ecue To to get the best out of people and and make them be even more ambitious than they already were The other thing Mike Morris did is he risked the franchise of sequoia You know, he did a bit what I was talking about before he was willing to Go into china Which was a whole different challenge Go through a period when he had to fire the co-founder of sequoia china because it wasn't working out But stick with it And if you ask the question What is the top venture capital company in china? The answer is sequoia china. It's the same as the answer to the question What's the top venture capital company in silicon valley? It's sequoia So Mike Morris is a bit anti clustering that point, right? It is. Yes. Yes. So China requires a whole And in a way It's a way it's a kind of confirmation You're going to say i'm twisting the the argument here, but China of course is its own ecosystem when it comes to technology because the government erects these barriers Which makes it tough for us companies to compete there And so chinese giants have dominated chinese tech But they've been funded at least in the early phase of the digital economy in china almost always by american Venture capital companies so sina so who net ease the the early sort of Web directories in china all funded by americans Then you've got by do alibaba 10 cent c-trip all of these companies backed by western vcs And so what it shows you is that in just like the same way that in silicon valley The vcs came in and built this cluster which had great circulation of ideas And people and money around the ecosystem So too they repeated the same trick in china And they built a new it's not really quite as tight of a cluster because it's split between Hong Kong and shanghai and Even hangzhou and beijing So it's not like silicon valley. It's it's multiple cities But it is a kind of china cluster unto itself But but interestingly built with the same Silicon valley dna the same lawyers in cooperation in the cayman islands for companies Youth of american style employees stock options Which you know were just not a thing in china until american silicon valley lawyers showed up in china and explained to people How stock options worked What has made peter teal such a successful venture capitalist and angel investor Peter teal is fascinating because he articulated um more fully the idea of the power law Than anybody else before in invention capital i mean the early vcs i studied Had the idea they didn't call it the power law, but they had the idea that One or two bets in a portfolio of 10 would would make all their money um But peter teal not only articulated it and took it further, but he then Actually designed his investing and his company To to even greater extent around that concept. So the logic of the power law Is that if all of your money will come from a couple of outlier bets? You better bet on outliers. It's no good betting on something that looks normal Because then everybody else will be doing a similar thing.  I think I can remember it was a ferrari or some other fantastically expensive car You know span it around, you know ruined it and was laughing his head off because he didn't have insurance and this is sort of That strategy seems entirely replicable, right? Anyone can take a lot of chances On smart eccentrics like what does peter have that is not so easily replicable? Well, you have to be You know, so then the key point is even when you decide you're going to bet on on Out of the box people who look as if they're in the tail There's still a difference between Smart bets on the tail and bad bets on the tail And to make the smart bets you do have to have a view and an understanding of Where technology is going? Um, you've got to skate to where the puck will be which means you have to think forward, you know Okay, so we've you know, just think about facebook where Peter teal did this, you know famous angel bet that that Is probably one of the best bets in us venture capital ever Um, so how did he do that? Well? He first of all was not put off by the fact that this was an eccentric 19 year old Harvard dropout who wouldn't look you in the eye Um, and that the business partner of this guy was Sean Parker who had been fired by vcs from his past two companies Um, so he had to look through that stuff but he also had to understand that with the coming of You know that with with the coming of with with the ubiquity of the internet new kinds of communication would happen And that this idea of social media which had been Tried and failed in other examples like frenster You know just because it had failed a couple of times in early iterations Social media could still be made to work if you did it right And and that's what he bet on and it was you know, it was It was still a long a long sharp bet, but it was a very high outcome bet if it went right Have you ever been tempted to try vc yourself and do you think you'd be good at it? You're smart. You know the history of venture capital quite well Um, you know, no, I think Because I enjoy what I'm doing. Um, it's true that whenever I spend five years on a project, which is how these how long these books kind of take me Um, my my objective is to get into the cockpit with the people I'm writing about and fly around the landscape and really understand How the world looks through their eyes to explain their thought process to people and so I wind up trying to think like them And my critics would say I write as if I was one of them and I'm not Critical of enough of the subjects. I'm trying to write but you know, my feeling is there's enough anger and You know mistrust in the world. I don't particularly want to add to that but I would like to add to Understanding so I I'm happy to be you know to plead guilty to trying to get into the skin of the people I'm trying to write about but it doesn't mean I want to be them. Ultimately. I'm happy Being a writer. Well, you're happy. They might be happy doing something else Uh, Mike Moritz was happy writing about soccer, right? The book about Alex Ferguson, but what's the non replicable feature that that they have and you think you don't have Um, well, look, I think it would be helpful to be Uh, more technical than I am. I mean most vcs Have an engineering degree or something or if they don't have that they know Peter teal is not very technical in that sense, right? And he's one of the very best That's true. And Michael Moritz also if it's that that model. Yes. So there are clearly, um exceptions Um, you know Peter teal had started paypal Before he became a venture capitalist and I think that was important. Um, you know, he's also Somebody who was deeply in that network. I mean he came from You know, he just so happened to come from stanford silicon valley. That was his roots Uh, and so he was embedded in the network with people And that's a very valuable thing. And you know the paypal mafia Um tells you a lot about you know, some of his returns come from The experience of having been deep in the trenches with people who were technical Um, and and backing them repeatedly Um, so, you know, ilan must was his business partner at paypal Also his business rival at paypal. They would fight each other and there was a famous coup when ilan went on You know honeymoon and he was fired from being CEO until was put in instead So it wasn't all uh, you know love and and roses, but but nonetheless Peter teal backed space x when um ilan must needed capital So that embeddedness in the network. I don't have Although I spent five years, you know, embedding myself enough to write this book Um, I don't have the engineer. I don't know. I mean, I think there are plenty of smart people who can go and be venture capitalists I like the idea that I spent five years getting to know one kind of business or financial specialty after another and I'm a sort of perpetual outside writer come tourist What are the cultural factors that limit the success of venture capital in south england? Because it has not developed a comparable network. It's fantastic for science. It has london financial center At least two incredible universities other academic contributors yet. It hasn't really taken off. Why not? So my theory about this is that You know the lacking thing has been historically venture capital because you're right. I mean To I mean oxen came bridge both have world-class computer science faculties in addition to strength in in human sciences and You know europe as a whole actually that's true as well eth and xeric is very strong And there are more traded trained computer coders in europe than there are in the united states Um, so there are you know and europe is a big rich consumer market So there are lots of strengths in europe what it's lacked traditionally people the kind of traditional lines though It's cultural people don't take risk in europe, you know, they're they're they're they're too keen on working for a big safe institution My view is that when you get venture investors Added to the mix That are willing to underwrite the risk of small startups All of a sudden people are willing to start them because the risk is paid for you can take the risk As an entrepreneur with somebody else's money And if it fails you would have burnt up your energy and time and that's not to be sniffed at But it will be somebody else's capital that underwrote it furthermore You will be helped to hire good people to help you Because venture capitalists will be on your cap table and will use its brand To bring in good people. I'm always struck by the story that Eric schmitt Told me about why he joined google as chief executive, which was a very risky move. He'd been You know schmitt had been chief executive of another company into it before And he was joining google that was controlled by these two grad students Who were notoriously, you know ornery and contrarian and might fire him and didn't particularly like people who were over the age of 30 So risk, you know the Eric schmitt knew that going into google he might get fired But he did it because the venture capitalist involved john door said to him Look if they bounce you out, I'll find you another great job somewhere else So that cultural thing about why is there this risk appetite in silicon valley? It's not something you drink in the water. It's not something in the air that you breathe It's venture capital That is de-risking Entrepreneurship and the business of joining tech startups But venture capital is mobile, right? You would think it could all spread to south england Which is not a backwater by any stretch of the imagination Very right and it's happening. That's uh, it's good. Yeah, I mean you're asking the right question And the answer is it should happen. And by the way, it is happening. I mean Sequoia has recently opened an office in london General catalyst and light speed to rather well known american vc firms Have also recently set up offices. You've got index ventures, which is big in San francisco is also big in london You've got excel ditto big in london as well as big in California This is happening the ecosystem is growing the number of unicorns in europe is growing exponentially And I predict that in the next 10 years Europe will grow a lot faster than the u.s tech ecosystem Geographically, what's the most underrated venture scene? Where would you place it? Well, it might be Europe as I just said, I mean another you say it's like south england or is it berlin or Where exactly? Um, do you think people under eight?  Because that's surprisingly hot, right? You've got spotify Based in stock home and all the spin-offs From spotify, which are bound to come in once you've got a unicorn Which is worth, you know, 50 60 billion like spotify is what that's telling you is that there's a whole carder Of people who have experienced Dramatic intoxicating growth from the inside. They are now wealthy They can start their own new companies. They can become venture capitalists Or they can just become angel advisors to to new startups I would say sweden is going to is going to set up just like seattle has Based off of microsoft and amazon I think the spotify halo effect in stock home will be pretty big So it's not just that it's properly rated.  I would say south england actually is the most underrated precisely because spotify is visible sweden's doing just fine But it seems to me properly rated Okay, so if we so going on what you say about Um sweden already being pretty highly rated then if that's right, I'm happy to agree with you Maybe maybe south england is the most underrated. I mean, you know graph core Is a great story. This is a semiconductor design company and at the beginning so there was a moment I think about 19, sorry 20 17 five years ago when sequoia decided that AI Semiconductor chips Were going to need to take a leap to a next generation product That all the ai companies that sequoia dealt with were saying the hardware that they were buying was just not up to scratch In video was good, but you know, it wasn't good enough And so sequoia told one of its partners to do a worldwide search for the best emerging Semiconductor company that was going to to knock it out of the park on ai Semiconductor design and the partner who lived in los altos california in silicon valley Began by looking at a company that you know, guess what was based in los altos california But he ended up by recommending an investment in a company in bristol south england called graph core And that company is now doing terrifically well and that experience is one reason why Sequoia decided to open a company open an office in London in 2020 To have some questions about other topics you have some highly regarded books about hedge funds and about the fed In the late 90s the bailout of long-term capital management Was that a kind of original sin that just set us on a path of bailing more things out at higher and higher price tags Should we have just let ltcm fall? No, I think the original sin was continental illinois much earlier in 1986. I believe when the fed You know bailed out at this spank Which it thought was too big to fail I'm not sure it really was too big to fail, but it was a moment when the latin american debt crisis Was still casting a shadow when the banking system was perceived to be fragile and the fed just wasn't willing to let it go That was the original sin because taxpayer money was used To bail it out. The interesting thing about long-term capital management, which people forget is that the fed convened the creditors of long-term capital management at the fed offices in new york But it refused to provide any taxpayer money whatsoever To backstop long-term capital management Long-term capital management And that was sadditary and I think if you look at what happened in the 2008 crisis actually hedge funds were not driving the crisis because the prime brokers who extend leverage to hedge funds learned the lessons from ltcm and they didn't extend loans without taking good collateral And so in actual fact hedge funds turned out to be the relatively more stable part of the system In 2008 which was a time when insurance companies and investment banks and money market funds and commercial banks And all of these other players cost the taxpayers billions and hedge funds didn't What is it you understand about alan greenspan having written a whole book about him that other intelligent educated people do not I think the key thing Was the sort of behind the title. I chose the book the man who knew The key thing was that alan greenspan had written the phd thesis mostly comprised of papers he wrote in the 1950s nobody had found that thesis until I found it And when I read it what it told me is that in the fifties in other words right at the beginning of his professional career Greenspan's obsession was with market bubbles stock market bubbles. And he was really preoccupied with the way that bubbles could drive recessions in the real economy Balanced sheet effects were at the core of his phd work Now once you understand that It tells you that greenspan Failed in his own tenure at the fair to prevent the mortgage bubble from inflating He failed not because he didn't understand about balance sheet effects He knew about Balance sheet effects. He was the man who knew He knew more about than the most of the critics who suddenly discovered in no way. Oh balance sheet effects Gee, we better go read hymen minceki. This is a big deal Greenspan knew about them and let and yet he let the bubble go wrong. What does that tell us? It tells us that the institution of the fed was sort of trapped into a position Where inflation targeting had become Irresistible and they just targeted inflation and paid no attention to asset bubbles And I think that was a big mistake. How is Jerome Powell doing? Well, I think Jerome Powell Is it's about to face a huge test as he tightens rates Starting next month history will Of course, tell us whether he manages to stable stabilize inflation without causing a major recession And that'll determine how he's going to go down my guess. Here's how I would frame it You know having acknowledged that it's a risky thing to to stick my neck out here Here's how I would frame it. I would say that Covid was this humongous shock to both supply and demand Unprecedented and the fed responded in an unprecedented way with A stimulus which as you know was was multiples of what Was delivered in 2008 and that did Get us through The the covid downturn with remarkably little Economic privations for for americans and That is an amazing achievement. Covid was tough in many ways in health terms and mental health terms But actually balance sheets of households Did very well now that is a huge achievement. You couldn't achieve that Without accepting some risk that you would overshoot I mean, how do you how do you target the right amount of stimulus when you're that in in that Completely uncharted water and you don't even understand the trajectory of the pandemic let alone the pandemic's effect on the real economy So they took a risk They may have overshot and overstimulated it And we'll see if they exit Whilst, you know without causing a big recession I think they will and I think that history will judge juror and power to have been A very brave risk taker and actually a success But I have to admit that I could be wrong And do you think the fed was excessively intimidated by the fiscal authority? So there's a joint effort going on right and the monetary authority fiscal authority. They often want different things Do you think it's the case the fed exactly understood the calculus or was shocked that inflation Came in at almost seven percent or even An alternative is they secretly welcomed it being that high Or they would have preferred it only had been at four percent But the fiscal authority had more power like what's your mental model of what went on in the government as a whole So there was overshooting, but how does the fed fit into your exact story? Yeah, so my sense Well, my my strong feeling is that the story of fed capture by the Fiscal authority or in other words the president Is wrong that Whatever since Paul Volcker The mental model of the fed everybody who works there including and especially the chairman Is that Paul Volcker was a hero. He got inflation under control Anti hero is Arthur Burns in the 1970s. He let inflation get out of control You don't want to go down in history like Arthur Burns. You want to go down in history like Paul Volcker That and that's just a sort of a threshold level.  Oh inflation is too low worry And so that encouraged some risk-taking with the amount of stimulus and now we've overshot But I still think the dominant mental model is don't be Arthur Burns people Volcker and no amount of pressure from the executive branch Would wipe that out and especially when you look at the executive branch and you ask, okay So who is the sort of chief economist in this fiscal authority that we're talking about is Janet Yellen The former fetch her herself Do I think she was trying to subjugate Fed independence? Of course I did I think it's a preposterous slander So I really think that the Fed Allowed over shooting not because they were being bullied by the president or by the treasury But simply because inflation surprised them on the outside Here's a very easy question adjusting for risk. Do hedge funds even yield? Supernormal positive returns on net So it depends on the period you look at. I mean the best way to think about super normal super normal positive returns is I think simply to say uncorrelated returns, I mean An uncorrelated return which isn't driven by just the market going up Is pretty hard to come by and a normal return. I think is zero if you're trying to do that So any positive return which isn't correlated with With the the stock market benchmark or whatever other benchmark you're choosing is good And in terms of a big diversified pension fund or endowment portfolio Will help your sharp ratio because it's giving you some extra juice to your return whilst without adding to risk because it's not a correlated position And so when you net out the volatility of your hedge fund exposure with all your other exposures It's actually volatility dampening. That's the idea So do they produce alpha is the question and when I published my book in 2010 About hedge funds the best research at the time said that net of fees there was positive alpha and it was 3% a year Between 1995 and the late 2000s. That was a pretty strong positive number I think since 2010 the positive number has declined to be much much smaller And maybe even to disappear in some time periods. I think that happened because of quantitative easing if you think about Hedge funds, you know, what is it that they do? They get paid to assess risk and price it And if risk spreads are being squeezed and compressed and reduced to almost nothing By the central bank through quantitative easing Then hedge funds are going to be paid less for doing their work Um, so I think that's why Supernormal positive returns to user language title have have have diminished I think they may now come back Um, because of the end of quantitative easing and the rising of of interest rates in the face of inflation So to ask the same question we discussed with venture capital What is the non replicable asset held by the very good hedge funds? Otherwise more capital comes in and competes that excess return away, right? Yes, again, I mean, I think the answer to your question is partly that super, you know Capital does come in and compete excess returns away as strategies are discovered. So As you look at the history of of hedge funds Um, they keep on being new strategies that get invented Or maybe new Areas of finance where all strategies can be applied And so, you know, now we're in a world of big data Which means that new data sets become available to investors you can crunch new sets of numbers using new ai algorithms and develop new insights about Patterns that appear to repeat themselves in defiance of efficient market logic And you can invest on those And the first people to do that will get supernormal returns and then after a bit that technique may become Understood by enough people that more capital comes in And the returns get to beat it away. But this is a dynamic Darwinian evolutionary industry And new methods and techniques will be invented as the old ones Seize to be profitable So let's say there are some hedge funds that are quite special But it seems a lot large numbers are not yet They're still doing some version of two and twenty fee structure. How is that sustainable? If a large class of hedge funds don't really offer the promise of beating the market Well, look, I mean You know, the famous case here is mutual funds Where actively managed mutual funds last time I looked at the data had a negative return Because the skill that's being reportedly provided Is less than the fee Even though the fee is just a one percent of assets on the management Um, so there you've got, you know, negative alpha And why do people continue to buy exposure in these Money losing actively managed mutual funds is because hope springs the turmoil And people always think that they are investing In the top quintile of clever managers Who are going to defy the what the average does Um, you know, people go to Las Vegas and Gamble Even though they know that the house has better odds than they do as the punters So I think, you know At a basic level, it's not so surprising that people continue to invest in Actively managed hedge funds because they're a better option than actively managed mutual funds And the same would be true of actively managed or eventually capital by the way So that's that's part of the answer. And the other part is that, you know, as I was saying over a long stretch of time Um, hedge funds have produced positive uncorrelated returns, which are good for a portfolio The common way that this gets reported in newspapers, which is to say, look The s and p went up 15 percent last year and hedge funds only returned four percent.  No, between us. I mean, I know this is a podcast but I'll use the expression anyway between us Ray Dalio the man in my estimation You can't answer that question Um, I feel sometimes when I listen to him that half is what he says Is obvious and the other half is nonsense Uh to the point where you know, it becomes a parlor game to look at his Ray Dalio's principles And say, you know, gee if you see an open door walk through it and was that the way you built the company? I don't think so um So I think if you want to understand why bridgewater did well for a long time although it's done less well recently It's not about him. It's it's not about the man. It's about the machine he built right the machine um just You know patiently built a research organization that uh carefully Built its own model of time markets worked by scraping data producing You know very precise clear Um studies of relationships one variable does this the other one will do that when you aggregate that knowledge based over time You build an investment black box that really did work for for quite a stretch. Um, and I think you know based on my conversations with uh People who work for Ray Dalio I come away a lot more impressed frankly Than I do by the public relations check. I've heard from him in the last five or ten years As a writer which topics and finance and investing do you think are under covered? Hmm I think actually the investing If you go at it from the perspective of you know, how does this really make money? What is the source of alpha? I think that in general is a bit under covered. I mean the starting point amongst um A lot of academics as as your questions in a way have been telling us is you know, there shouldn't be any returns alpha should be impossible to to generate to understand how it is generated is difficult It requires a lot of getting close to the subject getting close to the people who work for the subject sorting through the nonsense As I've just tried to do with my answer about bridge water Um, and I don't think that many people frankly have the time to spend Five years on one topic and try to provide an answer and so That's how I've tried to differentiate is to is to make five years of time to go look at a specialty and and I hope that is an under covered thing And therefore that I'm adding value to the to the books which are on the bookshelf Putting aside family connections your wife being editor of The Economist But what is it in media or journalism? You're especially bullish on is it sub stack? Is it old style msm? Is it whatever? Um, I'm a big admirer of The information which is a silicon valley Media startup. It's been going about 10 years Um, and the reason I'm a fan is that I think that's the right model for for media startups in other words I think sub stack where it's a lone writer Doing a newsletter Can be excellent When you've got an unbelievably talented and energetic and exceptional person You interviewed Tyler Andrew Sullivan Not so long ago and I would put him in that category of just an exceptional person and he's If you want to do sub stack, you know, fantastic but you know He's exceptional and I think most of the time Somebody can be great at a sub stack for three years possibly five years But it's pretty hard to sustain That level of focus and energy And if you don't sustain it You've built up brand over three to five years and then you throw it away because you stop Or you just go off the boil and Therefore it's more sensible to create a collective journalistic effort And that's what um, that's what the information is doing. It's going out a particular area, which is Tech it only covers tech it has good people who know what they're doing Uh, and it's got a team and it's building that team and I think that's terrific As a journalist you covered Japan for years. What do you see is the economic future of Japan? They seem to have shrinking population fairly low growth Not much appears to change. Is that just going to continue forever? And there'll be a tiny japan with hardly any people in it or what's going to change I think japan has areas of innovation that remain Um interesting, uh, it's been a while since I lived there. I was there uh in the late 1990s No, sorry the mid 1990s. I take it back. Um, uh, but my sense is that in areas like um innovation around provision for uh old age Um, you know, that's an obvious area where japan has a lot of old people and they're doing quite a lot of things to To make that work for society. So the silver economies. It's called Is something to look at in japan. I mean it remains a highly educated um Sophisticated both in terms of sort of, you know, the visual aesthetic side of engineering and the technical side of engineering Um, so although the macro in japan often looks Underwhelming because with a declining population and bad and you know, restrictive immigration policies, you're going to get A negative path in terms of of macro growth Um, nonetheless, there's some some interesting things for If you think about the role of japanese culture and possibly discouraging risk taking Is it like south england where you think venture capital can step in and get people to take the risk or just There's no way through that barrier because there don't seem to be That many new truly excellent japanese companies at a large scale And so he mostly sells insurance at this point, right? So what's wrong? What's gone wrong with risk taking in japan? Masayoshi's son, of course, is japanese Um, now you'll point out correctly that he's korean japanese And that might make all the difference. Um, he felt marginalized growing up Uh, because he was korean because he felt marginalized. He left japan in his teens And got a degree from uc berkeley And so he's a different kind of japanese person But he did go back to japan and build this huge software distributor called softbank and then he parlayed that into This sort of tech investing holding company So it shows that entrepreneurship is not entirely impossible in japan Uh, I think it could happen Can happen Um, I don't I'm not close enough to it to understand why and more of it hasn't happened Who's the greatest living british historian? Uh, gosh, um That is a great question. Um I would say, uh, the most entertaining is neil ferguson um There are historians of uh early modern europe um Who who uh peter? I think i'm going to say peter goodwin. I uh, but the fact that i'm blanking on his name shows that i'm not hugely hugely confident about the answer but so i'll stick with neil ferguson. He's he's he's always uh, a wonderful writer, uh, and provocative to read What's the the economic future of the world bank once china stops borrowing from it, which of course at some level has to make no sense Yeah, I mean china for a while. Let's say between uh 1990 2005 Generated just a vast amount of the world bank's Um success both because you know, it reduced poverty hugely And that made the global statistics and poverty reduction look good Which made the world bank look good and because it just borrowed money and paid it back on failingly and that was great for for business um You're right that with china Graduating it creates a problem and it also is a problem because China as a shareholder Is problematic? um You know part of the reason why the world bank and imf worked well for a long time relative to other multilateral institutions Is that you had this lead shareholder in the form of the united states? which uh Basically called the shots and that provided some clarity and strategic direction for both the bank and the fund I think now with china Being pretty big and influential within the world bank it can disagree with united states on Something like you know where china should be ranked in the doing business report that uh that the world bank produces uh, and this was a famous case where China seems to have put pressure on the world bank president And as a result or indirectly or something we don't exactly know uh, china got re-ranked in that uh ranking of where to do business And so that's the kind of thing that undermines the world bank credibility So china is problematic because it's graduated and because it's a tricky shareholder um, and I think that it needs to find its way and until the u.s. china economic relationship goes better at a macro level The world bank risks becoming the un security council Where the p5 The veto wielding big powers Were at loggerheads on great power politics and therefore just froze the un and couldn't agree on things So if you could pick someone now to head the world bank unconstrained, who would you pick and why? Well, that's another great question. I think you need somebody who's who would be good at this uh great power Politics, I think bob zealik when he was head of the world bank A few years back was a good choice because he was an american But he was also close to china and understood china um If you had to pick somebody now, I guess you would go down the list of um americans who have been a master in china or maybe vice versa um But uh, I think it would it would take somebody like that who has Who has the stature. I don't have a name that pops into my head, but That's a young young bob zealik is what you want young bob zealik. Yeah Why does classical liberalism seem to be dwindling today? Admittedly you may choose to challenge the premise But many countries are going in reverse democracy is not more popular. There seems to be less tolerance of other people's ideas What's going on? What's your most fundamental account of that? I think there's You know, there are lots of fans to this debate and I would just highlight two of them as to sort of set up my answer you know I think fact the fact is that that that liberal capitalism has declined in popularity enormously since the 1990s when we thought that You know it had won and that you know fukuyama's statement Was was blunted in other peoples when he said the you know history was ending But it wasn't different to most peoples and um, you know, it really did feel as though Other countries were converging around that model Now, you know, you poll young americans and they prefer or they say they prefer what they call socialism What they quite mean by that is another debate. But anyway, liberal capitalism is clearly falling out of favor now is that because Capitalism wasn't as great as we thought it was in the 1990s And now we're kind of spotting the floors Or is it because it was great, but now it's gone wrong um, and I think there's some truth to the second Um that it was great But we've allowed some things to go wrong In particular, we need more competition um to prevent both monopoly and monopsony Um, we need to do more about state capture by lobbies um, we need to do more about inequality because It's all very well saying you only care about equal opportunity Uh, but income and wealth inequality is okay But we know that the latter The income inequality bleeds into the former Because once you've got really severe inequalities of wealth People can transfer that advantage Uh to their kids And so I think you know things that happen in the 2000s like the virtual Elimination of the estate tax are a really really bad thing And we need to undo that because it's one thing to say we want strong incentives for entrepreneurship And I'm all for that because I've just written this book about venture capital and I believe in In disruption in the economy and I believe in risk-taking and I believe people should be compensated for taking those risks But at the same time you need to reset to safeguard Some equality of opportunity and I think things like the estate tax ought to bite and that's really important Last question. What's your favorite movie and why? Um Let's see. I think um the Grand Budapest Hotel has a kind of zany Sterilistic charm that is irresistible. So I'm going to pick that as my favorite movie And that is an example of venture capital also, right? Yes, that's right Sebastian Malaby. Thank you very much. Again. I'm very happy to recommend to you all Sebastian's new book all of his old books as well But the new book is the power law venture capital and the making of the new future Thank you. Tyler. That was a lot of fun


OK, thank you so much for joining from Tuesday and our talk. We are ultra excited to be here in person to talk about a non-trivial challenge of combining short-living jobs and serverless with cloud native point-touring. But before that, short question to the audience.  Do you know what fleeting means? Only a couple of hands. OK, good, thank you. I was worried that everybody will know what this means, because I didn't know when I was proposing this talk with Saswata, because I never heard this word in my life. So let's start with a short explanation. So the dictionary says that fleeting means something that passes very quickly, and generally is gone sooner than you expect. And I found this amazing cartoon art by an artist called Mikke Bach that explains the word very well, in my opinion. He actually did a word-a-day cartoon series around 1940s. And I think I have to start doing those to kind of learn English better, to be honest. So in this cartoon, we see a person called Wilbur losing something that looked like a fist duel or fight. And the woman watching this said that he stood up for a fitting moment. And I think this is pretty clear for me at least, hopefully for you as well. But how this can be applied to metrics? My colleagues at Saswata will explain that in a second in details, but we could summarize fleeting metrics as follows. It's an ability to collect metrics of numeric values over time that aggregates information from fleeting moments like Wilbur's fist duel. For example, imagine Wilbur have a lot of those fights. Perhaps you want to learn with what ratio he is winning or losing those fights. Or perhaps you want to know the median duration of those fights per day. As you can imagine, it's hard to get those information from our poor Wilbur, who potentially passed out already after a lost fight. And maybe forget what happened. And generally, this is what we will be speaking about today. So gathering metrics from those fleeting processes, let's say. Short disclaimer, we won't solve all your problems there. There is no silver bullet, essentially. But hopefully, we'll give you more context of what are the options and why it's not a trivial problem and where our potential pitfalls of this solution. Of the solutions that are available in the space. But before that, short introduction. Together with me, I have Saswata on stage, who flew over 20 hours with two stops from India, first time in Europe. Woo! Woo! Woo! Woo! Woo! Yeah. No, no, no. Try again. Hi, everyone. My name is Shashad Mukherjee. I'm a software engineering intern at Red Hat on the monitoring team. And I was a GSoc 21 student developer under Bartek's mentorship in the town's project. I love working with distributed systems and observability-related technologies. And when I'm not working, I love to read and go out for long walks. And my name is Bartek Podka. And I'm a principal software engineer at Red Hat. I maintain various projects in open source, mostly within GoLang, including Prometheus and Thanos. And I'm a CNCF tech observability tech lead. And I also write a book with Aureli called Efficient Go. And I put way too much content about observability. So it's kind of related as well. So yeah, it's a fun talk. Let's go. So for today's talk, let us begin by trying to answer some initial questions that are what exactly are short-living or fleeting jobs and why monitoring them is so crucial. And also, what are the problems that we face when we try to monitor them? So when we talk about monitoring, we usually refer to the metric signal of observability. With the continuous growth of system complexities and the data that we process every second, we need monitoring to understand the state of our workloads. And this is accomplished by tools like Prometheus designed to let us determine whether our systems are performing to expected levels of service by collecting and querying metrics and notifying humans about any problems. And the most common use case for these metrics is to construct drill-down visual dashboards and to trigger alerts when a system is behaving anomorously. So let's first understand how Prometheus normally monitors your workloads. So in any process, you can have numerous events occurring as it programmatically does things that you need it to do. You can even say that every single CPU instruction is an event. And many of these events or groups of events are of interest to people like us who are responsible for running these processes, as this can tell us if the system is behaving as we expected to. And we want to generate numeric data from these events by aggregating them over time. Since we know that we will need to alert, analyze, or debug the processes that we run. And Prometheus creates these aggregations of event data from time to time and generates a series of samples for us, which are a pair of numeric aggregation values and timestamps. And these series are what we call as metrics. So with this pull-based model of Prometheus in mind, let us try to understand how the infrastructure and the functionality that we control has changed over time and why our metrics are now fleeting. In the past, we would prepare physical servers for software to be deployed on, which would involve installing the operating system, hardware upgrades, device drivers, and so on and so forth. As there was strong coupling between hardware and software, this could be called as static infrastructure. Afterwards, virtual machines emerged. So now you could deploy your software to a kind of simulated physical server. This provided a lot more flexibility and made deployments repeatable and started a shift towards dynamic infrastructure and reduced the amount of functionality that the user needed to control. But there were still some overhead and limitations with this, and that is when several containerization tools and technologies started to be born and made it possible to run several different applications on the same system, without any of them interfering with each other. The surface area of user control further reduced with this. And then we now have serverless, which abstracts a lot of this away. Users now no longer have to care about where their code is running or even when their code is running, as this is taken care of by serverless solutions. And even if it is not serverless, applications with predefined completion states have existed for a while now, such as bad jobs and cron jobs. Now this is sort of an oversimplification of a very, very long timeline of changes, but highlights a very important trend, which is that users like you and me now have way less control over the code that we run. Now, this is especially relevant in monitoring as we do not have the full overview of the situation in such environments. We author the code, but we don't run it. So one exactly is the nature of such short-living or fleeting workloads. So when we talk about short-living jobs, we usually indicate some processes like serverless functions, batches, bad jobs, or cron jobs, or even ephemeral containers. And when we talk about such processes, they usually are running and doing the work for a limited, but arbitrary amount of time. But short-living is kind of a wrong name here or a misnomer here. We cannot define a hard limit or a boundary for when a process goes from short to long. Say if a job runs for 10 hours, is it long or is it short? So then how do we make the distinction here? Well, we can define it as a process that has a well-defined completion state. Any process which has some completion state after which we don't expect it to perform any operations and it can be terminated gracefully can be termed as a short-living process. The most interesting information for us is usually after the completion of such a process, which is what we usually want to enumerate as metrics. So now that we know what a fleeting workload really is, let us try to understand how monitoring is different based on the type of processes. For longer-living workloads, which are by far the most common workload that is monitored by Prometheus, like for example, a web server, it is relatively easy as it can be scraped and metrics can be collected at any point of time until and unless something goes wrong. As the recent aggregations of events are always there in the process itself. It can even have several fleeting processes inside of it, but as long as they are instrumented, the aggregated data can still be found within the longer-living process and can be scraped successfully by Prometheus. However, for workloads composed of only fleeting jobs, Prometheus pull-based model becomes problematic and effective monitoring cannot be achieved. A scrape might miss the fleeting job completely, in which case it simply cannot collect any metrics, and in the case it does coincide with the running fleeting job, it can only collect some of the partially aggregated data but will miss the rest of the events. Just like Wilbur, who was a fleeting process but got knocked out and now we cannot know how many duals he had or how many he won. Prometheus in a typical setup also has the inbuilt capability for monitoring the health of the discovered or configured targets. It does this by generating a dedicated time series called up with job and instance labels. And on each scrape, it appends a sample to the series with a value of one, if the instance was reachable and scraped successfully. But if the scrape fails and the instance is unreachable, Prometheus appends the time series with a value of zero instance is now down. That is not possible with fleeting processes because of the difficulties I mentioned when with regards to how pole-based monitoring can scrape fleeting processes. So to summarize, we have three major problems when it comes to monitoring fleeting jobs using Prometheus semantics. Fleeting processes can miss Prometheus scrapes completely or partially. Health and success can't be deduced from upness of the process alone. And finally, it's hard to aggregate data correctly while you're only aware about a single event, which is a point that we'll be talking about later on. And before this talk, we also discuss the same problems with open-fast and key-native communities. Hello, hello, yes. Okay, so what if you are Prometheus user or you want to be a Prometheus user but you have a lot of those fleeting jobs like serverless functions, bad jobs, or any kind of short-living processes as we define them? What do you do? Well, initially, there are two categories of solutions that are often mentioned maybe as an ultimate solutions, but the truth is they're just some options in the space. And let's go through those. And those options solve some of the challenges that's what I mentioned with fleeting jobs. So let's focus on the first one, which is simply relying on event-based observability and deriving metrics from it. As we mentioned, processes are full of events, and some of them are useful to observe and count, so then we can create alerts, monitor, dashboards, and really use that for observability. Particularly when we look on bad jobs and fleeting processes, they are mostly about single events. So it feels naturally to have observability that collect logs of events. And there are, of course, many solutions for that, like logging, tracing, and even dedicated solutions that allows us to capture information on the context about each of those events and forward to some collectors and back in some storage. With tracing, it's even better because those things are linked together in a structured way across requests. With events stored in just one place, we can process that information and provide really any metric. We often call those jobs rulers, aggregators, generators, sometimes exporters. And the ruler can then deliver this data to Prometheus, or Prometheus-compatible system, by allowing Prometheus or some kind of collector to scrape it, or maybe backfill through a TSDV block or something like that. To sum up, with this solution, we know about all the events, so it's kind of easy to aggregate and find common dimension for your metrics. And there is no regular interval of scrape happening. So process can have whatever lifespan. It might be fleeting, short, medium, long, long-living. As long as we are able to deliver reliably this single trace about each event to some back end, we should be fine. And there are many solutions in open source that you can grab, and they will do that work. And it's still growing. So for example, you could use Grafana Loki with Loki ruler. You can have Grafana Tempo. And there is, I think, just, I don't know, I seen that two weeks ago. There was Tempo metric generator. And there are ideas to have some rulers as well. You could have some very nice collectors on steroids, like open telemetry vector, which can derive some metrics from the events that it is consuming. And there's M-Tail, which is kind of an old Google project that just parses some log line and produce metrics. So there are some solutions. There are more. And of course, there are vendors who try to solve that for some amount of money. And there are downside as everything. And because all of the engineering is really about trade-offs, the first one is enormous cost that really usually results from complexity of such system. You need to gather every interesting event from all fleeting jobs, all fleeting processes. Then process and derive metrics from those. And then, so then you can alert and monitor and do the whole observability story. That's a serious traffic per second, just for observability data. And then store all this data in your backend. Now, imagine yourself as a function. That's one thing, one event. And then it meets perhaps few spans, tracing spans, about the situation, and wants to send it to some collector. That's probably a few kilobytes of data and some network call. So it feel maybe OK from the first glance. But several less are meant to scale from zero to 1,000 queries per second in millisecond-second span. And you're receiving pipeline for those events have to scale in the same way. And you have to pay for that. That's a lot of compute power in engineering time. So it means larger cost. Not only that, but you have to store and reach index and process that data in your backend to derive meaningful information from it, metrics. And you are in very much in the big data world these days. So then you need to pay. And again, on top of that, also a metric system that will use those metrics. So generally, it's more costly solution. And at the end, it's very tempting to use this for all observability as a silver bullet, but it's not. And I presented some ballpark calculation on the conference in UK two weeks ago. And just tracing backend cost for some vendor, you will pay 30 times more than just for your compute power for your application logic. And I'm not counting here even additional cost of collecting all of this and sending to this vendor. So you can imagine it can go very sideways if you really are short of maybe, I don't know, you want to save some money, right? So it's kind of crazy. And I actually use pretty cheap and nice vendor for this calculation. And there are more expensive one. And then, of course, there is a solution in tracing world which is called sampling. It's a very popular technique of filtering information about events to only send those that matters, like requests that were failing and maybe very slow or maybe interesting for some monitoring purposes. And sampling is really must have to have reasonable cost of tracing event nowadays. But then it means we cannot really derive metrics from that very easily. Because even storage or collector has only partial information of what happened, like maybe 10% of things. So you don't have a big picture and it's kind of inaccurate, right? So it kind of defies the solution. But in some way, having just the solution, like full sampling solution for just fleeting jobs, which maybe is a small part of your overall system, maybe makes sense. Maybe it's a fair solution for you. But with complexity, reliability is another problem. There is a reason why Prometheus is so popular. It's just a single binary that just works and close to the wall close. And here, there are so many stages that something can go wrong and it makes your monitoring just less reliable. And some of this you can mitigate, of course, by pushing more money and engineering time. But yeah, it's just a trade-off. And last, but not the least, something that is maybe overlooked is that we treat serverless function as a single quick job. And you really want to make another TCP connection to send observability signal and wait for it and pay for that. Like price for function depends on its execution time. Like for example, in Lambda, you pay for megabytes per second you use. So if the majority of your time spent on the quick function that does some logic is really spent on sending some observability information, that is not sustainable. But also, we are not experts in serverless jobs. So we are just looking that from a monitoring perspective. There are other problems, too. Like again, you cannot buffer and batch touch this monitoring data together to maybe send it more efficiently. So it's much more expensive and slower than if you would have maybe a longer living job. And what do you do with, you know, if tracing collector is down? How do you buffer? Those things doesn't have state. So, yeah. So let us take a look at another category of solutions that exist in the ecosystem and allow us to push partially aggregated metrics instead of just events and how this might be useful. So let us start again at the beginning where we have a fleeting process. We have multiple fleeting processes that have some events internally that we are interested in. However, as normal Prometheus script does not work here, we need to collect that data somehow. Something that we can do is aggregate some of the recent data about these interesting events within the process itself. This partial aggregation cannot give us the full context that we need, but can only give us the very small context of events within a fleeting process. As Prometheus is not a push-based monitoring system, we require some additional gateway or collector component which can receive these partial metrics and might be able to aggregate it even further before exposing it for Prometheus to scrape and generate samples from. So in theory with such a push-based approach, all of your processes can push partial metrics which are essentially aggregations of their recent data collected from the events within the process itself. With a push-based approach, your fleeting processes are no longer dependent on the time of a scrape. Fleeting jobs can simply flush their metrics to the gateway as soon as they are completed and allow the gateway to aggregate it further. Now, distributed aggregations with this approach are possible, but let's try to understand what it is and why we need it. So in this kind of scenario, each of our fleeting jobs can aggregate their internal events in their own process. However, as these aggregations can only give us the very small context of a single fleeting workload, we cannot use it as this is not an overall view. As you can see here with the full counter metric which is sent by each fleeting process after it has been incremented to values like one, two, or three, but for us to get an overall view, the metric can be aggregated at the gateway or collector level to the value of six. And this is what we can term as a distributed aggregation. However, different types of aggregations are needed for different types of metrics within Prometheus. For counters, metrics where all the labels match can just be added up. So as you can see here, once a counter is pushed twice, we can expose it with a value of two. For histograms or even summaries, the buckets can be added up and if the bucket boundaries are mismatched, the result can be the union of all buckets. So as you can see here with when two HTTP request durations of five seconds or 3.14 seconds are pushed, they get added to 6.28. And for gorges, metrics can also be added similarly to counters. But we might want to provide different semantics for aggregations. For example, this CPU temperature gauge would benefit from aggregating the last push temperature instead of just simply adding up all of them. So several open source solutions can help us in building such a partial metric push-based architecture for our fleeting processes. All of them with varying semantics, implementations and configurations and each with its own trade-offs. Also, functionality like distributed aggregations are complex to implement. So for example, Prometheus push gateway supports push and cache functionality but not distributed aggregations. Other solutions like Weaveworks from aggregation gateway does, but aren't really customizable. And there is even a new rust-based aggregating gateway that does this via labels. And it's built by an awesome engineer at Cloudfriar called Colin Douche who we actually spoke to before the stop. And he is giving a coupon talk about the same. And you can rely on ecosystems of open telemetry and stats D as well for partial metrics, push-based architectures too. Thus, you need to carefully consider what fits your workloads. Now there are also problems around metric staleness semantics. Metrics staleness is not handled well by such push-based solutions. Solutions like the push gateway never forget the time series that is pushed to it and will expose them forever to Prometheus until you manually delete it via the push gateways API. Which might make it frustrating to use. And this is also the case with aggregation gateway but it does not even have a metric deletion API. This means that to forget old or stale series, you would simply have to restart your service. And some solutions like open telemetry allow instrumentation to tell when to delete the metrics via specialized OTLP protocols. But it is not done automatically and if you would pull with Prometheus, it can cause issues. So we have to also in a way share cardinality between our fleeting processes. As each of them pushes some partial data, often the same exact metric with the same labels. But one process cannot see how the other processors are aggregating their events or how they are pushing. As a result, it becomes really easy to explode with cardinality with inconsistent labels and dimensions. And this can also lead to some pretty serious accuracy issues which can be really hard to fix. So with push, reliability also takes a hit. Push requests may fail from your fleeting processes. And the gateway not being scalable in the majority of cases might become a bottleneck by dropping push requests or simply not being able to process them. And getting ummed as more metrics are pushed to them. Thus, with such solutions, there are way more things that can break and need to be fixed. So as Bartek mentioned earlier, push pushing in any form might mean an enormous latency hit as you have to initialize some TCP or UDP connection. While partially aggregated metrics are easier to push instead of actual event data, this might already make it too slow for your environment. And we also need to worry about scalability. For many of the existing open source solutions, the gateway or collector components are not horizontally scalable, like the Provence has pushed gateway or the aggregation gateway. Multiple instances often cannot be run together simultaneously, so these components will likely become single points of failures in your monitoring architecture as you start to push more and more metrics from your applications. Hello, hello. Can you hear me? As we learned, the two mentioned solutions, pushing events and pushing metrics, they have their smaller or bigger problems, and the main theme is complexity, and it might be fine for your requirements, by the way, but maybe there is something simple we could do. I think once you solve, by the way, the challenges with this push of events or metric, at least for me, it's easier to accept some kind of limitation of pool-based metrics. And following that, in our opinion, there are two solutions, and I think we are over time, so we have three minutes, so I will just explain one solution. And so, first of all, rich, closed box monitoring. So, generally, the open and closed box monitoring is something that you might know from previous white box and black box monitoring, same thing, but different name, because we are all moving to more inclusive language, and premise is very simple. It's open box monitoring if the source of the formation is directly from the source of the event, for example, from Wilbur directly. He might be the best person to tell about some details and what he planned to do in his mind and what went wrong. Closed box monitoring is when you deliver similar or good enough information from someone or something who participated in this event, maybe computation, maybe network call, but they are not source of it. And they might just capture information on the way. So, for example, the same questions around winning grade and duration, we might ask Wilbur's partner, perhaps wife, that we're watching all those fights. She might not know exactly what was on Wilbur's mind, but she saw the whole situation, right? And this is really some lesson from our requirements, gathering discussions with open fast community. It's funny, kind of, we met with the maintainers and including Alex who is here, and we ask, hey, with the Prometheus community, what do you need? What do you need to make open fast better? And they were like, what do you mean? We are happy. I was like, okay, why are you happy? And the reason why they are happy is they build a monitoring, and they get this monitoring information cheaply from surrounding of those fleeting processes, which also means there is no instrumentation, manual instrumentation needed, it's auto-instrumented. If you think about this more, there are many long-living processes around your fleets. For example, orchestration system like Kubernetes, you have CubeState metrics, so if you have BadJob, you don't need to instrument BadJob, you can tell it's kind of successful rate or durations by just CubeState metrics, right? The network proxies or gateways are part of your request, so you can, they can provide metrics about from network traffic perspective, so you can totally build red method monitoring like rates, errors, duration, all of it just from service measures. Finally, our OS knows about kernel, knows about everything. Every CPU instruction, IO, that is happening with your serverless function, so also memory usage and performance and so on, so everything else you can kind of derive from existing tools like C-groups, eBPF. The thing that you might miss is a open box monitoring, so things that are really within the context and logic and have the function itself, but because function is super, super small, the context is usually very small, so the question is do you really need that, right? And second, I don't think I have time for that, we can talk about that later, but essentially it's like recommendation for serverless architecture as a user or serverless, you cannot do more, but we have to skip that unfortunately, but all of this, the recommendation I didn't mention and also the close boss monitoring, what you can do if you are just a user of like Lambda or Amazon Lambda or Cloud Run or any of those existing solutions, right? In some way, the reality is that clouds, maybe cloud vendors want to lock you in in some way, so users are left without kind of possibility to use normal Prometheus metrics, normal code registries and normal instrumentation and those close box monitoring things, so it's really about also incentivize those vendors to integrate more with opens up and CNCF kind of monitoring, so I guess it's some time for some feature requests here and there if you are users of those platforms. So really, yeah, that's the sum of it. The big learning here is that there is no silver bullet here, unfortunately, both even by solution, pushing metrics, close box monitoring, has their pros and cons, and hopefully we'll learn today about some trade those and implications, and hopefully all of this will give this important debugging, monitoring and observability data to Wilbur who can improve fees, duels, or ideally better negotiation skills to solve disputes in a more friendly way. And yeah, that's it, thank you very much. We have a cut over our four minutes, so maybe we can try one question while the others are getting set up. Are you already cabled, Alex? Okay, then we're going to use this time. Yeah, would it make sense to run something like the push gateway as a demon set and then run your, then push to basically local host? You can answer that if you want. So, yeah, so as far as I understand it, push gateway is not recommended for those kind of solutions. It does not have any such distributed aggregation capability, so it's only recommended for service level, ephemeral bar jobs, and running it on like a demon set or even multiple replicas of it isn't recommended. I think there was some mailing list about it too where Brian commented, but no, it isn't safe to run it like that, so yeah. Any other questions? Maybe you can give us two minutes on the worker approach then. Okay, so worker approach, oh my god, yeah, I should pick this, actually do this, I didn't know there was enough time, but anyway, it's about making sure that there is, so usually you serverless platforms, really architecture the serverless processes, those fleeting processes as isolated process. But if you move a little bit step back and really schedule those in multi-thread environment rather, so have like as one bigger living, long living process that schedules those functions and maybe only one type of the function and then only you scale those long-living processes that orchestrates those functions because those functions should be super small, so there is overhead to even start the process. So it would be good because finally you have some long-living worker that will share some state that then you can batch different things, not only monitoring, but maybe other state as well for even for your logic, right? And yeah, but the state of the world right now is that the native use Kubernetes, so they would need to move a little bit away from that, so it's just a recommendation from your someone that looks at this architecture from monitoring perspective, so yeah. Thank you. Thank you, thank you. Any more questions? Just pop it in Slack, three promises there, and I'm certain you will answer them. Thank you. Yep.


We're going back to July 11 2015 Cardinals at Pittsburgh. You got AJ Burnett on the bump. And he's going to overthrow this fastball little in Surreal. He says, Hey, calm down, buddy. Hey, calm down, buddy. And this guy tries to get cute with his girl. And she says, I don't care. Why'd you bring me here? This is boring. Next pitch comes the fastball down right down the middle. Yes, that is the ball. Next pitch goes with the hook off balance swing. They love it. Let's go back to that well. Bam, get the K end of the second inning. Oh, no, wait. Umps says that he fouled it off. Burnett's like, Okay, back track, back track. That umps not going to get any help. Let's go for the slow mo replay. Extreme zoom in right now and slow it down even more and looks like there's space between the bat and the ball to me. The spin doesn't change. It's not exactly 100% conclusive, but I'd say 98% he swung and missed and struck out good on the umps. They say, All right, let's get together. Look at the ump on the left.  They don't even look at him hands in the pocket. He's like, Okay, cool. Thanks for at least involving me in the huddle is foul ball. Surveillance him hurdle comes out like what are you talking about? Now I couldn't read their lips exactly. But I got a lot of keywords and there's a lot of sound says, Well, did it hit the bat? Is no, but I heard the sound. That's how you make the call. And he's like, Yeah, but how do you hear on the sound? Did it actually hit the bat? That's what I got out of the context. Who's he's like, you can't just go off the sound. You got to tell me you hit the bat, but they do just go off the sound. Anyway, next pitch, they go back to the curve ball. Well, but this time, Reynolds sits on it, bangs it is a hanger and surveillance instantly turns yells at the ump yells at him again. We got a kid almost makes a great play with his glove. But then the dad coming back with pizza scoops it up and says, Hey, son, I got you pizza. And I got you this home run ball father of the year. Boom. Surveillance still yelling right now you can tell AJ Burnett reacts because they just threw Surveillance out but Surveillance doesn't understand yet he's in his own head. Now he's like realizing and Burnett's trying to get some distance in between him. Surveillance walks up to Burnett and says, did he just ring me and says, yep.  He's gonna yell and scream. Can't read Surveillance lips. I mean, I don't think people that are right in front of Surveillance when he's yelling like this, you can understand him yet alone read his lips and try to figure it out. But he's heated and yelling. It's only the second inning and now your catcher's thrown out of the game. That's not good. It was a strikeout. Now you don't have to lead. Surveillance heated and I get it but also it's Surveillance. That's what he does. I love the dude got to show off the haircut.  If you're out of the game, at least show the crowd what you got going on underneath yelling and screaming. This game ends up going 14 innings and Chris Stewart has to come in. Now they can't change out the catcher but Chris Stewart in the bottom of the 10th inning with an O2 count ties the game. How about that? And then the Pirates do go on to win this one in 14 innings. Cutch with the walk off and a great trot to home plate.