Good evening, good afternoon and good morning to everyone who wherever you are from around the world. So we have got the Sudhir Joshi, our next speaker. He's the head of customer success at Lambda test in New Delhi. He's going to talk today, the continuous testing at scale. Hello everyone, my name is Sudhir and I'm heading customer success at Lambda test. First of all, I'm honored to be here talking at Selenium conference and thank you to the audience for your interest. Today I'm going to touch base on some of the key challenges when organization are adopting to continuous testing, especially at scale. So, let's see, okay, so what, where does continuous testing fits in? Very simple, we all know right, we want to test more at each stage of SDLC and the idea is as business is shipping more code, they want to really ship it faster. That puts immense pressure on the testing community and, you know, I, in fact, interestingly, I was speaking to, I've spoken to about 100 plus QA leaders from Fortune 500. Interestingly, they both have two KPIs, they all have two KPIs in common, which is one is increase the automation test coverage, and the other is, you know, we reduce the test execution time. And with the continuous testing, automation plays a key role because the team is able to run it more frequently and at each stage of SDLC. So NetNet, this team is testing, you know, doing, doing a lot of testing. Now, but it doesn't end here, right, quality is everyone's responsibility and we have heard great things about shift platform. And if developer contributes in, in, in shorting the unit testing and integration is done seamlessly, it will make the whole e2 experience much faster and flawless. And why do we want to do this, right? This is all known, no science here, right?  So with continuous testing, every action has a reaction, right, and one of the biggest challenge is once you adapt to automation, you end up writing too many test cases. And now how they are relevant, they are relevant in a way that if you keep adding with each commit, you're adding the test cases at some point of time, it'll clog your pipeline. The other challenge is flaky test cases. Now, with so many test cases, since there's no quality control, the test cases are getting added, but eventually you don't know if this test case adds any value or not.  The last is, which is perhaps the most common one is who broke the bid. Now imagine, since you have too many test cases, you can't run the suit or your regression with each one of, with each commit. So what you do is you bundle it up with, say, 10 commits or 50 commits, or run it every four hours.  The problem doesn't end here, right. Flaky tests are everywhere. Even in the unit testing, flaky test cases are equally problematic, right. Google claims to have 1.5% test cases as flaky. An interesting number is about 80% of software companies are claiming that they have seen the same error in the test environment before, but were unable to reproduce and they have seen the same issues in the production. So we know flaky test cases are expensive, injuring team invest days, in fact, weeks and worse, there's no guarantee of the outcome. So it's very critical for us to test our own test cases, right. The quality of test cases has to be there. So how organization are combating the flakiness. So first thing first is they do manual rerun, right. So if a test case failed to handle any false positive, they will run it multiple times. With few advancement, you now have framework driven auto retry. A frame will take care of retrying a flaky test case or a flaky test case. With more advancement, organization have also built a programmatic approach where they build a core relationship between the test case results today. And what it was the same, what was the result of the same test case in the last execution. And if there is any inconsistency, it remarked as a flaky test case. And the most advanced, which is what I'm going to talk about more in detail is proactively testing for flakiness and what are those techniques. So those techniques, which some of the large enterprises are handling is or managing is running the job in number of times. This ensures that my test case is not flaky if I run it X or N number of times. Now this N number could be anything. For example, for Netflix, they run the test case 35 times. Before they say, yes, this test case, this case is not flaky. The other approach to ensure there's no flakiness in the test cases is jumbling the test cases. How it works, the developer usually writes test case assuming test case will run in sequence. Now with running them in a non sequential order, you will bring in flakiness in the test cases at not written problem. The third approach, which is, you know, a good number here is about 15% test cases are flaky because of infrastructure instability. So looking at that, my test case fail if any problem or any infrastructure issues. So with exponential degradation approach, what we do that we can reduce with each subsequent run to a point where test case has to fail consistently. And if test case shows any different behavior, most likely it's not a flaky test case. The most advanced is called a mutation testing technique where test case stays the same. But what we do here is we change the underlying code. What we do there, if there's a test case which looks at passing, there's a code which passes string value. We would pass a variable value, integer value there and see how test case behaves. Are those values are hard coded or does it have space for dynamism? So this is flakiness, right? But there's more than this. Just flaky test is very critical, but at the same time, we want to have some intelligent mechanism that takes care of, you know, that builds a mapping between the test cases and the source code, right? So organizations are, you know, doing a lot of work in their own way, which is going back here. So which is they have built an intelligent system, like Google has built a dependency trees where they know what test cases are related to what source code. And if there's any commit or pull request on that module, it will trigger the only set of test cases. That saves, you know, a lot of time because you're now not running all the test cases, right? The first, I mean, the add on to flaky test management is if you are able to control the kind of test cases or the number of test cases you execute against each build. Let's take another example of if there's a change in static asset, right?  So you save on, because that's adds to your execution time. How some of the techniques are available or the tools available. So this system is called test impact analysis, which means analyzing how code change impacts what test cases and we should only trigger that. So Google did something with test time back in 2004 with some limited capability. It didn't go well, protest work by ThoughtWorks again. And the latest addition is Azure Pipeline, which is more of a commercial model. But again, it is limited to C sharp. And then you've seen a lately just done the same capability, but it does it does the mapping. It tells you what test case needs to run when there is a code commit on specific code. And that is that is also limited because it's running the local environment. So once, you know, imagine that you've got flaky test management in place. And then you have you have, you know that what test cases needs to run basis. The code changes having the visualization is very important because this will help you understand to do something about it. So this is Spotify's test flaky test management dashboard, big fan of the this and adopted by, you know, the approaches adopted by Google, Facebook, you know, in their own way, having the flaky dependency trees or AI driven systems. So what we looked at is, you know, having some control around flakiness and some control around test impact analysis and be able to have a visualization that that helps your empower you to do something about it. What we have done at Lambda test is we built test at scale. And yes, it is open source. So what is test at scale test at scale, we call it task is looking at three components of test testing and testing at scale is one is a test impact analysis. So it tells you which code has to be which test code has to be triggered against the pipeline or against the change. Then it does the proactive flaky test management that some of the technique that that we were talking about and an inspiration was all about those problem statements. And third is is is basically visualization. So I'll take you straight into the product and show you working, you know, the product. Yeah, so I hope my screen is still visible. Okay. So, so these are basically, you know, these are my past jobs that has run. These are some of the insights that we generate post the jobs have run. And we are giving you each test case, what it's doing, and what's the behavior. So for example, it's for each of the test case, I know that how it ran. And, you know, the gray one are basically the skip test, it's very important for for you guys for leadership to know how many test cases are being skipped because they're still counting into your repo but they are not adding any value. Then you're looking at the test cases which are getting failed with each execution and you want to see how much time well how long it has been failing for that means it's taking your compute. It's adding up to your execution time but it's not adding any value. And then once you fix it, how long it take to, you know, how long it took to fix the, the test case to be able to run successfully. There are additional trends we have just doing a quick time check. It should be the additional trends we have is running your takes execution I spoke about you want to combine and run it against couple of commit. So this is this graph is telling you how many times the job, what is the average commit against against the job runs about 30 commits. And then I have a very interesting, which is which is where the test impact analysis comes into the play is the one with the blue line is my total tests, and the one is the impacted one. Now, with each commit, I'm actually running the total, the whole suit, which doesn't make sense, right, we just, I just spoke about, but now the problem is you never had any mechanism to even know about. We, we have to run the whole suit to be able to have that confidence that this, this will be a stable belief. Now, another interesting catch from this, from this graph is, is between two commits, you've, the developer has done major changes that impact in the entire test suit. And this brings in big problem of, you know, the coding practices are not being followed, you know, you don't want to do a change that leaves, you know, that that's that create a lot of big risks, having major release, rather than doing you know, the minor releases, right. So with the time check, I'll stop here. And, like I said, the test dot lambda dot com is open source free to try. You know, the documentation will be shared post this call, and happy to take any questions now. Hello guys, thank you very much so they it was a very detailed and insightful talks. So I hope everyone enjoyed the session so it was marvelous. So guys, we are now handling the Q&A for next five minutes. So if anyone has any questions, please type it into the Q&A sessions at the moment. So, okay, so Sudhir, do you could please turn on your camera please. Right here. Sorry. Yeah, so there is one question here. That's sorry that's my keyboard, my bad. Yeah, no worries. I think someone typing the question one question is there so is it paid one or is it free to use TAS. It's free to use its open source. Okay, the second one is does it show the browser specific issues as well. Right now just the support in the unit testing we are integrating with one of the functional suit as well functional product that support functional testing so you will see that feature coming up very soon. Okay, so I'm just asking one question is do you have any community version or for that or is it just totally package base. It's the community version. Okay. I think we'll wrap this now. So, thank you very much Sudhir. It was a very detailed I think yeah it's another one coming up. Does it support all the sort of test. For now it supports unit testing. There's a roadmap to be able to support all conduct tests. So last one is coming up what are the major reasons for flakiness of test cases. I mean, I'm happy to take this over the discussion table, because we've just got two minutes left but there are primarily there could be many reasons and some of those are like how the test cases are written, what kind of consideration had been taken care of. There's a lot around the, the infrastructure plays a key role, like I said in. And yeah, I mean, the the ordering issues, though some of the problems that what we have seen is is what brings in flakiness. And I think if you look at from a from a from a tester's perspective, we are writing too many test cases. In fact, duplicate test cases sometime right so that also brings in multiple issues around when when treating the whole thing. Okay, so thank you very much Sudhir. It was fantastic talk, insightful talk, detailed talk. I hope everyone enjoy. So thank you very, very much. Thank you. Thank you everyone. Bye.


In the course of quantum chemistry of atoms and molecules, some of us might be wondering are we not wandering too far away from atoms. It has been a while that we have talked about atoms as such. Well, we will talk about little bit of atoms in this module or the next. But what we are doing right now is that we are learning the tricks by which we can understand how to go about handling atoms and molecules for that matter. So, we have talked about perturbation theory so far, we will recap it in a moment. Today we are going to learn about perturbation theory of degenerated states. Of course, all of us know what degenerated states are. These are degenerated states for example, states that have the same energy. Perturbation more often than not changes the energies of degenerate states with respect to each other that is called lifting of degeneracy. So, you can think of such examples that you know says Zeman effect or Stark effect application of a magnetic field or an electric field which changes the energies of orbiters that were earlier degenerate in such a way that they are no longer degenerate. So, this is one important reason why we must acquaint ourselves with perturbation theory of degenerate states. In fact, think of say metal ion complexes. Think of a free metal ion all the d orbitals are degenerate. The moment you bring in this octahedral field or tetrahedral field or whatever field the d orbital split into 2 or more groups. So, lifting degeneracy is a very important thing that perturbations often do. But it is not necessary that degeneracies will always be lifted as we will see in the 2 examples that we discuss in this context. In one degeneracy will not be lifted at all but the energies will be affected. In the other some of the degeneracies will be lifted like what we have shown here. So, in this schematic we have taken 3 energy levels and we are saying that some kind of a perturbation affects the system in such a way that in the perturbed system 2 of the energy levels have lower energy and 1 energy level has higher energy of course, very same. Before we go there let us remind ourselves briefly about what the salient points that we have learned in our discussion of perturbation theory of non-degenerate states. So, first of all we have learned that you have to write the Hamiltonian as the unperturbed Hamiltonian plus a perturbation term. We have written it in some other form earlier. Now the form that we are dealing with sorry about that the form that we are dealing with now is this Hamiltonian of the perturbed system is unperturbed Hamiltonian plus lambda into V where V is the perturbation potential and lambda is a parameter that we can regulate to increase or decrease the amount of perturbation that is there. And lambda finds its place in the expression for corrected wave function and corrected energy here we have lambda k is equal to lambda k 0th plus sum over j equal to 0 to infinity lambda to the power j multiplied by psi k jth similar expression for energy we have learned this. We obtained this very important relationship very important expression for the expression for the energy which is nth order correction to the energy. We learned this rather important expression for nth order correction to energy E k nth and that is equal to integral psi k 0 star V psi k n minus 1th integrated over all space. Here if you put n equal to say 1 then the second wave function also becomes psi k 0th remember what k is k denotes the energy level or the quantum number that decides the energy level. So, for hydrogen atom k is going to be n for harmonic oscillator k is going to be V. So, this is the expression we did not derive the expression but we at least showed it to you and we told you that it is possible to come up with this 2 n plus 1th correction term for energy in this manner and we found an expression for this correction to wave function as well in this manner V i k is this interaction integral you can say of involving i a i a than k th terms. Now, when we go over to the degenerate states then we encounter an interesting situation here for the kth level that is for the level associated with energy say E k 0th we do not have one wave function we have say n number of wave functions we designate them psi k 1 psi k 2 psi k 3 psi k n and so on and so forth. So, please do not get confused here in fact we are going to drop this k little later k here denotes the energy level the second number denotes the index of the wave function. So, you can think like this think of hydrogen atom in case of hydrogen atom let us say we are talking about n equal to 2 in n equal to 2 you have several l levels do not you. So, I can have n equal to 2 and that means that the energy level we are talking about is E 2 0th and we have associated with n equal to well I will make it simple sorry n equal to 1 let us say and we have to write too many terms if it is n equal to 2. So, I change it to n equal to 1 say let us say n equal to 1 to change that. So, for that we know that l is equal to minus 1 0 plus 1 right. So, how many energy states do I have how many wave functions do I have yeah how many wave functions do I have that is better corresponding to this energy E 1 0th I have 3 so I am calling them after this fashion psi 1 1 psi 1 0 psi 1 minus 1 if I write like this that means I am writing in terms of n l. So, this is the psi n l form or I could do something like this I could simply not write l but just write a sequence and simply say psi 1 1 psi 1 2 psi 1 3. So, 1 2 3 are well like role numbers indexes for the wave functions here. So, what I am saying here is that this minus 1 this is given the index 1 0 let us say is given the index 2 1 l equal to 1 is given the index 3 and so on and so forth. So, that is how we are writing please do not get confused here. So, all these n number of wave functions that I have written here actually have the same energy E k 0th of course you can have wave functions that are not orthonormal to each other but we can always take proper linear combinations and arrive at the orthonormal functions and we have discussed many times why we like to handle orthonormal functions because they are the coordinates they are the functions that define the function space completely. So, we have to take complete set of orthonormal functions let us say this psi k 1 to psi k n constitutes the complete set of orthonormal functions all with energy E k 0th this is my unperturbed system and this is the deviation from the case of non-degenerate states remember let me erase this because something is going to come up there and it is going to look very ugly. So, complete set of orthonormal functions and the problem now is that I am not allowed to write the expression I wrote earlier psi k m cannot be written as psi k m 0th plus sum over j equal to infinity lambda to the power j psi k m jth why not because these are all indistinguishable from each other I do not see a wave functions I cannot see a wave function. How do I know which wave function I am handling from energy because energy is an experimentally measurable quantity the problem is all these n number of wave functions have the same energy. So, we are back to that which which is which kind of situation remember we had encountered this when we were talking about multi electron atoms when we have 2 electrons and we say that we do not know whether 1 electron number 1 is in 1s orbital electron number 2 is in 2s orbital or the other way around right when we talked about helium ground state excited state we encountered such here in a little different context we have the same thing these wave functions are actually indistinguishable from each other because they are the same energy. So, if I write like this then I am saying that this let us take some wave function here let us say I am talking about hopefully here nothing will come. So, I will write here I want to know what is say psi 2 3 if I write psi 2 3 equal to psi 2 3 0th plus sum over all these things then what I am implying is that out of all these wave functions only psi 2 3 0th the 0th order wave function psi contributes to the part of wave function psi 2 3 how do you know who has said that you cannot have any contribution from say psi 2 5 0th can I have a contribution from psi 1 okay what is this 1 2 3 I wrote so can I have a contribution from psi 1 3 0th actually I cannot because the moment I change the first index the first index remember stands for K the index of energy energy will change it will no longer remain degenerate but if I am talking about psi 2 3 there is no way in which I can say confidently that psi 2 3 will a contribution only from psi 2 3 0th and not from say psi 2 10 0th or psi 2 1 0th or any other wave function which has the energy E 2 0th. So, what do we do in such a situation well we do what we did earlier we take a linear combination what are the coefficient coefficients of linear combination how do I find them we will worry about those later but to start with the way to go is take a linear combination so that we do not rule out a priory the possibility that other indexes wave functions unperturbed wave functions with other indexes also contribute to this part of wave function okay so we take a linear combination like this psi k m 0th is equal to sum over j equal to 1 to n psi k j 0 C j m C j m is the coefficient okay so what kind of coefficient it is the contribution of the jth wave function in the among the wave functions that have energy k 0th in the mth wave function of the newly created set okay and m of course has to be equal to 1 2 3 4 until n. So, just take a linear combination right now let us not bother about the form of the coefficient and how do we know it and we will talk about it later but if I write this then I am safe psi k m the part of wave function is equal to phi k m 0th plus this correction term okay see what I have done is in the correction term I have written it in terms of psi it does not matter it is okay you do not have to write phi there because after all phi is generated by taking linear combination of psi and this is how we write the expression for energy also. Now, so what do I get I get n number of perturbations n number of expressions for this correction to energy. So, I get n number of wave functions which may or may not have the same energy alright when will they have the same energy when the all the energy levels or some of the energy levels are increased or decreased by an equal amount of energy as a result of the perturbation when will they not have degeneracy when different levels are affected to different extents. So, we will have a look at both kinds of systems but before that let us work out this nice expression this is what we have. Now, what I will do is we will write the first order perturbation equation for degenerate states you know how to write first order perturbation equations we have done it earlier. So, I am skipping a few steps but I hope it will not be a problem for anyone. So, this is what it is h 0th minus e k 0th operating on phi k m 0th equal to e k first order multiplied by phi k m 0th minus b operating on phi k m 0th remember I have jumped steps. So, I strongly recommend that you please work this out yourself because otherwise when I say things in the flow of the lecture you feel is as if you have understood everything but then if you sit down to the pen and paper and try to work it out yourself then then if you do not have a question about how I have done things that means you have not perhaps understood everything you should be surprised you should be puzzled you should get confused should have questions then only you will understand this. So, please work this out yourself even though I have skipped it you should not and you should work it out after the fashion that we use for the non-degenerate case. Alright, so this is the first order perturbation equation let us see what we get from here. So, as usual like what we did earlier we are going to left multiply by psi k 1 0th well complex conjugate of that and integrate over all function space what do I get I write psi k 0th well complex conjugate of that on the left hand side integrate that will be the left hand side right hand side when I integrate I get two terms in one this e k first is going to come out because it is a constant. Second one will involve a triple product second integral of phi k 1 0th p and phi k m 0th. So, this is what we get we are not very difficult to understand now what what do we do now well first of all you might remember that this h 0th minus e k 0th is the annihilation operator not for the phi 0th terms but for the psi 0th terms right and I can easily get psi 0th and phi 0th to interchange using a tool that is by now familiar to us the turnover rule. So, when h 0th minus e k 0th operates on psi k 1 0th I get 0 then we do not even have to bother about the other way function. So, left hand side is taken care of that is equal to 0. What am I trying to do I am trying to find an expression for the first order correction to energy right. Now, the next step obviously is to take this expression of the phi is phi 0th and plug it into these integrals ok plug in these linear combinations into these integrals and this is what you get well actually what I should also do is this this coefficient is a constant right. So, it has no business being in the bracket so we will bring it outside the bracket but we will not bring it outside the summation right please do not forget there is a c j m j is there. So, if you bring out bring it out of the summation then it has no meaning so we will not do that we will bring it outside the integral but inside the summation next this one is going to become again phi k m 0th is replaced by sum over j equal to n sum over j equal to 1 to n phi k j 0th c j m c j m is a coefficient. So, I bring it outside the integral but not outside the summation sign right. So, when we do that we get this well we have to remember something we have to remember that these size are actually your I should have written 0th everywhere this for I mean please note that but when we remember that these are actually orthonormal functions and when we plug them in then what will we get we will get something like this right. We will get this well okay let us see what we will get but before that we will repeat the operation for 2, 3, 4 and so on and so forth. So, when we integrate over all function space this is what we get we get n number of what are called secular equations okay remember there is one summation over j right that gives you from left to right of an of the left hand side of an equation there is another summation over I let us not forget that. So, this is what we get now how do I solve this remember to make things simple we have written this as b i j and also we have written this first order correction to energy as x. Now I hope you see that this x here is multiplied by delta i j why because we are working with a complete set of orthonormal functions. So, when i is equal to j then we are going to get 1 the integral will be equal to 1 when i is not equal to j then the integral is going to be 0 which integral this integral not this because here you have this v in between this one is going to be either plus 1 or minus 1 that is written as delta i j okay. So, now this is what we have got so far x equal to first order correction of energy and we want to find a better expression for that great. So, as we said we have created this n energy level some of which may or may not be generated. Now we write this n secular equations in the form of a matrix whenever we have a system of secular equations always write it in the matrix form and when we do that this is what you get we get v matrix multiplied by c matrix is equal to c matrix multiplied by x matrix.  So, see how beautifully this problem gets formulated it becomes a matrix equation. So, v i j essentially i and j denote the position of v i j in the matrix v capital V. So, this perturbation takes up these i j positions in the matrix c's are called the eigenvectors if you want to know more about eigenvectors please go through our lectures on group theory in chemistry. So, eigenvectors eigenvectors are essentially coefficients. So, the coefficients are the components and capital X turns out gives us the values of the first order correction to the kth level energy and this is also diagonalized. So, beautiful form can be solved by many different techniques no worries really. Now the field is set for discussion of two examples that we are going to do next day. But before that the salient points first of all the degenerate states are such that several eigenfunctions can actually have the same energy. So, you cannot express a perturbed wave function as a an unperturbed wave function just like that plus the correction term you have to generate a linear combination of the unperturbed wave functions from where we are going to begin because there is no saying which unperturbed wave function has contribution in which perturbed wave function it is not just a wave function and its perturbed form they are indistinguishable. So, many can make contributions that is point number 1 point number 2 is when you accept that you are going to get this very nice matrix equations in which these X values are going to turn out to be the X values here first order correction to energy. So, that is where we stop this module today. In the next module we are going to discuss hopefully two examples one is non-rigid rotor the other is stark.


All right, Martin, it's so exciting to have you here and talk about cloud economics. Everybody's talking about the different choices in cloud and you and Sarah Wong just did this really in-depth analysis about the economics and looking at public cloud versus alternative. So tell us a little bit like why did you even like study this in the first place? So the Genesis was actually very simple, which is I sit on a bunch of boards and started noticing that like the number one item by far was cloud spin. And it was one of these things where we're like, oh, like over time it's gonna go away but it like tended to just get bigger. And when I talk about like number one item, I'm talking about like maybe like, you know, 50% of COGS. And so based on that private market experience and you know, all public companies were once private companies, Sarah and I asked a question like is, you know, like in the last six years we've had something like 126 companies IPO and all of them are cloud native. And this is a pretty new thing. So we just asked the basic question, like is this, you know, just the companies that I know about or is this everybody? And so we just asked the very simple question, how is the cloud impacting the economics of cloud native companies? But that was the beginning. That's, yeah, and that it seems like something that all these companies are wrestling with. And so it seems like it generated a lot of controversy. There were certainly tweets flying everywhere, people arguing and, you know, maybe you can tell us a little bit about some of the big takeaways from what you found when you actually dug into the data points. Sure. Okay, so the first thing to, I think that's important to stress is that this is a very new phenomenon. So like the cloud has been around for a long time. Public cloud companies have not. I mean, like that are actually built only on the cloud. I mean, literally maybe five years new.  So what we did is we looked at public software companies that were cloud native, we looked at 50 of them. And we asked two questions. The first question we asked was, if you look at the COGS, what percent of the COGS is cloud? Now remember prior to cloud native companies, cloud was some back office thing. It wasn't a part of COGS. So it impacted the economics differently, right? So this is just software companies. This isn't banks. This isn't old IT. This is just, you know, public software companies that are cloud only. So we asked what percentage is COGS? And then we said, and then we asked the second question is, you know, if you could reduce that, how does that impact share price? So here are the results. The first results, both results to me are just totally staggering. So the first result is, if you draw, across these 50 companies, on average, 50% of their cards is cloud. 50% of it, across 50 companies. And by the way, the way that we did this is we kind of looked at the rest ones and stuff at the committed cloud spend. I mean, we actually spent a lot of time doing this. And then the second result was, if you reduce that by half, however you decide to do that, right? Like let's say that you optimize your code or you know, you repatriate some set of workflows or whatever you do, it increases the share price across those 50 companies by about $100 billion. So if you extrapolate those results, yeah, these numbers are just so huge. So if you extrapolate those results to the industry, it's a $500 billion to a trillion dollar problem. And this is like market cap that's being captured by the cloud, assuming you can reduce your crowd spend by 50%. Now, you've talked a couple of times about COGS, which is cost of goods sold. So this is the raw cost that these companies account for across everything that goes into taking their product and service, basically out to delivering it to the customer. Now, one of the things that I noticed in the conversation that happened around this, I saw that people were pushing back and they were saying that the only thing to prioritize and optimize in terms of, getting a product or service to the customer is basically time and speed. And that by even though this is a very high cost, 50% on these public clouds, it was helping them with speed to market, with time to market. And that seemed to be like a pretty common sort of response and counterpoint to some of your findings. How do you respond to that? What's your feeling on that? Yeah, so, I mean, for those of you who are listening that don't know my background, like I was part of Cloud Wars 1.0. So Cloud Wars 1.0 was cloud versus internal IT, right? And so let's say, you're at a fortune, whatever, 1,000, and you need to get some desktop, you work with internal IT like this is kind of like the framing of the first Cloud Wars. And it's a very common one. And here it totally makes sense to kind of consolidate and have professionals do it, et cetera. But again, the study that we're doing are all these public cloud companies that are these very, very large companies. And to reduce a discussion to something as simple as well, you should just be focusing on growth. Would be the same thing as saying you should just take on technical debt, right? Somehow it doesn't like make much sense, right? And what's interesting is because the economics changed so much actually reducing cloud is an argument for growth. So let me explain that. So for example, if you reduce cloud costs by 50%, for most of these companies, you free up one to $2 billion of market cap that will flow through to cash, like cash money, which you can use to hire and invest in growth, right? And so the actual economics are so acute, you can make an even stronger argument that the way to grow is actually to cut your cloud costs so you can have the spend to do that. And the reason that I brought up Cloud 1.arguments is because like I think most of the discussion around the piece was like the old discussion around cloud versus internal IT, which had nothing to do with the piece. The piece is like Cloud Wars 2.0, which is cloud versus SaaS. And I think that's where the interesting discussion is. Yeah, so it seems like your analysis is pretty compelling at these really large public sort of cloud-native companies that it is, as you said, a very new phenomenon. And perhaps when you're really small and getting started out, yeah, public cloud, swipe your credit card, iterate, like it's not a big economic issue, but is there a crossover point in your mind? It seems like you found that sort of the clear case at these 50 largest and then perhaps you would agree if you're really small, iterate, it's not a big cost, but is there some sort of crossover point in your mind? So to me, the interesting setup for the discussion is if you're a large public cloud company, on average your costs are 50% of the cloud, right? It's impacting your market cap by a few billion dollars. And that market is controlled by an oligopoly of three companies whose margins have increased from 18% to 30% over the last 10 years, right? And so it's quite unlikely that like, you'll get much margin extraction out of that. I mean, oligopolies just don't work that way. So then the question is to your point is what do you do about it? Is it something that you worry about early? Is it something that you just let the market take care of? Like what do you actually do about it? And I don't think there's actually a good answer for that. However, we are starting to see signs in the industry of people recognizing it and thinking about it quite early. So for example, Suheel, who is the founder of Mixpanel has created a new company called Mighty for exactly this reason of building the entire hardware stack. If you've watched what Cloudflare is doing, Cloudflare is taking direct aim at AWS and announcing things like R2, which is kind of an S3 competitor by doing egress pricing. And so again, I think the question is less, if you are doing a startup, should you or should you not use cloud? The answer of course is yes, you must use cloud, for sure. Like that's the only way to do modern development. I would never recommend anybody do not use clouds unless you're a core infrastructure component. Then maybe you should think earlier about that. And then once you start getting at scale, you may want to start looking at these alternatives. There are now these kind of niche clouds that are focusing on cost optimization in certain areas that you may want to consider using. And so I don't think, the purpose of this piece was not to tell SaaS vendors to repatriate, it's to say that you've got a trillion dollars that are locked into this unsteady truce with an oligopoly and something's gonna happen. And it's gonna be big and a lot of spend is gonna be moving around and I'm all for it. So one last question for you. You know, this piece was awesome. Thank you to you and Sarah for all of the time and analysis that you put into it and publishing it and starting this awesome discussion. I think one of the reason why it got so much reaction from so many people was that in some ways it sort of defied conventional wisdom that you were talking about where it's just accepted that you go and you go to a hyperscaler and that's it and that's all you ever have to think about again. So I think that was great to put that out there. My question for you now is, what other kind of pieces of conventional wisdom do you think there are, especially in the tech industry that are changing or that are wrong or that, you know, what's the next firestorm you're gonna start, Martin? Well, I mean, I could go through the firestorms I have been starting, right? I mean, like the one previous that I'll actually work back for is I'm not quite sure what the next one is. I kind of talked about them as they come up. The one previous to this was I made the statement that kind of irritated a lot of people which open source doesn't really matter anymore. And the reason I said that is because I think the primary way of monetizing infrastructure as a service these days, and as the service, the hard part about that is less code and more kind of operational stuff. And so, you know, a lot of the, you know, top infrastructure companies are strictly closed source companies, we're just talking about the cloud. The cloud is almost strictly closed source, right? We're looking at AWS or GCP.  Like if you run a workload on GCP, how much of that is actually open source? Way less than if you're running on Red Hat. So that was one. And then before that, and I believe this very strongly, you know, like I just think that we think of data as this magic thing and we say things like, oh, data network effects and data is the new oil. And I just think it's the wrong analogy. I don't think data has network effects. I think that companies that rely on data actually have diseconomies of scale. Like the more data you have, like the worse your economics are. And so I think that you need to trade like unit economics is the first class thing of your dealing with data. So those are three contrary positions I've held in the last two years. And then the next one, listen, you'll be the first to know. All right. Well, thank you so much. You know, we've had you on the weekly show before. We'll definitely have you back on to talk about data, debate, the open source imperative or not and all these other great topics.  We'll have you back on for that show. But thank you so much for joining the keynote today. Oh, it was a pleasure. Thank you so much.


Let me start by asking the audience a question. Now who here has ever lent their car to their family or friends or colleagues? Yep. Okay, some of you. So have I. Well, this is what we call sharing. My thesis is specifically about the sharing economy where consumers and companies such as Uber uses technology to connect consumers who have a car with those consumers who need a ride. So here consumers are playing two roles. They play a consumer and they play a supplier. Now the sharing economy is huge and it's growing every minute. Uber, for example, has gone from zero to 90 billion New Zealand dollars in just eight years. Now, despite the importance, there's not much we know about the sharing economy. So my research is about a key success factor of the sharing economy, balancing supply and demand. So the marketing literature has already stated why consumers uses services like Uber and become passengers. But what about the supply side? What motivates consumers to become suppliers in the sharing economy? So for example, why do they want to become Uber drivers? I will answer this question by using two studies. Firstly, exploratory interviews and then confirmatory questionnaires with recent consumers who've decided to become Uber drivers and thereby suppliers in the sharing economy. I will look beyond the obvious financial appeal of extra income, but look at a more nuanced get a more nuanced understanding of other motivations and different segments that may exist. Now, motivations may include a need for social interaction, a desire for a better world, or perhaps you just want to be your own boss. Well, here's a burning question. Why is this research so important? Well, this research will address a critical gap in our understanding of what makes the sharing economy work. And from a practical perspective, it will allow companies in the sharing economy to balance supply and demand adequately and allow us consumers to make sure our assets are being utilized efficiently and effectively and thereby encouraging sustainable consumption on a global scale. So now who would like to share their car with me? Thank you.