Okay, let me begin by voicing my agreement with Oscar Morgenstern, who in a 1972 Kyclos article said the following, economic theory is unending because we are confronted with an open system. The idea that we could have a closed system of economic theory say of the Valerian type is a feudal one, end quote. So I think it's important to recognize that much of what I'm going to say mirrors my conception of what the economic system is like. If the economic system is truly open, then I think our economic theory is going to have to be open and it's going to have to reflect that. This philosophy I think is in sharp contrast with what I will call the quote unquote comfort school of Austrian economics. Now the comfort school believes that all basic propositions of economics have been solved and that Austrians are in possession of the whole truth and nothing but the truth and that our only mission is to spread the word. Now this comfort school and my view of what Austrian economics is all about are not compatible. In discussing a better concept of coordination, I obviously believe that some really basic work still needs to be done in economic theory. Let me say at the outset that the better concept is one that portrays coordination as imperfect, as plans which do not mesh completely, arising from an incompletely specified process. In other words, I'm elevating to a level of almost principle here the virtues of a certain degree of vagueness. Why do we need a new perhaps looser concept of coordination or equilibrium? We need it because we need first a concept of coordination that makes sense as an idealized or ideal typical result of a process. It does not make sense to use a concept of equilibrium that is inconsistent with our characterization of the dynamic process that generates it and this is the point which Jerry and I made in our book. But earlier than our book in 1977, Malinvaux justified the same claim in a different way. He said to rely on a general equilibrium formalization is to accept a shortcut, that is the consideration of only those equilibrium states that would result from dynamic adjustments. So the idea that Malinvaux proposes is different than ours, but nevertheless I think compatible. If we're going to deal with equilibrium theory at all, then we ought to deal with equilibrium theory in a way that pays its respect to the fact that equilibria are generated by processes. So we shouldn't talk really about equilibria that no conceivable process could ever generate. The second reason that we need a new concept or a better concept of equilibrium is that we need a concept that makes learning processes possible. In a way that the concept of disequilibrium cannot. Austrians often say that disequilibrium in a certain sense generates learning, but I think that's not quite right because disequilibrium does not have enough structure or order for learning to take place. In other words, what I'm claiming is that learning can only take place with an intermediate degree of order. So dichotomizing the world into equilibrium and disequilibrium doesn't take account of the fact that the so-called disequilibrium that's supposed to generate this learning must itself have a certain amount of order, a certain amount of structure in order to generate the learning that is requisite. So for these two reasons I think we need a better concept of coordination. Now let me throw light on each one of these reasons by referring to some previous theoretical developments which I think are consistent with the ideas that I'm trying to elaborate here. The first again, equilibrium as an idealized outcome of a process. Now this has roots in some of the work of Hayek, but let me first start out by showing the way in which Hayek argues for a concept of exact coordination, that is to say a concept which is at variance with the looser concept of coordination that I'm trying to promote. The idea of exact coordination, that is to say the idea that there's perfect coordination of plans, requires I think a rigid distinction between equilibrating and disequilibrating movements because only then can we freeze the disequilibrating shocks and allow at least notionally for complete adjustment. So the idea is that we have a shock, a disequilibrating movement, and that once we had that shock we can take it as a separate factor separate from what comes next, namely the adjustment, the equilibrating adjustment to that disequilibrating shock and allow then for all the equilibration to take place resulting in a perfect adjustment, a perfect coordination. Now as you can see the analysis requires that on the one hand we can say this is disequilibrating, on the other hand this is equilibrating and the two concepts don't get mixed together, there isn't any question of ambiguity of one or the other. So it requires this idea of exact coordination as a useful analytical tool, requires the cooperating idea of the complete separation of equilibrating from disequilibrating movements. Now Hayek clearly recognized that disequilibrating shocks would continually occur, but as we saw he carefully segregated disequilibrating shocks from equilibrating adjustments. In his article competition as a discovery procedure Hayek calls discovery what is essentially discovery of adaptive means that is to say discovery of equilibrating adjustments and he uses the word discovery only in that context and anything else in other words any other kinds of discovery in the colloquial use of the word are called quote-unquote unforeseen change. That's very interesting because I think that Kersner follows that same tradition. For Kersner, Kersner has used the word discovery countless times and sometimes you think he's getting at the idea of a completely open system in the way that I would mean and the way that a number of other people would mean but it becomes very clear that what he means is really what Hayek meant. Discovery is discovery of adaptive means. Discovery is discovery of equilibrating adjustments and it is rooted in a use of the term that Hayek I think initiated but I think it's illegitimate to separate these two kinds of change and the reason for that can be found in Hayek's work itself and this is exhibit a certain tension in Hayek in economics and knowledge which is earlier than competition of discovery procedure Hayek says the following quote, the tendency toward equilibrium is only toward an equilibrium relative to that knowledge which people will acquire in the course of their economic activity relative to that knowledge which people will acquire in the course of their economic activity end quote. Now what is going to cause people to acquire knowledge in the course of their economic activity? Well one thing that Professor Kersner has emphasized very much in his work is the attraction or the lure of profit opportunities. Profit opportunities will cause us to make discoveries but these discoveries I suggest will disrupt any implicit equilibrium toward which the system might move. Just as an existing technology might be disseminated by this discovery procedure by the lure of profit opportunities and this of course is what one of the things that Hayek and Kersner have in mind that there may be knowledge of a technology that somebody has somewhere in the system and that gets disseminated by the fact that other actors in the system want to make profits but at the same time as this dissemination discovery takes place it's also the case that the very lure of profit opportunities will cause people to discover entirely new technologies that nobody in the system knows and these are by definition disruptive of any implicit equilibrium toward which the system might be at any point in time moving. Now this distinction between what I would call dissemination discovery and truly originative discovery has also been made by Ulrich Witt in his own work using another terminology. Moreover, Franklin Fisher in a 1982 book called Disequilibrium Foundations of Equilibrium Economics saw something of the same problem but his perspective was a little different perhaps a lot different but still he saw the same problem. He says the following in an ongoing economy what constitutes an exogenous shock he asks how is such an original shock to be distinguished from the endogenous shock brought about by adjustment to the original shock and quote uses the word shock a little too often here but basically what he's arguing is two things one is you're dealing with a chain of interconnected events you have an exogenous shock and then you have adjustments but they're really in a basic sense interconnected so separating them out is a bit artificial but secondly the very idea of an adjustment could be looked at as a shock to the system because people don't expect those adjustments in competition as a discovery procedure Hayek makes the entirely valid and very important point that if we knew the outcome of competition we wouldn't need competition in the first place in our chapter whatever it was cool on competition I don't remember the numbers of the chapter was chapter 6 maybe I don't know we make the point this very point starting out with this example of a sports game and all that I mean why would you need to play the game if you knew the outcome to begin with right and also the idea is that if you knew that the results of competition why would you need it in the first place precisely because you don't know the results well part of the results of competition are the adjustments that are going to take place to these exogenous shocks if you knew what the adjustments were going to be in the first place right you wouldn't need the method of competition to arrive at the best adjustments so that these adjustments are necessarily a surprise to many people within the system and as a surprise they are shocking I mean if we want to use that terminology right they are shocking just as much as the change in tastes which is considered the change in the data the exogenous shock is shocking so that adjustments are every bit as much a shock to people as original shocks to the system exogenous shocks, the changes in the data are and the only reason that people don't generally recognize this the only reason that people don't generally recognize this I think is because in neoclassical economics the idea that adjustment might be have surprising elements that adjustment might not be something which could be known before a process takes place doesn't really figure in very well because first of all there aren't any processes most of the time and secondly because there isn't a conception that competition discovers anything really new now there are always exceptions to my generalizations about neoclassical economics and it's interesting even back in 85 when we wrote our book I think there's a chapter somewhere in which it said that one of the difficulties in arguing against neoclassical economics is it's so spongy it's so flexible that almost anything is going to have some exceptions, any generalization but nevertheless I think that's still the basic idea inherent in neoclassical economics is that adjustment isn't anything isn't much of a surprise and therefore the distinction between exogenous and endogenous can proceed extremely rigidly most of the time now Hayek as I say did understand this feature of competition and that competition discovers things that otherwise people don't know but of course that's at variance with his view that you can keep a strict separation between exogenous and endogenous but there are other ways in which Hayek sees that the loosening of a concept of coordination is important in the very same article competition's discovery procedure he developed a concept of order his words, his word, order which quote can be preserved throughout a process of change end quote now this looser concept of coordination came out of a recognition that adjustment processes involve the disappointment of some expectations and this is where he seems to see that the concept of equilibrium as he has previously used it has limitations because in competition's discovery procedure he says quote a high degree of coincidence of expectations is brought about by the systematic disappointment of some kinds of expectations so what is he saying? equilibrium or equilibration is brought about by disequilibration it's kind of paradoxical I have a few other paradoxical statements to make here and I'm kind of embarrassed by them because I don't want to be people to go away from this saying that Rizzo has developed this sort of eastern philosophy way of talking that equilibration is brought about by disequilibration is only the first one I can treat you with but it's true I mean it is true that the degree of coordination that we in fact do enjoy and let me keep that open or bracket that in terms of what degree that is but the degree that we do enjoy is brought about by the systematic disappointment of some expectations by a certain amount of disequilibration a certain amount of shocking of the individuals within the system okay so I think this begins to tell us some thing about the background to why it's important for a concept of equilibrium that makes room for a concept of equilibrium that is consistent with processes of equilibration be developed because if the very process of equilibration as we generally call it involves a certain disordering a certain disequilibration then it is inconceivable that the equilibrium that could be brought about the equilibrium as an ideal type that could be brought about would be a perfect coordination would be a perfect dovetailing of plans because the very process that brings about whatever dovetailing that does exist is fraught with uncertainties and shocks and disappointments okay now remember the second reason I gave for why we need a looser concept of equilibrium is that we need a concept of equilibrium as a precondition for discovery driven processes remember that I said that disequilibrium doesn't have enough structure for there to be learning as far as the concept of disequilibrium tells us it could be totally chaotic I mean there is no limits placed on what a disequilibrium is but I think limits have to be placed in order to talk meaningfully about the discovery that takes place in markets so the learning that we are all concerned about requires then this intermediate degree of structure let me talk a little bit about that what this intermediate degree of structure can be first maybe we can begin with the level of the individual because I think here it's very commonsensical to see why we need structure in order to learn an individual we know from the philosophy of science can learn only if certain things are taken for granted or treated as given there's something called the Duhem quine thesis and all that really says is that we can never test a single hypothesis but only sets of hypotheses in other words not everything can be tested simultaneously everything that we believe can't be tested simultaneously there's background knowledge there's auxiliary hypotheses for example in the natural scientific context one auxiliary hypothesis might be a hypothesis or a hypothesis about the observational instruments that we use we're looking up at Jupiter and we have a hypothesis about the rings of Jupiter or there are rings of Jupiter now not only Saturn but there are rings of Jupiter we might have a hypothesis about the moons of a certain planet and we're testing that hypothesis but the observations that we use go through this telescope well there are hypotheses about telescopes and whether they show us things that are really there and the time lapse and various factors like that so there are auxiliary hypotheses that we have to take for granted in order to test the main hypothesis there are also initial conditions that we're presupposing so there are all these factors that we are in a way simultaneously testing so in order to test any single hypothesis we have to take a lot of other things for granted and the things we take for granted are in a way the order that we're presupposing it's that set that knowledge that we take is given that constitutes a context in which continual learning can take place additional learning can take place so at the level of the individual the individual scientist in this case but we can go beyond science into everyday learning in the sense that in our everyday perception of the world we're taking a lot about the world for granted in an attempt to learn some additional or marginal factors it's very much consistent with the marginalist approach in economics small changes of what we're concerned about so the point that I'm making here is that order is necessary at the level of the individual in order for learning to take place it's also necessary at the level of society and here I want to give an example which has been discussed by Brian Loseby in a very important book called Equilibrium and Evolution unfortunately it's a book which did not remain in print very long but I think it's one of the most important books for people to read in order to get insights as to as to what might be the future development of ideas within the Austrian tradition Loseby does not consider himself an Austrian but I think his ideas are extraordinarily important for Austrians to take account of and the example that Loseby gives takes off from the dictum that we all recall from Adam Smith Adam Smith said as we all know the division of labor is limited by the extent of the market and even George Stigler wrote an article with that title so it's very important Loseby says that we could in effect says that we could easily read extent of the market as degree of coordination so we could rewrite this the division of labor is limited by the degree of coordination of markets now why is that well if you think about it highly divided labor and specialization can't take place unless there's trade and trade isn't going to take place it's discoordinated if I can't be sure that I can go out there and buy clothes when I want them then I'm going to wind up making my own clothes same thing with other products so the degree of coordination is extraordinarily important for the division of labor and specialization to take place now one of the consequences of the division of labor according to Smith and I guess it's the fourth reason that's generally not paid much attention to is the idea that division of labor promotes increase in technological knowledge as people concentrate on specific tasks they learn more about the elements that go into those tasks and therefore they learn more about the possible ways that Smith argued of mechanizing the tasks so that technological developments are more technological advances are more likely to occur in a context where people have much more precise knowledge of the elements constituting a productive process now this has a very very important implications because first of all it immediately throws out a whack the whole idea that the division of labor could be viewed in a purely static context which is of course a thing which we're often taught in economics that we talk about the division of labor and still maintain all of these nice what are called marshallian curves that allow for a existence of a competitive equilibrium now we also know that Marshall had certain problems with all that but one of the reasons that Marshall had problems with this is because Marshall understood better than the neoclassical descendants of Marshall Smith's point that the division of knowledge the division of labor leads to an increase in technological knowledge so there are a couple of things that we need to recognize we need to know that for knowledge to grow there must be a certain amount of coordination both of the individual and the social level coordination of ideas at the individual level is most especially what I'm getting at at the social level a certain coordination of individual actions and that secondly that this degree of coordination is itself unstable because changes in knowledge that the coordination makes possible constitute changes in the data and therefore disequilibrating ok so now let me get more specifically to the point what concept of coordination I've been vague about that and I'm going to try to be more precise but I don't know how precise I'm going to get because I'm not sure how precise it is in my own mind go back to the Hayekian idea of order Hayek says in competition as a discovery procedure once again the expectations of transactions to be affected with other members of society and on which all the plans economic subjects are based can be mostly realized mostly realized now the interesting thing about this is that he's obviously developing another word other than equilibrium for a concept of coordination because here he's trying to separate out the concept of coordination from the concept of order because in one hand he used a concept of coordination as a purely mental construct in this case he's using a concept of coordination as what is called an ideal typical construct that is to say a construct which is he believes as he says later in the same passage the real world to a certain extent approximates so being driven by the need by the recognition that these processes themselves create disorder he says well look the real world can't look like what my mental construct looked like that is perfect coordination the real world is going to look like a world in which well most of the time people can affect their plans but he's getting I'm not saying he has developed an idea which is equivalent to what I'm trying to convey but what I'm saying to you is that in Hayek there are the roots of this idea of a looser concept the roots are made necessary by his concern about describing at least in approximate terms the real world in another article called principles of a liberal social order which is about normative issues again an attempt to get at the real world but here in a normative way and here he gives a definition of abstract order which is very much consistent with the basic theme I'm trying to convey to you he says that abstract order he says that we ought to try to achieve an abstract order which as a whole provides the best chance for any member of society selected at random successfully to use his knowledge for his purposes for any member of society selected at random successfully to use his knowledge for his purposes but it's only the best chance so that means that a lot of people will be frustrated even in this ideal world of the rule of law that he advocates so once again a concept of imperfect coordination now what's unsatisfactory or at least not entirely satisfactory about where Hayek goes with all of this is that this is entirely the concept of order or equilibrium as an end state or a result of some process but nowhere in the discussion does he provide the kind of basic structure necessary to get the process going so what I'm saying is that a concept of order that is that is loose is not only to be conceived of as the end state of a process but also to be conceived of as a generator of that process now they may not be exactly the same structure may not be exactly the same structure or order that does both jobs but nevertheless I think that's what we've got to be headed for so let me talk a little bit now about something that I've briefly raised throughout this talk and that is the idea of the or at least briefly alluded to the idea of the scientific research program and learning Brian Loseby again makes a big deal about this the importance of the scientific research program he puts it in a Lakotosian framework what I'll say will follow that a little bit although I don't think we need to take that all that seriously because really all he's talking about is this structure of learning which is not completely deterministic but before we get to the scientific research program let me talk a little bit about learning in Austrian economics at this point the major theory about learning that we have in the sort of Austrian economics proper that's sort of narrowly conceived is Kirzner's theory of entrepreneurial learning I suggest that Kirzner's theory of entrepreneurial learning is instantaneous learning and therefore it's out of time now I have I have perceived that for a long time but just the other day I went into his office one of the virtues of working with Kirzner as you can sort of ask him directly and usually he gives you a straight answer and he says here yes I believe that entrepreneurial learning is instantaneous it's a flash of insight those of you who have seen him lecture notice that sometimes he does sort of snap his finger when he talks about the entrepreneurial alertness so it is instantaneous the familiar arbitrage examples that he gives illustrate that right? Apples here at ten cents, apples there at five cents you arbitrage the difference at a point in time right? There's no time elapsing but even the inter-temporal exchanges apples five cents today apples ten cents tomorrow even allowing for time preference and all that stuff there's a difference and but the insight that there is this difference this arbitrage inter-temporally is again instantaneous so learning in Kirzner's system has many of the attributes of neoclassical economics namely it's out of time and that accounts for the sort of deus ex machina version of attribute of it that is it is a alertness to the rescue right? Alertness to the rescue of what? Alertness to the rescue of movement toward equilibrium I don't think that's adequate I think what's happened is this view has replaced one error with another error the Bergson the philosopher Bergson makes a distinction between two errors in dealing with these issues relating to time the first error is radical mechanism now we Austrians have beat radical mechanism to death as far as Austrian economics is concerned and that's a big as far as radical mechanism is dead it can't recover and that's it the stake has been driven in the heart alright? but there's something else called radical finalism radical finalism is the view that the end state implicit equilibrium and the current data sort of exercises a kind of pull on the system so that at any point in time there's an implicit equilibrium and at that point in time all the adjustments sort of pull toward or a pull toward that implicit equilibrium right? that's the idea that Kersner has expressed as entrepreneurial alertness is equilibrating right? meaning it's drawn toward the data implicit in the equilibrium at that moment in time that's radical finalism now the reason it's inadequate from my perspective it's precisely because it ignores the whole problem in clearly distinguishing and or rigidly distinguishing between equilibrating and disequilibrating behavior by its nature because competition is a discovery procedure because we can't predict the results of markets equilibrium adjustments so to speak are shocking they surprise people and therefore they're disequilibrating Hayek says what coordination we do enjoy is the result or is brought about by the disappointment of expectations of some people because they can't predict the results of the market process so I think the radical finalism ought to be avoided as well as the radical mechanism helps us to a certain extent because it is a it is the concept of research program is an order but it's an incompletely specified or incompletely determined order let me give you a basic idea of what I mean a research program and this is consistent with our colloquial understanding of what the words mean you don't have to be an expert on Lakatos to get this idea a research program tells us among other things what we can and can't do to acquire new knowledge it gives us the rules of the game Lakatos calls it the negative and positive heuristics but it just gives us the rules of the game of acquiring new knowledge but it doesn't tell us what the new knowledge is going to be it doesn't determine it to say there are certain rules of the acquisition of new knowledge doesn't prejudge the outcomes otherwise it wouldn't really involve the growth of knowledge at all it would just be you know saying what you already know in different words so a research program doesn't determine exactly what it will be learned but provides contours, provides certain order, provides certain hypotheses which you don't for the time being challenge you know the hard core Lakatos calls it I once wrote an article trying to compare mesas with Lakatos in this respect and probably I wasn't quite right in a number of respects but nevertheless there are certain things that we as Austrians at least for the time being are not willing to challenge and maybe we'll never be willing to challenge it they're really so basic about our understanding of human beings but then there are other things we will be willing to challenge well that's the order that that sort of hierarchy of things that are unchallenged or for the time being unchallenged things that we do challenge things that we protect by more elaborate hypotheses ideas about what we can do in the search of knowledge or what we can't do this provides an order but an incompletely specified order but the learning that occurs in a research program is not only in that sense partly orderly and partly surprising but also in a very basic and important sense for me anyway who worries a lot about time and as I get older even more about time in a different way but it is learning in time this is learning in real time it's learning in the extended present that is that by the extended present I mean a present which takes account of what we know in the past memory and takes account of memory and takes account of anticipation memory a research program's memory in effect is the past knowledge perhaps the hard core of irrefutable propositions what we're left with by previous investigations the expectation is the filter the filtering of our new experience that the negative and positive heuristics provide it's the limits that it sets to our learning but these limits as I say are incompletely determined so this is a way I think of grounding learning in real time not making it a deus ex machina not making it a sort of spontaneous something to the rescue but making it grounded in what has gone on before and what we expect to happen in the future and in a way that I think is much more consistent with the reality of how people learn the empirical reality of how people learn research programs could be viewed as islands of coordinated learning we could talk about firms for example as embodying a research program VIT has done work in this respect and I don't believe he's used the term research program but nevertheless I think that's one of the ideas that's embodying in this the entrepreneur conveying his sense of the mission of the program to other workers is a critical idea the community of scholars and researchers are obviously involved in research programs to some lesser extent we might consider an economic system has having a common framework to the extent that there is a common framework of law a common framework of other institutions ok in the final analysis then there really are two concepts of coordination that we must use there really should have been titled better concepts plural of coordination the first is that of an outcome of the market process the ideal type people can mostly achieve their goals can mostly implement their plans they are mostly coordinated with those of others but mostly leaves room for a lot of messes second a shared mental model or structure of learning that's also an order a kind of coordination but that's one that generates learning and generates the process that leads to this result of we can mostly implement our plans so the imperfect machine of plans is the state of affairs the state of coordination the research program or the activity of coordination of coordination the coordinating activity is rooted in something like the research program and again I don't want to be any of us to be hung up on the Lakatosian framework here but this is just one way of conceiving it let me end up by according from you from a book on the natural sciences called science order and creativity David Bohm and F David Pete 1987 and it's about structure but when they use the word structure we can we can read things like order coordination those are the words that we have been using in this talk he says that they say structure is often treated as being static and more or less complete in itself but a much deeper question is that of how this structure originates and grows how it is sustained how it is finally dissolved structure is basically dynamic and should perhaps be better called structuring while relatively stable products of this process are structures so structuring creates structures but even these later ladder structures should not be considered as basically static for they are the results of processes which sustain them for a time more or less within certain limits so the structures are not deterministic but more or less just what I was getting at more or less coordination as a result as with order so which structure there can be no complete definition rather to put it again now this is another one of the eastern things I'm invoking whatever we say whatever we say structure is it isn't alright there is always something more than what we say and something different well that's the research program right you can't exactly define the research program take something that's neoclassical economics can you exactly define what it is the problems that Jerry and I were having I think just go to validate our theories how's that for seed your views at everything but I think it's true because the research program couldn't give rise to new knowledge unless it were incompletely determined so this so this structure or this structuring whenever we say structuring whatever we say structuring is the way he's using the word structure here is in the sense of structuring whatever we say structuring is it isn't there is always something more than what we say and something different at every given stage it is possible to abstract a certain structure as relevant and appropriate but later as the context is made broader the limits of validity of this abstraction are seen and new notions developed a research program right as certain limits are seen you make adjustments you change what you what your goal is you change your technique it happens all the time you can't for once and for all lay down the absolute rules of learning otherwise you never really will learn anything so structuring has to be incompletely determined well back to what we said the process of coordinating is incompletely determined so the equilibrium the order that generates learning is incompletely determined and what it results in is an order as an end state or an ideal type that is incomplete in the sense that it doesn't work fully because the very process is a messy process involving shocks and disequilibration so it's really two concepts of order but I think both are really necessary for us to have a better handle on the issues of spontaneous order that we are all concerned about today thank you


A lot of empty chambers here. Oh, guess time to start. All right, so thanks everyone for coming. It's a pretty big room for the size of the audience here. But I guess that should be OK. So my name is Michael A. And I'm here to talk about how we automated the process of creating a Spark cluster that is secure and that is HIPAA enabled. So this work was done at IBM in collaboration with some of my colleagues, notably J-Rom, Shu, as well as Daniel Dean. OK, so it's probably not too much of a surprise for most people in this room here that there's a lot of information that's personal, that's being collected, stored, and processed by various companies. Some of this information can be in a form of health information, like our level of glucose, our heart rate, some results from medical labs, and so forth, as well as what disease we have and medication we take. And besides just health information, there's also financial information, like how much money we have in the bank or how little money we have in the bank, as well as the different investment portfolios, and so forth. So in order to make use of this information and generate values from this information, companies usually will deploy analytics back and in order to process information, either real time and also to use this information along with the information that they've amassed throughout the years in order to generate some values and some prediction from this information. So in a lot of these cases, what you have is this back end that needs to process information. A lot of the information can be very private to us. And the problem with that is that even though it can generate a lot of value, both for a company in order to create interesting applications, it also can prove to be a security concern. So a lot of organizations have or governmental regulations have been created in order to protect this data. Some of these regulations include HIPAA or its extension to HIPAA, the high tech act, as well as that's sort of in the health industry. Also in the financial industry, there's this PCI and various other type of regulations. And a lot of these regulations try to do is protect the privacy of the data that's being generated as well as to govern how this data can be accessed by others. A lot of these regulations also put into effect these rules in order to help companies and regulators audit how does data access. So it's just in case that there's data breaches, information can be obtained in order to figure out what data were leaked and how is that leaked cause and who accessed that data last. So all that's really important for the auditing capability of the system. So what I really want to talk about today is really a combination of two things, how regulation like HIPAA affects things of how analytics back end are designed and architected. So as developers and administrator, when you create these back ends for analytics purposes, you have to take into account a lot of security concern or regulation concerns in order to build out a platform that is compliant to these regulations. So building it out manually is actually very challenging. And there's essentially three reasons I listed up in the slides here.  And being able to translate these regulations into specific security mechanism to implement is actually pretty challenging because it's not really clear how to map one to one what these general regulations mean and try to do to a particular mechanism in your system. So that's one aspect. The second aspect is that even though you have these mechanisms, there are actually, for mechanisms, there's actually a lot of options, a lot of things and knobs you can tune that will affect how the mechanism will act and behave. So there's a lot of these things like different type of user account, how to maintain access control, a lot of key certificates, databases that has to be maintained in order to actually maintain a full secure environment. And the third aspect is not really the aspect of the environment itself, but it is more like ops environment or ops capability. A lot of time when you build out this system and to deploy the secure platform for other people to use, and you're maybe within your own company to test out, building it manually is not very effective since maybe some different groups who also want this platform want to test out their particular solution against this back end. You have to then manually redeploy all this thing, which can be really time consuming and actually can lead to a lot of failures. So you want to be able to have a mechanism that can automate this process for you.  So the approach we took and what we did was we leveraged OpenStack Sahara to give us the capability of automating and configuring the back end platform. So the crux of the solution here is to expose secure building blocks that users can, or administrators can choose and select in order to build out their security platform. Given this ability, you can then automate the process of creating a back end that is, or a Netflix back end that is compliant to some regulation. So what we did was we prototyped this mechanism in Sahara by creating a spark cluster for our analytics purposes, running on top of Yarn and you see why later, why we chose Yarn because some of the security aspect of it. And also in this talk, I'll be just focusing on the HIPAA aspect. I know there's a lot of regulation out there and if you have the secure building blocks, hopefully the idea is that you can mix and match different security mechanism in order to make your solution applicable to a particular regulation. Okay, so this is kind of outlined in the talk. First, I'm gonna give a brief overview of what HIPAA is, this is what we're talking about and then discuss briefly the design architecture of our secure spark platform and then I'll say how those mechanisms that we implemented maps into HIPAA.  But then I wanna talk about some of the Sahara features that we've augmented and extended in order to help us with this process. And I'll summarize and give you guys some lessons in future erections, I think. The Sahara project will, or the Sahara project can take on that could be very useful for the rest of us. So HIPAA in general, for those who don't know, is essentially a health insurance Portability Accountability Act that was passed, I believe in 1996, in order to very broad stroke to help the efficiency and effectiveness of the healthcare systems in the US. So it had regulated some aspect of how data is health information is standardized, the health information record keeping and how those information is transmitted and so forth, right? It makes it easier and more efficient in some way. Also it extended to security protocols to ensure that the privacy and security and integrity of this data is maintained, right? So what data is protected here is essentially any data that's designated as protected health information or PHI that relates to medication, kind of disease you have, all information, measurements and so forth, right? All that data is protected. And who does this particularly apply to? So it applies to any entity that is providing healthcare, as well as entity that is business associated. So this is important to IT providers that consume this information, store and process information, right? So you touch this information somehow, you are now under such supervision of this particular regulation, right? So companies that do analytics on this data you sort of have to be concerned about this particular regulation, right? So in general SIP has two rules. This is privacy rule and this is security rule. So privacy rule is kind of intuitive in the sense that it helps protects to privacy to PHI. Now security rule is the one that actually operationalizes this privacy rule, right? So in general what it takes is that we need to provide confidentiality, integrity as well as availability of the data, right? And beyond that, the regulation also specifies that we have to take enough precaution to anticipate any type of, anticipate any type of potential threat to the system and potential leak of the data, right? As well as ensure that your workforce, your environment is compliant at all time, right? So high tech took that and kind of extended a little bit, adding regulation that require maintaining audit logs as well as policies to help evaluate the system as the runtime of the system continues, operationalizes the system, right? So maintain your, make sure that the appropriate level of security is maintained and so forth. So just focusing on the security rule aspects and that's the one that you can actually do something about, right? There's actually three aspects. There's administrative, physical, and a technical safeguard. So I'm not really gonna talk about the other aspects here in terms of administration and the physical part. I'm just really gonna focus on the technical part since that's something that we can do in software, right? So that's why I titled this talk as not HIPAA compliant analytics platform. It's really HIPAA enabled. Since it be compliant, you have to take into account many other aspects, not just the technical aspect, right? There's administration aspect too. So within technical safeguards, right? There are four things, this access control, audit control, integrity control, as well as transmission security. So you see later on when I describe the platform and I will go through the different mechanisms that were used in different levels and I'll say how those levels then actually match up to the different control mechanisms that is specified in the regulation. So let me just pivot a little bit here that I just kind of really briefly went through what HIPAA is and let me pivot a little bit, describe essentially the analytics platform that we've developed and put it into a particular context so you understand why we built this platform the way it is built and then give you the mapping between that platform and how HIPAA's regulated, right? So the goal of this platform we want to build out is very general in the sense that we want to be able to plug this Spark Analytics platform pretty much as a service to any, as a piece of the entire platform that processes protected health information, right? So this is an analytics backend for that piece. So there's a lot of other pieces like storage, data lake that has to be there, there's a lot of authentication mechanism that has to be in place to enter into system but at some point as data will flow and hit the analytics and analytics engine will have to pick this up and process this very protected data, right? So this piece is sort of an independent piece we want to build out, it's very secure in a sense that it has maybe potentially more mechanism you need to be HIPAA enabled but it helps the developer in the sense that you take this piece out and plug it into some other platform that requires this particular security, you have it there, right? So that's also why I think the ability to build building blocks, small pieces of security mechanism to be able to plugable into your system is helpful in a sense that if you have a different platform, you different assumption about how security works, you can either remove or add in different mechanism as you go, right? So that could be a very useful thing there. So there's some of the assumptions I've listed here is this, right? This system has essentially two types of user. One is an administrator. So the administrator can go in and create, using Sahara, stand up this platform that is the analytics platform that's secure and then there's the user of the platform which is essentially the user that just goes and make use of the cluster or the analytics part, right? He or she doesn't need to know what are the mechanisms underneath it. He or she just submits a job and runs the analytics on this data and this whole process will be secure and the different user will be isolated from each other, right? We assume that given one analytics cluster, there are different users in this cluster, right? So there could be like one hospital but many doctors in that hospital can use this analytics cluster to process information for different patient and each of the doctors can maybe have different access control. Maybe he can access, Dr. A can only access a group of patient, you know, Y and Dr. B access patient Z but not the other way around for instance, right? So there's different isolation mechanism in place for the users in the Spark cluster as well. So we also assume that there's one essentially yarn cluster that runs Spark per tenant. So you have this different hospital come in, we're gonna spin them up a different cluster that's separate from the previous cluster, right? So different, it's not multi-tenant in that way. Even though the definition of tendency can be, I know, different for different people but that's how we stayed in our particular use case. So any data that's external to the cluster, if you have a data lake that stores information about patients, we assume that there's a driver that can securely ingest that data in. So essentially that needs to be a driver's implemented in Spark or you can be leveraged by Spark to pull data in from a lake. And then from that point on, our system takes over to make sure those data when analyzed is protected and is compliant to the regulation, okay? All right, so here's the design of the Spark service that we have. Essentially broken up to three different layers, right? The very first top layer is our Spark cluster. That's where all the Spark analytic services running, right? So the different user runs and different Spark jobs will run in isolated containers. So in yarn, they call this secure container and what that just really means is that the user, sorry, the container executing the job is executed as a user ID of the job sub-inter. So in general, sometimes you build a Spark cluster and there's essentially one user, let's say Spark, and every job is submitted to that cluster, you run as this Spark user. So that is good in some sense, but not great in when you want isolation between the different jobs, right? If you have multiple users and you need to have multiple user account that you can actually have access control across the different jobs, right? So secure container is great in that way. That gives you the ability to start containers with the process ID of the job submitter. Below that Spark cluster is actually the yarn resource manager, right? So it has different components in there which is not too important. One important component is that it has Kerberos. So there's yarn components really tightly integrated with Kerberos authentication mechanism, right? This allows you to force the authentication of different components in the yarn, different components like the host itself, the VM that is launched, that is associated with the cluster, as well as the framework that's running on top of yarn, in this case that's Spark, right? We want to ensure in an environment where you can't have some other user going into your cluster creation mechanism, create a spin up of VM and say that, hey, I'm gonna run a yarn process. Let me attach to your yarn cluster without being authenticated because then you have essentially a rogue yarn node that now can consume and participate in all the data ingestion of your original secure cluster, right? So you need authentication that way. You authenticate across machine, VM, as well as the framework running on top of that VM in order to ensure that there's isolation and security across the entire stack. So all that is run on and provisioned by the Sahara service, right? So we use Sahara to automate this process, a spinning up this cluster that has all the security mechanism in place. All the nodes are set up with all the SSL encryption, all the authentication mechanism Kerberos, and also extended to have different user account created for a particular tenant, right? All that has to be there in order for this to work. Okay, so that's sort of the layout of how our Spark cluster is designed. Now let me go back and just map some of the security mechanism I just said into this HIPAA regulation, right? So I broke that, this table sort of breaks down the different four controls that I mentioned earlier, the access control, audit control, integrity as well as transmission and security, right? So it's probably not mind-blowing in any way, but you can see the mechanisms that I've listed in terms of access control, for instance, right? There's Kerberos for authentication, the different components, there's secure containers for isolation of different users, right? There's HDFS file permissions you need to when you use write and store data for a particular user, I need that, and HDFS security zones in order to encrypt the data at rest, right? So encryption happens both on the disk level when data are stored, as well as during transmission, right? You need to have that encrypted as well when you're transmitting because Spark does shuffling so it does sending data across different nodes for processing you. That path also needs to be encrypted, and any data that is being generated, cache and so forth, those also need to be outputted to directories that has encryption on it to make sure that no data is ever exposed, right? So I sort of skipped the path of what happens to data when it's in memory, now it's unencrypted, right?  You can have mechanism that also does encryption in memory as well, right? So often there's a penalty in performance, but you could do that as you want to. And actually that goes back to my point that there's all these different mechanisms, and the regulation like HIPAA never specifies exactly how to actually implement this thing, right? So it just tells you that you need to make sure that these things are really secure and maintain isolation, but then tell you actually how to do it. And you can be paranoid as you want. So having mechanism like Sahara that we built as a heart allows you to essentially select different levels of security. It can be a very useful thing to help you deploy your system in a way that you can test out the functionality to compare the performance of those things versus the security level you get, right? So that can be a very useful aspect. I won't actually go through the rest because some of it's kind of intuitive. I want to point out is in terms of audit controls, right? We have the ability out there to already log what type of job is submitted, what type of files are accessed, but one thing is that the framework accessing the data, right? In this sense it's Spark. Injusting the data to process, Spark itself right now doesn't have mechanism to log what data did it ingest, what RDD did it form, what piece of data it touched, and when that piece of data is written out to disk for like caching and so forth. None of that is log, right? So what we did in IBM Research, we actually extended Spark to create a logging mechanism that allows us to actually log the pieces of data that is read on HDFS that it can log and send to the auditor so that it can also be part of the logging mechanism that is there, right? So to make sure that the whole process is, every piece of data's access is logged, essentially what I want to say. Okay, so let me now touch upon the Sahara piece that we've implemented in order to make this possible, right? So there's essentially four pieces of work here. One is automating the security and enablement, right? That's one piece. There's also a piece that allows extension of Sahara to enable adding essentially different users to the cluster, right? So that's not quite there yet, and it's also debatable whether this belongs to the Sahara API or not, but we extended API to give us this ability. And third and fourth thing is the ability to submit different type of jobs given one particular yarn cluster. So I know this exists in the Cloudera version in Sahara, but the vanilla version of Sahara running like just vanilla Hadoop or yarn, right now it doesn't have the ability to submit both like a Spark job as well as a Hadoop job very easily. You can do one or not the other. So now then I'll summarize some of these lessons. So just in case people are not familiar with how Sahara works or how to spin up a cluster that is using Sahara, right? So Sahara is done in a VM image-based way.  You do that out of band by some administrator. You upload it to something like Glance. Once you have that, then you use a Sahara interface and to define what your cluster looks like. I wanted to say a cluster of let's say five nodes. Each node has certain flavors of VMs, certain size, what process I want to run in those VMs. And then once you have that particular template, then you can go to Sahara to deploy in provision, right? So the provisioning engine of Sahara essentially just leverages heat, right? So it generates a heat template and uses heat to then create and spin up the VMs for you, right? And once that is done, the heat talks with different components of OpenStack, Nova, Neutronic, Glance and other things to create those VMs. So once that is done, there's the last step where Sahara actually comes in play again is that it will SSH into each of those VMs and then configure the VMs in a way that is appropriate for your particular deployment. So if you want to run Hadoop, it goes in there and does some configuration settings and changes and then it launches the Hadoop daemons to then finish the process, right? So in some sense, it's not very clear in terms of how our API, in terms of how much do you want to bake into the images and how much you want to let it configuration at runtime, the configuration take over. You can actually SSH in, then do the pool to get the binaries and then configure, right? So there's actually no clean way of how to set it up so you have to use your judgment in terms of how much you want to pre-create and how much you want to do it at runtime. So this is the proof of concept that we've... I want to show you some of the screenshots of how our system looks like and what changes we made into the gooey part of the Sahara to instantiate the different configuration mechanisms that I mentioned before, right? So this is a familiar horizon interface. We stripped out everything except the Sahara interface. So you have your typical clusters, your cluster template, no group, job submission, and so forth, right? So those things haven't really changed. So what we've added was a couple of things. One is when you create your no group template that we've added a few more services here that pertains to security, right? So there's the Kerberos that you can instantiate. There's the KMS, which is the key management system in Hadoop, right? So that's useful for HTFS encryption. We've added something that's not related to security but it's very useful service I think in general when you wanna run a kind of analytics platform, right? There's something called IPython for interactive query with Spark. There's the Spark job server was an external project to Spark that allows you to submit jobs using REST APIs to Spark itself rather than going through the Sahara interface. I'll say why we use that because Sahara interface API is missing one crucial thing that we needed that wasn't supported. I think that's something that we can ask the community to interested in adding. The other aspect of this is more security aspect which is when you create a cluster you can select different types of security modes. So right now we only essentially have two, right? We have HTF encryption, you can enable that or you can enable both the secure mode of yarn which is all the authentication maximum turn on. Now that's between the different components of yarn as well as authentication with the framework and set up as an encryption between the different components of yarn as well as the components of the framework in this case Spark and that's enable, make sure the shuffling is encrypted so all that gets turned on by just that click. So once you click that option when you deploy your cluster the full mechanism of data at REST encryption as well as encryption during transmission is enabled, right? There's also a Kerberos server location here so I'll bring up a little bit later why there's a particular field for that because there's a different way you can design how Kerberos server should be run, right? There's you can place it within your cluster you create so you have a Kerberos server per data analytics Spark cluster or you have one that can be shared by everybody, right? So there's con and pros for each of that scenario. So the last thing that we've added was essentially extra credentials because what we did was we extended Sahar to not only instantiate a cluster on top of an open stack cloud but we also extended to use other cloud in this case, software. So then we obviously need credentials for the software account, right? So that's essentially what this is. So since I mentioned that we used Sahar to instantiate on software there's a lot of components that we make use in open stack but also some components that we don't have to use anymore. So this is only pertaining to our particular proof of concept, right? Not necessarily general to everybody. So we make use of essentially Sahar horizon to Keystone we don't actually make use of any of the Nova and Glance because the images are now uploaded to the software's image repository and then we use the software's API to instantiate the VMs on software account. So from a Sahara portal we define what a cluster looks like when we click launch it actually goes to software's account and creates the VMs in software. So now you have Sahara essentially instantiating configuring VMs and cluster of VMs on top of software. So this is the flow that I showed earlier how Sahara works and the only thing we've changed is the one I highlighted in slightly orangeish color there. So instead of the heat template talking to different open stack components now we have a new heat resource plugin that we use that will then instantiate VMs on software by going through the software Python binding, right? So that's the only piece that we changed that allows us to now create and launch VMs on software instead of open stack. So the process of setting Sahara up to do what I've just said is pretty much follows the normal route of booting anything in Sahara except for the four, five and six. So this is what we had to do to change Sahara's internal code in order to enable the configuration and the deployment of security functions in Yarn and Spark, right? So we have to change how when the cluster comes up we have the SH into these machines and we have to configure the different configuration properties to make sure that we enable some of these feature, right? So that's pretty straightforward. Maintaining the Kerberos, the keys and SL certificate is also that we have to do as well as creating user account and setting up the different HDFS directories for each user is also automated in this whole process, right? So in terms of the image that we prepared what we've ended up doing is that we've baked these particular binaries into the image, right? So we have Hadoop, Spark, the Kerberos engine as well as the two SparkJob server as well as Jupyter or this IPython thing. So that's essentially the type of binaries we have in our image, right? So given those image we can then spin up the cluster and then SSH into the machine using Sahara to then configure all the different options that we want, right? So right now this image creation is very manual. I sort of have to do this manually. There isn't a disk image builder that you can extend to do this, but this works mainly, this image builder is useful for the OpenStack creation process. To automate this for like software, you need to add some extension in order to make it automatically push it up to the software account that you want to use. Because we set up our cloud on the software account rather than OpenStack. So one aspect I want to touch upon is that a lot of this security mechanism is leveraged on how well you can authenticate your user, right? So Kerberos is a very central key to this piece, right? So this essentially shows you what housing security and Kerberos work, but I just want to point out one particular thing that has to do with how do you manage when you're spitting up clusters and you have to interact with Kerberos. And that is how do you manage authentication with Kerberos, right? So normally as a user or a human being you can type in your credentials as you log into the Kerberos system with processes and things like that. It uses what is called a key tab. That's essentially an encrypted key or file that is associated with that principle's password. So that essentially is a file that is stored somewhere and that process has access to that file. And that file essentially the key that you say here Kerberos, I am this person that I say I am, and then Kerberos will give you a key that goes through some exchanges to then give you the credentials to access that service you want, right?  And this key tab essentially is a file. So what we do now is we install this file per user in the local file system of all the nodes on that cluster, right? So that is only probably a temporary solution that's probably not good in the sense that you able to break the account of that particular user or break the account of different users you can access to your keys, right? So probably something that could be useful later on is actually use some kind of key management system where you put the key tabs in that management system separately from the cluster of analytics cluster you have. So I just want to point that out as an important piece of security that is kind of left out in this design. And also with respect to Kerberos itself also there's different ways like I mentioned earlier how you can run Kerberos server, right? So you can have Sahar controlling various cluster of analytics, right? So you have a Spark cluster that's one, two and up to N and it's controlling all these pieces, right? So now the question is what do you do with this Kerberos server? You can, you have different configuration you can actually insert it into your particular cluster so the Kerberos is only local to that cluster or you can have a separate Kerberos server that's shown that figure there and then different cluster can then access that Kerberos server, right? So this pros and cons of this thing obviously if you share it that's more efficient but then you can run into issues of collision in terms of namespace and so forth. Now you have to, there's ways to overcome that but that requires a more complicated setup in Kerberos, right? So you have a Kerberos for each cluster then you can overcome that issue but then you have the overhead of maintaining different Kerberos server. But what helps you is that if it's Sahar you don't have to deploy this manually once you set it up once, configure it once you can click a button and it will deploy multiple instances exactly the same and you know with different number of VMs and sizes as you want but once you do it once you can overcome this problem, right? So this is a little small, it's a little crowded but this essentially goes to some of the steps we took in order to automate all these setting up of the secure Spark service, right? So during creating the cluster, right? There's multiple things you have to go through. You have to do things that are per node that is setting up the Kerberos, setting up the key tabs per VM host, right? And generating the different keys for different yarn services and then pushing those out to different nodes. You have to set up the configuration to secure container that requires editing a lot of Hadoop configuration, all that's done for you automatically and then per user you have to configure also the principle for that user, right? If you have a different user they have to create a different principle the user will store them somewhere and as well as setting up the SSL security certificates and all that stuff per user and per VMs, right? And you push that out and you use Sahara mechanism you sort of have to program this once and now you have this ability to just push and play essentially. You also have to worry about, you know Sahara gives you ability to scale in and out your cluster and so you also wanna make sure that when you do this you can leverage the fact that when you scale out you're adding another node what that means is that now you have another node joining your system, this is a legitimate node so therefore you have to add the different key tabs and CA, sorry certificates for that node and update all your existing node to make sure that those existing nodes now know that this node is legitimate and that it can successfully join the cluster that you created, right? So just make sure that the security that you push out is going to be a legitimate one rather than having some random node coming up in a cloud and joining your cluster. So that is sort of an assumption we made in the sense that different nodes can join your cluster unsecurely so you have a setup where you do some kind of VLAN where you have separate network for your system that maybe that mechanism might not be too useful and you can not use that mechanism at all, right? That depends on how you're set up if you're configuring your network stack. So that's sort of enabling security, right? So there's also the aspect of adding multiple users to a Spark cluster on using, or any analytic cluster in your environment, right? So right now Sahar doesn't give you the ability to essentially stand up a cluster that has multiple users. It stands up a cluster with a single user and every job you submit to this cluster essentially run as a, I think it's a Hadoop job right now by default. So that's not very useful in terms of you wanna maintain isolation between different users of your cluster, right? So we would like to, or we have done and we would like to make this known that we extended the API to add an added user through Sahar. So this way the administrator can go in as part of the onboarding process I'm sure there's probably a lot more complicated than that but at some point you probably need to call this API to add this user into your cluster and this user can be instantiated creating different UNIX account setting up HDFS permission directories so that the new user can now be isolated from different users in your system, right? So what we did was we extended the REST API this v10 file to add a new API for adding user. So this user when you invoke this API all it really does is these three things, right? Go on to every node to create a new UNIX account, create HDFS directory account and then distribute the Kerberos key for this user to all the nodes in the system that it needs to access in order to run its Spark job, right? So one of the challenges we found while doing this was that the assumption in Sahar is that the cluster when a cluster is created there's only a single user and that's a dupe. So we have to essentially go through and modify how the assumption of Sahar code works so that we make sure that it's not hard coded and there's a single user being used. We want to make sure this is more parameterized so that it's later on be easier to extend the different user in the system. The third thing that we did here was allowing Spark jobs as well as Hadoop-based job to be submitted to a cluster. So I know there's ways to submit a Spark job already and ways to submit a Hadoop job already. Combining those two in the vanilla version of yarn isn't there, so we did something very simple, right? We didn't have to extend any API or anything but we allow the Spark jobs and Hadoop job to be executed in the same cluster by essentially just taking in what tab was submitted based on that job type in the cluster type which essentially create a string that is Spark submit string with the right appropriate parameters for submitting to yarn and submitting that string to Spark master to run. So that's not, it's a simple matter and it doesn't take too much work there and I think the implementation is pretty straightforward as well. So the last thing we did was, as I mentioned, we did run our cluster on SoftLayer and this work was actually done not with this project but with something else some years ago by different members of the group that added a heat plugin in order to allow heat to work with SoftLayer, right? So essentially added another heat resource that's a SoftLayer VM and given that resource and the Python binding to SoftLayer you can create a cluster on top of SoftLayer. And this is just a screenshot of what had to be extended in Sahara in order to make use of this feature and that is like I mentioned before once Sahara generates or provision a cluster it creates a heat template, right? So the only thing you have to change in order to make Sahara work with another cloud is essentially to change the output of this heat template to make use of the resources that a heat template is tied to in this case a SoftLayer VM. And once you do that then Sahara essentially works which can work with another cloud. Okay, so let me just quickly summarize here. So I talked about Sahara requirement and some usefulness of being able to use something like Sahara to automate configure your deployment of your system to make sure that these deployment is compatible or compliant however word you want to use to this particular regulation you have, right? And I talked about some ways to extend Sahara to make this happen. Now one interesting or a few couple of things that we've learned from doing this is that it's not always clear what security mechanism you need to employ in your system to make some aspect of your system compliant to a regulation, right? It is never clear, it's not, it doesn't list out a particular mechanism you have to use. And having ability to selectively allow your administers to select a different component security is very useful so they can try out different mechanism and test the performance versus security that they get, right? So that's a very useful thing to have. So component-sized security is very useful and being able to automate that using something like Sahara is also very useful for the community, I think. And also the one aspect we found out is kind of interesting that when you, even Sahara with the GUI, right? You can just click and point and click to define your cluster. It's actually very intimidating for a lot of users who use the system first time, right? So it's very useful to have very simple templates that maybe doesn't cover all the option but give them very simple templates to use and users can just select one of those templates to use directly, right? So even with the ability to select different options to define the template, it's actually already intimidating for a lot of users we found out. So some of the improvements that we think could be made is, first of all, is to actually identify some more of the security components that you can take out and isolate and implement that in Sahara as essentially a unit of security. You allow the people to enable that through Sahara that could be a very useful thing. We've added API to add users so we never delete any user or modify users through Sahara so we don't have any of those mechanisms. One thing that I mentioned that with the API for data analysis through Sahara or job submission through Sahara, there's no way, once you submit a job run, there's no way to actually retrieve any data or results using the Sahara API. So when you submit a job, you sort of, it's assumed that the data's outputted to HDFS or something and that's it. Their user has to go some other mechanism or other route to get this data, right? So it'd be very useful to have Sahara API to actually get this data when it finishes the job. All right, I'm gonna skip the last point there. So thank you for listening and I will take any questions you might have. So it was perfectly clear and everybody agreed what I said, that's good. Okay, so I'll be around if you guys need to want to talk and chat. I'll be happy to take your questions. Thank you.


The Venezuelan president just asked a Hong Kong reporter to speak Mandarin and this clip is going viral. Yeah, this went all around the world and back. Let's play the clip of Nicolas Maduro during a state visit to Beijing being asked a question by a Hong Kong reporter from Phoenix TV. I'm the journalist from Hong Kong, Phoenix TV. So my question is, you have mentioned... No translation. No translation. Hello, Mandarin. Oh, yeah. Why? No translation in English. Oh, yeah, how was it? The New World. Oh, yeah. Boom. Long story short, Andrew, the reporter opens up in English, right? Because he's figuring they're going to translate the English into Spanish and the guy's like, No, habla Mandarin, habla Mandarin. But then he also says, Nuevo Mundo. So the Venezuelan president is like, oh, it's a New World. Right, right, right. But I do think a lot of people are saying like, basically there wasn't a translator at the moment that was translating English to Spanish because they're in Beijing and the Beijing translator for him was translating from Mandarin questions to Spanish for him. Right. But his last comment sort of sparked a ton of controversy, at least in internet circles. All right, guys. So let's talk about what the Venezuelan president meant by this and what people are discussing. Please hit that like button and check out other episodes of the Hot Pop Boys right after this. Hey, Andrew, from Silly to Serious, I'd say it's a silly clip. Maduro is laughing. Everybody's laughing. But in terms of how the internet took it, I'd say they took it pretty serious. Yeah. But you know what else is serious, Andrew? The flavor that you're going to get from adding Smala to any noodles, pastas, rice, dumplings, get it at SmalaSauce.com. Andrew, real truffle. Yeah, made with real truffle, made in America. Pre-orders are still going on right now, guys, but we're very excited to ship it out soon. So please check out SmalaSauce.com. Let's just get into some quick thoughts, Andrew, because this could get very murky. This could get very muddy. Like we said, sort of any sort of talk, Andrew, of changing of the world order. What are the dominant empires and ultra bird's eye view in the world? It gets people emotional, right? Yeah, and I think at the end of the day, listen, a lot of people are writing for their team. You know, if you are born and raised in America, you're probably going to be more pro-American and you're going to be listening to the pro-American news. If you're from over there, if you're in China, maybe you're going to be more proud that China's coming up. But I'll tell you this, both places have problems, okay? Everybody in the world is having problems.  I'm not going to say I'm an expert on it, but I just know that. Everybody's got problems. Everybody's got problems in their country.  Amongst individual citizens on the internet, right? I mean, let's just put it out there. Before we get into our quick thoughts and the comment section, essentially what the Venezuelan president was kind of getting at and hinting at kind of jokingly was saying, hey, it's a new world. Like, yeah, speak Mandarin, not English because more people are teaming up with China. Look, I'm in China right now. So, like, you know, maybe the world is moving away from teaming up with America and Britain and stuff. Guys, this is what it is. And obviously that made a lot of American people or Western world people feel some type of way by him even saying that. Right, right. Because it sounds like an anti, like, I guess, anti-Anglo-dominator or English-dominated world statement. Right, right, right. But people are saying, oh, how much was it? Was it just like a casual joke? Was it this? Was it that? Like we said, there is a smattering of opinions. Real quick, Andrew, is it really slipping on a global stage? I would say this. I would say this. It feels like- What is slipping? Like American dominance or Anglo-dominance. I would say it feels like the lead is shortening, but it's still like definitely number one. Yeah, I think it's very hard to tell the truth on what exactly is happening because you'd have to factor in probably 100 different factors. You'd probably have to talk to some experts from different industries and different sides of the world. But I think anecdotally, obviously a lot of people are saying, yeah, America's slipping. Right, right. And this is the whole concept of Ray Dalio's book, right? The changing world order and things like that. I do think the Venezuelan president, he is very cavalier and sort of trying to make cool statements. He seems like a guy who wants to be cool. Yeah. I mean, is it fair to that a lot of people question the Venezuelan president and being like, well, who is this guy? Like his country's got a ton of problems and he's not known to be a good leader. So who cares what he says? Right, right. If you guys know about Venezuela, it's like got like hyperinflation right now. It's crazy. Like a lot of people are trying to leave and come to places like America, et cetera, if they can get out. I think a lot of people, they don't want a dual polar and they don't want a multipolar world, right? I think obviously since the UK and since the US, the world has been more unipolar and definitely I can understand the fear of changing that order, right? Well, real quick, what does it mean to be unipolar? Basically, it means that there's like a pyramid of like power distribution hierarchy. Geopolitically, this is like super bird's eye in America was the only country at the very, very top of the pyramid. Yeah. So you're saying that the system was simpler when America dominated the globe essentially. And now that America is probably still dominates the globe, but now is losing some steam. That now they're going to have to share a space a little bit more with China, making things more complicated because it's less complicated when one person rules. Yes, yes, yes. But it's obviously it's not just China. It's India. It's any country, you know what I mean? That wants to be more in that top power. But I guess you would say that maybe like China is the one that everybody thinks about, right? They're like the mast of that ship, right? And I think it really also brings on a way more micro level, Andrew. It just makes you think, man, it's kind of complicated to be Chinese American, you know?  I don't know. They kind of deserve a chance to, but then you see how they're doing it and you're like, yeah, I can see why everybody doesn't like it. It's complicated. Yeah.  You're America. America is not a race. That's a nationality. And I'm like, yeah, of course, for sure, you know, like I'm American. I'm not moving to China anytime soon. But what I'm saying is like, I guess I'd like to think that, you know, our motherland could come up as well without it tearing things apart. Yeah. Why does it have to be so stressful, right? Right. Right. For sure. For sure. And like we said, guys, I mean, everybody's going to have their own opinion about it. I'm sure the comment section is going to be crazy. But yeah, I mean, definitely, I wish it wasn't going how it's going though with the whole like beef and the rivalry. Anyway, let's just get into the comment section, guys. Again, make sure you like this video. Somebody said, I'm sure Maduro had an interpreter to translate the Mandarin in Spanish so he could understand this situation just happened and it got blown out of proportion like it was meant to be this gigantic like viral moment. No, I think it's legit that they didn't have a translator because a woman comes on and says in Chinese, no translator. Yeah. She's like male finding. Yeah. So that's that's not him saying he just played into the moment, but literally they didn't have the translator on hand at that moment. I'm sure they have the translator in general, but because he's from a Spanish country, they probably didn't expect any reporters to ask questions in full English. And then, of course, Andrew, there was a bunch of comments saying, yeah, but they mostly speak Cantonese in Hong Kong.  But then, of course, Andrew, they were in Beijing. He's not going to ask a question in Cantonese in Beijing to get translated into Spanish. Right. Right. Yeah. Um, somebody said, why doesn't someone ask Maduro? Why his people are fleeing Venezuela and droves for the U.S. Him and his predecessor Chavez didn't do a good job in governing. They switched it to a socialist country and now they got hyperinflation and everybody wants out. So basically someone was like, who cares what this guy thinks this guy got his country going bad, right? Yeah. Right. Right. Yeah. I mean, it's true. Right. I guess people kind of like would judge the statements of a leader of like if it was France way differently, right? Um, somebody said, you know, one of the major things that China has to do is that it doesn't have to deal with Venezuelan refugees or any sort of refugee situation. Um, they, whereas America does and they're talking about like, I guess when two countries are trying to be competitive, the U.S. does get a lot of refugees, whereas China essentially gets none. Right. Makes sense. Yeah. From a functional perspective, somebody just said, um, yeah, like, of course, when a nation becomes more prosperous, many people want to, um, migrate to prosperous countries. Right.  Most people want to move here. Yes. Most people want to move and stay here. Yes. I agree with that. And I just think that how we manage that diversity is just, I think we just got to do a better job of it. I think we did do a good job in the past, but maybe the way we were running the scheme back then, we got to think of a new scheme and we got to get everybody on the same page. That's not going to happen. Everybody's just gridlock trying to fight each other over the system changes. Right. Right. Um, somebody said, man, if you guys really understand it's all the sanctions and the US State Department always meddling in Central and South America, it bungled everything. Now, of course, now they all want out to come to America. Right. Yeah. I mean, this is what a major criticism of American foreign policy is, right? So, so, so some people are criticizing saying that so many people from Venezuela want to flee Venezuela and come to America, but then here's the president of Venezuela kind of sort of taking a jab at America. Right. So, because, uh, they, I guess they got a big oil deal to go to China, right? Um, somebody said, we are moving to a multipolar world and that's a very good thing. Somebody said, yes, I'm fine with that, but I'm ultimately in the minority because I'm not scared of seeing the possibilities but changes constant and those who remain in charge are more ignorant than necessary. So basically, um, yeah, somebody was just saying, don't be scared of the changing world because this always happens. Uh, that is sort of Ray Dalio stance, but I could see a lot of people, they are scared. I guess anytime there is change, there's usually some conflict. I don't know what that looks like nowadays. You know what I mean? But that's what people are probably scared of. I hope not. I hope that there's not conflict. Somebody said, the world is choosing sides. I had hoped that my kids would be able to live in a world of peace, but it doesn't look like they will get to. And someone said, indeed, waves of lunacy repeat. Andrew, this is a crazy comment to emerge from just the Venezuelan president saying Habla Mandarin, right? Yeah. It goes to show you that I felt like this topic was really on people's minds. And then this is just like almost like a, it's almost like an emotional visual incident to draw it out. Right? I guess to see a president of another country, no matter what the country is, say something like this. Uh, yeah, it is, it is kind of like visceral. Like it hits people. They're like, Oh man, what now? It's not just the internet conversation. It's not just a conversation between economists. It's like presidents are saying this now. Right. Like it's so matter of fact. Right. Even though technically it's the president of a country that is not doing so well right now. Um, somebody was saying, you know, the reason why America is going down is because we're divided internally because we, uh, China and Russia got a hold of the tick tock algorithm and they're dividing people through identities. So we can't get anything done anymore. Yeah. That's ridiculous, man. I mean, that is on tick tock in China being controlled, man. This, this, that's like not the thing. I, and then somebody said, no, listen, guys, I'm a proud American and Americans always fight internally and we get divided just cause that's what we like to do because we get bored with how good everything is. But then once there's a foreigner that becomes strong enough, we band together and then we remove that threat and then we go back to our normal routine of just fighting each other again. I thought that that was a really funny break down. Um, somebody was just saying, Hey guys, if you speak English, Mandarin, Chinese and Spanish, you could probably communicate with about 85% of the entire world. That's actually true. Andrew, the next biggest languages are Hindi, Arabic and French, but of course a lot of people in India actually speak English as well. Um, somebody said choosing to support China as its economy is crumbling is smart, huh? And then somebody was saying, uh, yeah, it has to do with ever grand Andrew and aging population. A lot of the kids in China right now cannot find work and this kind of goes to like another thing that's really popular in the media right now where a lot of people are saying that China is never going to reach that one B status because of things that are out of its control.  It's never actually going to break into the top tier because it has a lot of a stifling issues right now that it can not control. Right. And I honestly, there's actually some truth to that too. I don't know how much I see what they mean, but does China need to be one B to even still challenge America? Not really. No, I don't know what does it mean to be one B? Who cares about the ranking or who doesn't care? Like, does that make you feel better if they're not one B or are you are you safe or like, Oh, America one of its one C for for China for one one China policy. Yeah. All right, Andrew, let's just get into our takeaways. Obviously, this is a hot geopolitical topic or people are going to be like, Oh, there's a dad because people feel so emotional about these type of things. One, you know, the thing that was interesting and real quick. This is a silly thought Andrew. The president's name is Maduro and they also eat Maduro's in Venezuela, which is like a fried plantain. Shout out. I've had a Patagon piece out over in New York. They have like where it's like a sandwich, but sandwich between mashed plantains together. So that's like the bun. Why do you think such an innocuous clip just sparks like this whole world of discussions? Right? I think there's some people love talking about on the Internet, man, and people feel so many different types of ways. And I think a lot of you see a lot of people on the Internet think they understand the situation. I don't try to pretend I have all the exact numbers, right? Everybody just generally is like, there's like a general consensus of what's going on, but you don't really know unless you actually study this stuff like in and out. Right, right. And then even then you might just have like a biased perspective. I think the truth is nobody really knows. My best guess is this, Andrew, I do not think that China is ever going to take over the world like the UK and America did. Like I think English is going to continue to be the predominant language globally. It's going to even be more of a lingua franca as it continues. English is a dope language. Actually, it's my favorite language. It's very flexible. It incorporates foreign vocabulary very well if you guys actually study the structure of English. I think the China's rise to number two, Andrew, if it does get there, it's going to be more like Skechers to Nike. Did you know that there's a few years, Andrew, where Skechers actually outsells Nike? Like the number one selling shoe model that year is a Skechers model, right? Like it surpassed New Balance, A6, Adidas, all these like much cooler brands. Nike's still dominant and Nike's not only dominant from a bird's-eye perspective, but also from a cool level. Skechers, Andrew, not cool. So Skechers sometimes, depending on the year, outsells Nike in revenue, but nobody really thinks about Skechers and nobody thinks Skechers are cool. But if you think about it, all the workers, nurses, a lot of older people, a lot of people where Skechers, even though you don't think Skechers is cool. Yeah, anytime you see a shoe and you don't know what it is and it's just like servers at a restaurant, there's a chance that those are Skechers. Right, right. So that's why I'm always saying like people are like, no, they'll never be another Nike. I'm like, nah, because Nike's dominant and cool, but you could just have a really big brand like Skechers. Right. So China might dominate in a way, but will never be that cool or on a kind of like soft power level. Yeah, that's what I honestly think will probably happen. Yeah. So I don't know if people want to be terrified of that. Yeah, I mean, I think a lot of people say think dominance, that means a bunch of Chinese people are going to start marching into every city and like taking things over and like taking your business. I'd like, dude, it's not talking about. It's just, I think in 2023, it's more like macro economic ownership, to be honest. I mean, ultimately, I think the thing is it might never have like the peaks of the US or the UK, but it might also not have as many like wars or like drug or gun issues too. Right. Yeah. I mean, it makes sense guys. It's just going to look different from the East from the West. Let us know what you guys think of the comment section below. Was Nicholas Maduro wild and out in Beijing? Who knows what? Hey guys, and what do you think about this quote? And how do you guys feel about it? Let us know about the multipolar world and what your opinions are. So please let us know in the comments down below. Yeah, we encourage debate in the comment section. Keep it civil, guys. Try to use facts until next time. We out. Peace.