Once again, it's a pleasure to welcome you all to MSP lecture series on Interpretative Spectroscopy. In my previous lectures, I started discussing about NMR spectra of various phosphorus containing compounds and also how to analyze if you get more than one product in a reaction. For example, if you get cis and trans isomers or if you get some other isomers, we should be able to elucidate and understand the structures of those molecules by simply applying phosphorus NMR spectroscopy. So, let us continue from where I had stopped. Let me take another interesting reaction here. This reaction, you can see I have used two entities. One is cyclopentadienyl bis-tryphenylphosphine chloro-ruthanium 2 plus that is reacted with bis-tryphenylphosphine amine ligand. So, that means in this reaction, although we know that when you are mixing in a no-starch geometry, we are supposed to get one product, but on the other hand due to some reactivity difference that we gauge and that is what happens later, we end up getting more than one product. So, that means we may get a mixture of products as well. In that case, if we analyze by looking into the 2NPNMR spectrum of that reaction mixture that can tell you lot more details and also that can also guide you how to plan our reaction by altering reaction conditions. For example, in this one as I said, this ruthanium 2 compound was treated with this bisphosphine. In the above reaction I have shown here, we are getting 3 products 4, 5 and 6 in different stoichiometrics although the reaction is carried out in 1 is to 1 ratio, the expected product is 4, but how other 2 compounds are formed and how did we got this information that one can see by simply looking into the 3NPNMR spectrum of the reaction mixture here. So, these are the 2-triphenyl chemical shifts are there, but if you recall the first compound 4 where 2-triphenyl phosphines are replaced by bisphosphine. So, here both the phosphorous environments are identical as a result one can expect a single resonance here that means this is assigned to the product 4. Now let us look into other products. Let us go back to the reaction. In this product 6, one triphenylphosphine is there and one bisphosphine is there, bisphosphine chemical environments are identical and we have PPH3. So, by simple thumb rule by looking into 2Ni plus 1 rule, we can anticipate a triplet for triphenylphosphine and a doublet for bisphosphine because bisphosphine will be coupled with both of them together coupled with triphenylphosphine whereas triphenylphosphine will be coupling with the 2-equivalent phosphorous centers from bisphosphine. So, we expect a triplet and a doublet for 6. So, you can see here a doublet is there and a triplet is there this is what triphenylphosphine and this is for chelated 2 phosphorous mites of bisphenylphosphine I mean. So, then we have left with one here, one here, one here for product 5. So, what is that product 5? So, product 5 you see we have one dangling phosphine is there acting as a monodentate ligand. So, this is different and this is different and these two are identical. So, that means, we are anticipating 3 signals and here this phosphine is further away from this phosphine as a result this can just couple with coordinated phosphorous to show a doublet, but on the other hand the chelated phosphines also can couple with coordinated phosphorous from the monodentate ligand to show a doublet whereas, this one would show a triplet with this one and a triplet is further split into doublets from this one. That means, we are seeing a a multiplied for mono coordinated phosphorous and a doublet for biscoordinated or chelate compound and then again doublet for this one. So, let us see whether we have those things here yes we have a doublet and we have a doublet with very small coupling constant value and also we have a multiplet here it is a triplet of doublets. So, we are seeing that one so that means, this is giving some idea that three compounds are formed, but again looking into the integration we can gauge how much quantity of each one is formed in what ratio and then looking into other two compounds where ionic products are there yes that means, probably instead of using let us say I have used dichloromethane here instead of using dichloromethane if I use a polar solvent probably the formation of four can be diminished and formation of these two compounds can be enhancer yields on the other hand again by taking two stoichiometry two equivalents of bisdefinifene or phosphine probably I can enhance the possibility of formation of 5 even more. So, this vital information comes for using those information so this reaction was carried out and this pure compound was obtained carrying out at low temperature and then by using two equivalents of this phosphine in polar solvent resulted in this compound and then using a polar solvent, but one equivalent of bisdefinifene and phosphine resulted in this compound of course, when once after making this compound if you bubble CO here this can form a carbonyl compound. So, this all vital information comes very nicely by analyzing the 31 PNMR spectrum of the mixture of this reaction product that means it is not just to analyze and understand what kind of compounds formed it can certainly help us in altering our reaction conditions so that we can refine our methodology and later all these compounds were prepared in pure form and analyzed you can see here so these compounds are formed and then this compound was again prepared in a separate reaction by getting that information from the reaction mixture and then this was obtained in pure form. And now let us look into few more examples here I have given three examples here one is a bisphosphine with both the phosphorus oxidized by selenium that is bis selenide derivative here and then I have here analogous to Vasca's compound both cis and trans chlorocarbonyl bisdefinyl phosphine bis triphenyl phosphine rhodium and also the its cis analog and they they differ significantly in their 31 PNMR spectra. First let us look into bisphosphine here of course when we look into bisphosphine with both the phosphorus oxidized by selenium here 31 PNMR spectrum would show a singlet and then it is coupled to selenium 77 selenium and that shows satellites here separation is called phosphorus selenium coupling constant so these are called satellite peaks and then if you look into selenium NMR again both the selenium centress are chemically and magnetically equivalent as a result they are equally coupled to phosphorus and we will see just a doublet and then this whatever the separation is there here in case of selenium NMR should be same as what we obtained in case of 31 PNMR that means coupling constants remain same no matter which nuclei you are using. Now let us look into trans compound here in case of trans compound you can see both the phosphorus are identical by performing a C2 axis of rotation passing through C1 CL as a result you can see only one type of phosphorus and when you look into 31 PNMR they are simply coupled to rhodium to give a doublet here something like the very simple doublet is here and then this separation is called 1JRHP but when you look into this one here one triphenyl phosphino moiety here it is triphenyl phosphine one triphenyl phosphine is trans to CO whereas the other one is cis to CO so we cannot have C2 axis of rotation as a result two phosphorus atoms are chemically equivalent but they are both chemically and magnetically non-equivalent as a result it resembles AMX spin system so PA is coupled with rhodium and then coupled with phosphorus it shows a doublet of doublet in the same way PP is also coupled first with rhodium and each line is further coupled with phosphorus so first this is rhodium coupling and then this is phosphorus coupling whether you consider PA, delta of PA or delta of PB they are very similar but slight marginal difference is there in chemical shifts and coupling constants so this is rhodium phosphorus coupling and then here it is 2JPP coupling so this is how we can distinguish cis and trans isomers in case if they are obtained in a reaction let us look into two more examples here and we have here we have a mixed ligand complex platinum zero a tetrahedral complex where we have two trimethyl phosphine and two diphenyl phosphine so that means platinum is in plus two state here two trimethyl phosphines are neutral ligands and here we have two diphenyl phosphines phosphides are there they are anionic so platinum is in plus two state and it is a 16 electron complex so now if you look into phosphorus NMR spectrum we can anticipate two type of signals so these two are identical they are coupled with these two to give a triplet these two will couple with equally to this one give a triplet and then these two will couple with this one to give a triplet so what we should get is a triplet something like this and then platinum coupling would come so basically platinum coupling would come with platinum satellites because of 195 platinum is about 34 percent NMR activity equals half so as a result this is for 196 platinum which is i equals 0 so then what we get is this is split into a doublet here something like this and then something like this so if we look into the spectrum it should look like this distance is called 1j ptp coupling this is similar for both of them this is similar for both of them whether we consider a signal for trimethyl phosphine or we look into that one they'll be identical but they'll be having different chemical shifts here this one is for 196 platinum and then if you look into the integration this would account for 66 percent and then this would be 17 percent and then this is 17 percent so this how we can analyze the spectrum and you can ignore this spectrum here you can ignore this one and of course you can also look into one HNMR spectrum here because trimethyl group is there and this trimethyl group what happens this is of course here you can see both are identical both the trimethyl phosphine groups and CH3 first it will be coupled with phosphorus to give a doublet and then each doublet something will be there rhodium to hydrogen coupling and then we will see a hydrogen to platinum coupling will be there something like this again so this is how the one HNMR would look like and most of the time by default when we are looking into phosphorus NMR what we get is proton decoupled one and of course if we are interested in looking into phosphorus hydrogen coupling then we can go for coupled one in that case it should be represented something like this the moment we write in flower bracket next to phosphorus 31p it indicates that one H is decoupled and that means interaction of hydrogen protons in the magnetic field when we are measuring phosphorus NMR is nullified so now I have another interesting compound here in this one it's a dimetallic compound here and two platinum atoms are there at terminally they are held by two trimethyl phosphine each and the middle we have diphenyl phosphide ligand is there so both the platinum if it takes symmetric it will be 1 1 or it can be one side it is platinum 0 and one side it can be platinum 2 also so now we can look into the spectrum of this one here how it looks like let me write the structure again here this is about 195 platinum NMR first we can look into 195 platinum NMR this is a structure so now if you see these platinums are both chemical and magnetically equivalent both the platinums are coupled on either side by two different type of phosphorus units so here let us say first this is coupled with two phosphorus units and if you just look into 2 N A plus 1 it should be a triplet and then this separation or this separation would be 1 J P T P I should say P P H 2 and now each of this signal will be further split by two end trimethyl phosphine so this will be coupled with first these two and then these two so this separation is 1 J P T P M E 3 so it shows a triplet of triplet and if you look into the spectrum it should look like 1 H 2 2 H 2 1 something like this so this is how the 195 platinum NMR looks like but on the other hand when we look into phosphorus NMR phosphorus NMR if you look into it these two are identical and they are equally coupled to 4 primethyl phosphine as a result what would happen delta P P H 2 let me write here it will show first a quintet so it show a quintet and then each quintet is further split into a doublet in the form of a satellite so as a result what would happen is we get something like this so something like this again this is for 66 percent and then this is 17 percent and this is 17 percent and then distance from the middle one to middle one or first one to first one this is called 1 J P T P coupling and this spacing is called P P coupling P P coupling this is how you can write it but on the other hand this would give a this one these two are identical with this one when they are coupled with this one we get simply a triplet and then this triplet is split into another doublet because of platinum coupling so this would be something like this something like this again here this is your 1 J P T P coupling so this is how the spectrum can be analyzed so now let us look into other applications we have several applications are there one very important application I am going to show you especially those who are working with nanomaterials and working with carbon compounds such as phallorines graphene graphene oxides and other things very very important one measuring epoxide content of carbon nanomaterials the presence of epoxide on nanomaterials such as carbon nanotubes and phallorines can be readily monitored by reacting these nanomaterials with triphenylphosphine how it helps in understanding the presence of epoxide so this method involves the catalytic reaction of methyl triaxorhenium so an epoxide reacts with methyl triaxorhenium to form a five-membered ring and in the presence of triphenylphosphine the catalyst is regenerated forming an alkene and then formation of triphenylphosphine oxide so when we add triphenylphosphine along with methyl triaxorhenium to nanomaterial containing epoxide oxygen is abstracted by triphenylphosphine to form triphenylphosphine oxide by simply taking the known quantity of triphenylphosphine and by monitoring the 31 P N M R spectra at regular intervals we can quantify the amount of epoxide present by looking into the amount of triphenylphosphine is converted into triphenylphosphine oxide and how much is left at the end of the reaction so this is the reaction for example we have an epoxide is here and epoxide when it is treated with methyl triaxorhenium it forms a cyclic system like this and when this cyclic system is treated with triphenylphosphine it abstracts oxygen from this oxo cyclic system to form triphenylphosphine oxide and regenerating a methyl triaxorhenium through the formation of this olefin so now exactly this is what happens so for example we have one graphene we have epoxides are there you just treat with this one and then as I said here triphenylphosphine abstract and this one is regenerated and then by simply looking into the reaction mixture N M R we can conclude that yes since the amount of P P H 3 used in the reaction is known the relative amounts of triphenylphosphine and triphenylphosphine oxide can be determined simply by looking into the integration of 31 P signals in the spectrum and that intense gives the amount of epoxide present on the nano cubes so this is very very important very simple reaction one can analyze and also one can also get rid of all these epoxides if they are not read it so you can see here 31 P spectra before and after the reaction before the reaction so full quantity is there and now after the reaction what happens partially P P H 3 is converted into triphenylphosphine oxide depending upon how many epoxide groups are present on nano material so then if you just look into the integration now only 0.4 percent is left let us say if we have used two equivalents 1.6 is converted into triphenylphosphine oxide and now only 0.4 percent is left in the solution so that gives how much is there in it this is a very very important method one can use effectively for determining epoxides present so let me stop here and continue with more interesting examples in my next lecture until then have an excellent time reading spectroscopy thank you.


Residents of M&D Trailer Court here in the community of Auxer in Prestonsburg walked to a gruesome discovery this morning as a man was found dead laying in the yard of one of the homes here. According to a press release from Kentucky State Police, KSP received a call just before 7 a.m. this morning responding to a fatal shooting here in the community of Auxer. Preliminary investigation indicates that 35-year-old Brandon Endicott of Prestonsburg entered his ex-girlfriend Crystal Shepard's residence. While inside the home, Shepard discharged a firearm striking Endicott. Endicott then left the residence and was found in a neighboring yard where he had died. Endicott was pronounced dead at the scene by Floyd County Coroner Greg Nelson. His body was sent to the state medical examiner's office for an autopsy. The case will be presented to a grand jury for consideration. Shepard at this time is not in custody. I spoke with one of the neighbors here at M&D Trailer Court who say that a stray bullet struck their home this morning as the incident was taking place. The bullet entered through the outside of the trailer through the hallway and entered one of the bathrooms striking the bathtub. Now they say that they are usually getting ready for work around that time in the morning and fortunately were not. Now just last week on September 5th, a warrant was taken out for the arrest of Brandon Endicott as Crystal Shepard filed a complaint of second degree assault. In that complaint, Shepard states that Endicott committed the offense of assault when she asked him to leave her home. He continued to allegedly hit and knee her in the face. It resulted in having severe bruising, 16 stitches all in her face and a fractured bone below her left eye. We'll of course continue to follow this story as more information does become available. Reporting from Aukcher, I'm Sean Allen for EKB News.


Okay, it's a great pleasure for me and honor to be here with Juan Maldesena in conversations about physics and how to do physics and about the ICTP. So Juan Maldesena is giving the Salam distinguished lectures here at the International Center for Theoretical Physics. We are in fact at the moment sitting in the Salam's office, Palma office and he has been telling us about connections between quantum mechanics and space time and very exciting developments at the very frontiers of physics. He is a professor at the Institute for Advanced Study in Princeton and is widely regarded as one of the leading physicists of his generation and he has made pioneering and very original contributions to several aspects of quantum gravity and cosmology and in string theory and perhaps his most influential work is what is known as the ADS-CFT Holographic Correspondence which has been a very active field of research for the last 20 years or so. So maybe let me start my questions starting with the ADS-CFT correspondence. So at some level it's a, I think for all the practitioners in this field, it's clearly something very deep that we have uncovered about an equivalent, quantum equivalence between a theory of gravity and the theory without gravity but sometimes I get a feeling that even our colleagues in the very nearby fields don't fully appreciate how remarkable is this equivalence and even more so for a common person on the street. So maybe can you try to bring out what you think are the most, a bit about the ADS-CFT correspondence? Yeah, so this is a relationship between a theory of interacting particles and a theory of gravity and it has a few interesting aspects. So one of them is that the theory of gravity seems to live in more dimensions than the theory of interacting particles. So somehow it looks as if one of the dimensions of space-time arises in an emergent way, so emergent is like in some approximate way or some way that was not manifest in the original description. And this also has enabled us to understand the aspects of black holes because you can have a black hole in the interior and it's described by a thermal system or some system in the boundary theory. And then this leads to a description which is fully consistent with the loss of quantum physics because it's embedded in the black hole in the ordinary theory of quantum physics. But also it seems to give an interesting picture for what space-time is. So it looks like the degrees of freedom that describe space-time do not live, there are no atoms for space-time that live locally in space but they live at the boundary far away. And while this is an aspect of quantum gravity, I think we are yet to discover the true lessons of this for cosmology for example and for the big bang singularity and so on that they probably will have some lessons or maybe it's even weirder than this idea CFD where so in this correspondence the direction that is emergent is the radial direction so it's a space-like direction. In cosmology we expect that perhaps the emergent direction is the time directions there might be some description perhaps without time and then yeah well now I'm speculating about the applications to cosmology. It's a very interesting question. But even without speculation so we've come to this realization by thinking about objects in strength theory, these d-brains so-called d-brains which are some type of soliton or some solutions of strength theory that have a very precise description found by Polchinski and using them and also the relationship between them and solutions of gravity, black brains and black holes, one was led to this correspondence but we still don't fully understand how it works so we think it's true and we can check some of the consequences of the relationship, some detailed mathematical formulas so if the relationship is true then you derive that certain mathematical equalities should hold and they do hold but in general the relationship relates strongly interacting systems of quantum particles to some gravitational systems some let's say Einstein gravity systems but one of the issues is that it's very hard to study this interacting systems of quantum particles so it's hard to find an explicit correspondence or the explicit map between the two variables. So in physics sometimes we have different descriptions of the same phenomenon so I mean the very simple case is that if we have a signal that varies in time we can give the signal as how it changes in time or we can give all its frequency components for example and those two descriptions of the signal are equivalent and this is an equivalence we understand in great detail and we understand completely explicitly and but in this other case we don't have such an explicit understanding that would be one of the goals would be to have an explicit understanding or mapping between the two types of variables that the variables we use to describe space time like metrics and the geometry of space time and the variables we used to describe the quantum systems the quantum mechanical system so yes I think this your work has influenced the work of many of us in the field last year in Princeton there was this 20 years of ADSRT and it was still and all the talks there were still it doesn't look like this idea has run out of steam so if you look at this last 20 years what development has sort of surprised you the most? Well there were many interesting surprises I there were surprises for me in terms of applications perhaps the application to condense matter is for me was the most surprising I thought that the systems are very different but nevertheless they were interesting lessons for condense matter and vice versa the condense matter physicists have given us also interesting perspectives in terms of the same more formal developments I think this connection between entanglement and geometry perhaps the best example is there's a formula that Ruh and Takayanagi derive for computing the entanglement entropy of a region that is a kind of generalization of the famous Hockenbeckenstein formula which computes the entropy of a black hole and so yeah I think those are yeah that that's that was an interesting development that has led to again surprising more surprising connections so the idea that entanglement is fundamental for determining the structure of space time then you can have connected space times even though the Hamiltonian interactions are not even that though the two systems are not connected so somehow space time seems to capture the correlations present in the quantum state so it's a way as a way to describe the the correlations present in the way function yes actually that brings me to a question that also I have asked myself sometimes that so at some level ADS safety correspondence is to teach us something about quantum gravity about black holes and so on and but there is another point of view that maybe ADS safety is just a kind of a framework for studying strongly cup you know it's not like the gravity is just some kind of effective description right right and how do you view what is your take on this do you view well ADS safety is something deep about gravity or do you regard this as a kind of a framework for gravity emerges as just effective description well yeah I think if I think I think the question is whether gravity is a bit like hydrodynamics so hydrodynamics is a theory that we used to describe fluids but hydrodynamics is not the most basic description of the system so we have some interacting particles that interact the loss of quantum mechanics or classical physics or whatever and then long distances or for insufficient in certain circumstances we can also have a hydrodynamic description the hydrodynamic description is in some sense approximate it can fail sometimes you can phone shock waves you can perhaps have some singularity so and and we are not surprised by this and it's also description where we lose information where entropy can increase so on now one perspective is that well maybe gravity something similar so gravity is an effective description of the quantum system and it will be approximate and well it breaks down sometimes and well it might be then replaced by something else and there's something else that replaces is the quantum system so you have to go back to the quantum system so that's a perspective that is actually quite popular and some people think about it this way but even if you accept this perspective it's a kind of effective description that keeps more information about the entanglement of the system and even microscopic aspects of entanglement so for example we just discussed this Rittak Kyanagi formula this formula is not computing the entropy of the system the usual entropy of the system is computing the microscopic entropy of the system defined so called fine-grained entropy and so that to me tells me that the gravity knows more is keeping some very detailed information now you can say well okay I'm skipping this data information but only the total value it doesn't know about the particular microstate so you could also say that but well it's a little more than hydrodynamics at least in that sense and I think the original idea was that it was a true duality that there were two possible descriptions one was the gravitational description and certainly strength theory gives us full perturbative description that makes sense and we think that even non-perturbatively we can define this theory and for some observers they some very supersymmetric observables you can actually do the full computation and some this perturbation series so that that was always viewed as the evidence that there is a theory of gravity that is independently defined it could be defined by strength theory for example and yeah I'm inclined to view this the second option I think gravity is too beautiful and too interesting to be an effective theory and I suspect that maybe once well this is now speculation well the previous statement was also speculation that once we understand gravity well enough actually the quantum system will be some will be immersion will appear when we cut the gravity description into two pieces so we we introduce some boundaries and we separate we separate the full gravity solution into a piece that we want to describe and the rest that we view as measuring or some classical aspect this classical let's say measurement apparatus that is measurement measuring the I see that's an interesting point of view but entanglement yes somehow you break the entanglement a bit yes yes okay so we're coming to I mean we'll come back to science but before that I also wanted to ask you a bit about so ICTP has this other important aspect to its mission that on one hand we have excellence in science right and on the other hand we have inclusion right of people who are normally left out of the scientific right and that was certainly the guiding principle for right and you yourself come from Argentina you have did your early work in Argentina then you went to Princeton so and I know that you are also a frequent visitor I think I remember also giving lectures with you a long time ago you have been here maybe I don't know 20 times 10 times yeah and you are the member of our scientific council so you have sort of seen ICTP is functioning so yeah so do you well I think ICTP plays a real singular role in connecting science to the countries to let's say well the old days they were called third world let's say least developed countries developing countries and then and indeed it had an important role to play well in Argentina it was pretty important for scientists to connect to the developments in the latest developments in science so for example that there was my my former advisor in Argentina he well did some his PhD in Argentina on some aspects of statistical mechanics and then he came here to the ICTP to the postdoc and at the time everyone here in the ICTP was very excited about string theory was one of the main well it was after 1984 with all these developments and so he came here and everyone was doing string theory so well he joined if he wanted to talk to everyone anyone here and well he he he learned string theory here he did some work on those string theory subjects here then he went back to Argentina and started a group doing string theory and well I joined them and I during my let's say master thesis early bachelor thesis and so on so yeah and that was very useful for me and also for other people in Argentina another example is Horacio Cassini so he also had a similar story yeah yeah so he did his work on particle physics in Argentina then he came here to the ICTP he started here working on entanglement in quantum field theory and well then he went back to Argentina and continued working for this field for this field for a long time and not many people were paying attention to this field and and he he and also Marina Huerta well they both came here and went back to Argentina they they obtained really beautiful results and using these methods that revealed as deep truths about quantum field theory so basically what they did was to show that there are certain certain quantities related to entanglement well and basically entanglement tropists are monotonic when you sort of decrease when you go in the RG flow direction so when you coarse grain theory and people did this was kind of a holy grail in quantum field theory was to find quantities that decrease when you follow the sea renormalization from yeah that's right so in two dimensions it was known that it was true and they found the proof of this in three dimensions they also prove the old results into dimensions using these techniques and also in four dimensions which also had previously been shown using other things so but they had a unified description of all dimensions yes using using all these techniques and well they found also many other things interesting things related to entanglement and he well they also created a group in yes and lectured last spring school yeah okay yeah and you see that they well it comes back to here to the ICP and yeah so so these are actually are two really very important sort of success stories I mean in a even in the indirect waves just guiding somebody in the right direction or sending people in the yeah exciting directions of science has this big effect on right right I mean the one of the ideas of ICP was to be a place where scientists from from developing countries can meet and one of the difficulties of doing science in the developing countries is the lack of connections and the lack of well the lack of communication so of course now with the internet you can you can see people's papers and so on but there isn't a face-to-face contact what is exciting yeah and what is exciting and the news and and all that and one of the ideas of ICP was to serve as a place where this could happen and I think it's working yeah it's working well if it has led to ADS-CFT success yeah so and okay I mean and what yeah so this is certainly a one very important aspect of ICTP's mission but okay now as a leader in this field and also being for example an advisor to the ICTP what areas of I mean where do you think ICTP is doing a very good job and or where we could improve what do you have any thoughts on that about well I think the the schools are an important part the associates programs it's also an important part yeah I think what is most those are yeah I think those are very very effective programs and I think they just to bring people here and yeah yeah yeah yeah bring people here making what the talk to each other interact with the the faculty here interact with other visitors that come for schools so in a meeting place as a meeting place I think that was actually his original yeah and I think yeah because it's a it's a question to be asked because you know in 50 years things have changed as you said countries which were regarded as third world are now the different bracket in India and China or big countries are in a different bracket so one needs to rethink about how to reorient ICTP's mission a little bit but right you said that some of these aspects still remain true today and right right and maybe we can do things different maybe maybe we can get money try to persuade for example one in this initiative in Brazil I think that's one way to spread our mission or yeah so that's another thing that the CTP was doing is to create sort of daughter centers yeah so that use the expertise that has been built here building an international institution to create other more local regional international centers for example I've been involved with one in Brazil there is also a similar one there that was started later in Argentina of creating a place where again would be a place for people to meet within the country and perhaps with neighboring countries perhaps bringing scientists from abroad so a center that would work well with international standards and the ICTP can guide them to to do this can oversee it yeah so and it's it's working developing and I think the CTP has done this in various places of the world I'm only involved in these two places yes also we have in yeah now we have an important Beijing yeah okay so maybe coming back to science when you see briefly mentioned I mean clearly one of the very exciting scientific developments has to do with cosmology and anti-disseter space that where the ADS-CFT correspondence comes and the deseter space in which we seem to be living in they're kind of closely related but at the same time they're very far from each other yeah yeah one has yeah one has somehow positive what we call positive curvature and the other one has negative curvature so we got the sign wrong you say why do you study this one if we live in a positive curve space with positive curvature why do we study the one with negative curvature and the reason is that for the one with negative curvature we understand more and in physics sometimes it's easier to well it's useful to study a case where at least you can say something then get some intuition perhaps we can say something also for the case with positive curvature in many ways they are not they're not too different I mean in fact if you do calculations purely in gravity you can translate between the calculations done in both cases just the level of solving the classical equations and so on if you solve the classical equations with negative curvature you can immediately get the solution to sign yeah you flip the sign you don't have to resolve the equation same same solutions can be come apply for the other case and indeed this led to some interesting ways of thinking about cosmological correlators so it's cosmological observables right that are computing in inflation so indeed the period of inflation is the period where this universe was well approximated by the cedar space and yeah so but we don't have clear examples of a full duality where you have you know the full quantum system for a system that is due to Einstein gravity there are some examples for systems that are related to higher spin gravity so some other kind of theory of gravity sometimes called so also was illic gravity where where you don't only have the gravity on but you have any number of particles with higher spins so yeah yeah certainly I guess this exploration of non-gaussian it is in cosmology was kind of inspired by you could say why yeah yeah yeah that's right that's okay but one has not had this something as concrete as the idea safety cosponsor yeah that's right that's right yeah the non-gaussian it is heard the well so you have the primordial fluctuations that have been studied using what they did they printed in the cosmic microwave background and are very close to Gaussian but non-gaussian it is a small statistical deviations from let's say the bell curve have have the potential to teach us something about the interactions that were present during inflation and yeah this is an this was an active field I mean people are looking for this non-gaussian it is they've been looking for this non-gaussian it is for a long time and yeah and hopefully with the new newer experiments will have the statistical power to to see this tiny non-gaussian it is so what we know is now is that well the non-gaussian it's not big they're very small but hopefully they will see them at some point and teaches something about the very early universe I mean this is the best probe that we have of the very early universe and we can call we can call this we can think of this as a kind of cosmological accelerator accelerator which or microscope that is allowing us to look at the universe when it was very small yeah the highest possible the highest energies that could be accessible by observations and and so the roughly speaking the non-gaussian it is tell us about the particle collisions that occurred in the very early universe so there were there were particles or waves in the very early universe and their collisions produce some non-linearities and that's this non-gaussian that they get translated in our present observations as this deviation that is statistical deviations from Gaussian anything I have another question about sort of maybe it's a bit about sociology of science so in the past there was kind of a much more direct link between theory and observation meaning whatever you imagine you could more or less imagine doing an experiment to your time or oftentimes the data was ahead of the theory and prompted we are somehow in a kind of a peculiar historical situation where our theories are extremely powerful and their structure is so rigid that we have great confidence for example the Higgs boson is an example where most more or less all physicists were convinced that okay spontaneous symmetric broken gage theories is the right framework okay how the details will work out they were the you know an electroveic unification salamence was responsible for it but it took 50 years and billions of dollars and thousands of physicists working on it so there is clearly some additional criteria of logical consistency and which is as important as verification I mean even before it want to think about verifying it with an experiment right so that seems to guide the research and it's even more true in string theory so do you want to sort of comment on this that how does one in a world where we don't have immediate contact with experiment how does a physicist choose good problems or right right right so as you said the the electrobic theory was an example where people could have some idea of how physics could be at the distances much shorter than the ones they could measure at the time and indeed it took many years to get to those shorter distances string theory or quantum gravity involves a shamping distances which is much bigger much bigger so it's like roughly the shamping distances between our scale and the scales that the LHC can probe today right so if we if we shrunk to those scales then quantum gravity would we have to build another LHC yeah that's right but so you might think well you I mean many things could happen and indeed the many things would happen but we think that the the the framework we could use through all those scales is the framework of quantum field theory and that framework would have to be changed when we get to gravity and and that that I think we we know and and and string theory tries to use a theory that for how this framework could be changed and I think that it's very difficult to put together quantum quantum mechanics and gravity and the fact that string theory managed to do it says that well this is only an interesting example at least to study and that it's a problem that looks so constrained that they might have a simple unique solution or much like in case of quantum gauge theory yeah yeah or at least we might have a unique framework or we maybe the framework is more general than string theory maybe it's not the current understanding of string theory but it tells us that string theory is only an interesting example or toy model to study or at the very least it's that it might be the full thing but I think our current understanding of string theory is probably to elementary but yeah and it's a worthwhile problem that can show and and perhaps one of the surprises is that by thinking about this very abstract problem you learn some things about ordinary problems yes that you could teach you an example yesterday yeah that's right that it could teach you something about quantum field theory about what strong interacting quantum systems and so I didn't have to be this way yes maybe it was something so there is theory but it turns out this so there is very closely connected to ordinary physical theories it's perhaps not surprising because actually string theory arose not from some esoteric thinking but by trying to fit you know string the strings that were observed in experiments the strings of quantum chromodynamics so and so it's closely connected connected to physics yeah in fact the EDS CFT is an example because the CFT is a very close cousin in many of this EDS 5 example is a very close cousin of the quantum chromodynamics which is sort of a glue that holds the nucleus together yeah it's closely related to it's not exactly quantum chromodynamics but with some minor modification that is closely related to theory of gravity so I think there is that connection but okay but what are the criteria sort of I mean if you were to if a young graduate student is asking yeah question well I think what guides your well I think the criterion is that the community of the community in general yeah it's so I like to joke the strings sounds it's an acronym that means solid theoretical research into natural geometric structures and that's roughly the thinking of the community in the sense that well you you have you can theorize how much you want but it has to be at least mathematical it has to be mathematical consistent there has to be some solid formula and the formula might be have very narrow application or it might only hold for some very special thing but at least it has to be correct within at least self-consistent if we use theorize so much that the formula even is not self-consistent yeah then there is no formula then then it's a little more suspect and I mean if it is officially interesting we can totally no formula but and and also also some natural geometric structures what does that mean I mean it's there surprising thing about nature is that there are some theories that are very powerful like quantum field theory that describes describe systems of interacting particles describes the standard model it's incredible successes and this is a structure that has lots of subtlety and and you can study it on its own and and much of the the work in our field is actually understanding aspects of quantum field theory even before trying to understand quantum gravity and understanding those aspects will probably teach us something about quantum gravity because any theory of quantum gravity has to recover quantum field theory and general properties of quantum field theory and so and and I called it geometric because you know by special relativity we know that what quantum filter is the the theory that puts together space time so the space time of general special relativity and quantum mechanics together and that does this successfully and well quantum gravity should do that but with general relativity quantum mechanics and general relativity together so it's only a first step and well of course general relativity is a theory of geometry and geometry is very important their interest in geometries like black holes and cosmological geometries that they are still very confused they are still very confusing they were very confusing classically so it took 50 years to understand the Schwarzschild solution so the simplest solution of Einstein's equations it took 50 years to understand what the classical geometry was not to detect the gravitational waves yeah yeah this is one but I'm now talking about the theory so just theoretically understanding what the geometry that the Schwarzschild solution describes took 50 years now of course it took many more years to understand how to see black holes in nature and whether black holes are are or not present in nature and well now we have great evidence that they are present but perhaps the most beautiful evidence is from the gravity wave detection that we had very recently and yeah and probably we'll understand those those things better and better but maybe there are other exotic objects that we haven't thought about waiting to be discovered yeah I mean I guess one comment I want to say that somehow when a common person says theory you know I think the physicists when they mean theory they really have some quite elaborate and very closely fitting structure in mind like quantum field theory but as for a common person theory means okay I have a theory that it will rain today or I have a theory that so sometimes there is a yeah so that the fact that the the theories we make of physics have to be consistent with all previous experiments and our previous observations puts a huge constraint how huge that constraint is sometimes yeah yeah it's a very very big constraint so the constraints also well just the constraints of special relativity and quantum mechanics are very very strong and they lead to quantum field theory and two particles yeah yeah they lead to antiparticles and predictions of this kind so yeah antiparticles is a generic general prediction of quantum field theory there they are theory independent in the sense that I mean they are present for electrodynamics for chromodynamics for yeah so it just follows from the theoretical constraints of the theory yeah that's right very general so I mean another example is Hawking radiation yes that's a that's another example that just quantum field theory in the presence of horizons and so on predicts this thermal radiation that's also generic prediction of quantum field theory I mean there's a very close casting of Hawking radiation where you don't have to talk about the black hole which is to even be in flat space and you are an accelerating observer in flat space sometimes called radiation so that the accelerating observer would see a temperature that fact is also a very generic prediction of quantum field theory they also closely related is this primordial perturbations and then the quantum fluctuations that right right right right yeah that's right yeah and and this yeah maybe I maybe one could say emphasize the fact that now people sometimes say okay well Hawking radiation okay hasn't been seen if you talk a lot about Hawking radio why do you care about Hawking radiation you can calculate it for astrophysical black holes and it's very very tiny you'll never see it at least in the black holes that have masses for the solar masses which are the ones that naturally form so why do you study such a silly thing so but the point is that this this effect is also present during cosmology very closely and it's it's the same thing it's the same the same thing there is a cosmological horizon and it's basically a same effect and and that's the fact is it's crucial for describing aspects of our universe so the the fact that the universe is not perfectly homogeneous and and it describes the long-distance structure of the universe very well so it's not that this effect plays a central role in explaining how the universe is like like it is all the galaxies why they are so yeah not uniform structures in the observe it somehow came from this very subtle quantum yeah situation which you think is why is it important but it's actually all important yeah yeah it's okay so and so maybe I should ask you maybe the final question I should ask you so you've been talking I guess your lectures alarm lectures are on sort of some of your recent work on quantum entanglement and space time it's a very beautiful connection between entanglement and quantum mechanics and geometry of space time yeah so maybe can you tell us a bit about what are the exciting future directions for your research of all these developments that you have been recently occupied with can you see something about that well I think well in this relationship between entanglement and geometry is basically understanding how how the basic quantum description of space time builds the space time and part you could say understanding how how holography works part is to understand better the interior of black holes because the interior to describe the interior of the black hole this disconnection to entanglement seems crucial and to solve to solve black whole paradoxes and so on seems to be an important element and we don't understand it well enough to solve all the paradoxes we but we do understand that it is important for thinking about the paradoxes so for example going back to the Schwarzschild solution one aspect that was understood about the Schwarzschild solution is that it describes two black holes and the idea is that these two black holes are due to separate quantum systems but they are in an entangled state so understanding more this connection is seems to be interesting that's now where where is it leading so I think it's hopefully leading to a new picture or some picture of what space time is and how space the space time and the quantum are related so because it's interesting that you can have a connected space time that comes from this connected quantum systems because it's saying that the local physical loss of that you see in this universe which is some some some evolution law and so on are somehow present in the patterns of entanglement of the quantum system they are internal to that pattern and how to think about this is well what we are trying to understand yeah so they are sort of connected in this extra dimension that even though the systems are far away from each other there is this extra dimension that we talked about right right right and they're connected in that right extra space time dimension and yeah and this this connection is has some interesting implications for quantum teleportation which is a general quantum mechanical property of the systems of entangled systems and yeah and hopefully one will learn to use some of these ideas for cosmology in some way that's always that's always in the back of your mind yeah that's in the back of everybody's mind how to apply some of these things to cosmology there hasn't been a killer application but yeah given the history of string theory you know you know it was applied I mean the I mean we have already seen an example which was Hawking radiation right this car this car for black holes then apply to cosmology and yes okay it was really exciting and to hear I mean it was very thank you for a very nice pedagogical introduction to your ideas and work and it was a great fun to talk to you yeah thank you I look forward to be here yeah


I normally wear flip-flops as well, but I was always told when you present, wear shoes. So we're going to get started. There's plenty of room in the front seat, front row, free way in the back. So thanks to all the good folks at Constant Contact for inviting me here. It's awesome to be in Boston with you all today.  The keynote I thought was spectacular. I had never heard Michael speak before, but I would welcome the opportunity to hear him speak anytime. In fact, I'd love to bring him to Netflix and have him present there. And speaking of Netflix, are there any Netflix customers in the room? Thank you very much. We love you. We love you a lot. So as you probably already figured out, I do work at Netflix. I run delivery engineering, which is the – we build the infrastructure that all teams in Netflix leverage to do continuous delivery of various pieces of software into production. And what I want to talk about today is how we do continuous delivery, what we've learned about continuous delivery, and the challenges and the results of it, and then where we're going with continuous delivery. And as you're probably aware, Netflix is a global company. We were – well, over four years ago, previous to four years ago, we were a DVD company, if you remember that. And we made the decision to go – to offer streaming video, and that enabled us to go global. And right now, we're in 41 countries and counting. That number will increase very shortly here. We recently announced that we have – we had exceeded 50 million customers, and that number is obviously still counting. And one thing about the 50 million customers thing is that's an account, right? And as many people in the room are probably aware, you can have multiple people view an account, so we estimate that we have about 200 million people watching various videos. Since we're all engineers, I'll talk a little about our architecture. Netflix is a giant SOA. The new term these days is microservices, so you could say that may be – in many ways, we kind of spearheaded the whole microservices ecosystem or best practices associated with it. Most people are aware that we leverage AWS, Amazon Web Services. I did say we are a global company. We are spread across multiple regions in AWS. And at any given moment, we have thousands upon thousands of AWS instances running. And so why continuous delivery? Streaming video isn't terribly difficult, and in fact, constant contact could be streaming this now if they wanted to, right? Anyone of us can stream video. So the fact that we stream video is not all that – there's a very low barrier to entry to this market, right? So what differentiates Netflix is our data. What we collect about what you're watching. How often you watch it. What choices you make after you watch that. And then we use that data to figure out what content we should buy, what content we should potentially even produce ourselves, and then how we make recommendations, what shows up in search and whatnot. So really, we're a data company. Streaming video is, as we all know, pretty easy. And so because we're a data company, we need to be able to move very quickly and we need to foster an environment that facilitates that rapid innovation across the globe. And I thought Michael's talk as Keynote was fabulous when he talked about some of the aspects of, you know, volatiles and stables and speed, and that's a huge concern of Netflix in the sense of we want to move fast because moving fast is a huge competitive advantage. So again, low barrier to entry for anyone to stream video. And as we've already seen it, you've got Hulu, you've got Amazon, you've got HBO. We have a number of competitors. And so what we see is our ability to make decisions based on data and then change the viewing experience for our, for you, for our customers. If we can do that faster than the Amazons and the HBOs and the Hulus of the world, then we'll stay ahead of them. And the last time, I don't know if people are aware of this, but at peak Internet usage times, or peak Internet usage, Netflix takes up 33% of the Internet, which is just awesome. I think it's Hulu, Amazon and HBO added up are less than 1%. So we're doing a pretty good job. But again, as the keynoter pointed out, we can't sit still because if we rest on our laurels, we'll eventually be beaten. So competitive advantage, it's all about speed. And the reason we want speed is so that we can make the viewing experience for you all better. It changes every day. And in fact, we see continuous delivery as the rails, so to speak, to facilitate all of that speed, all of that innovation. So there are teams at Netflix that are deploying multiple times a day. You may not see the changes to your home screen or whatever it is. But all the algorithms, all search speed, everything is being changed at any given moment. In fact, I'm willing to bet there's probably a deployment. Well, it's pretty early in the Bay Area right now, so there's probably not a deployment going on right now. But by the end of this talk, I'm sure at least one deployment will be executed. So continuous delivery is extremely important to Netflix. We see it as it facilitates our competitive advantage to move fast. And I'll get to some details about continuous delivery at Netflix. But in general, continuous delivery, if you want to do continuous delivery, you need three primary facets. And the first one is you need a process that's repeatable. Moving from check-in to some environment requires automation. I think we're all engineers, we all know this now. Manual steps are error-prone, it's just a non-starter. So we use, where Netflix is primarily a Java shop, and our automation is largely driven by Gradle, which is a build system kind of the next generation.  So a lot of our automation is spearheaded through Gradle, and then the tools and platforms that my team writes. Next facet for a successful continuous delivery process is it's gotta be reliable. What I mean by this is implicit in all this is you have to have lots of testing. If you wanna move some piece of code that some developer wrote into production as quick as possible, you need some confidence with that.  So when I talk about reliability, it's testing at all levels. Whether it be unit testing, integration testing, functional testing. And I'll show you how we do that. But if you don't have kind of a testing mantra or philosophy, I would start there before you even try to push stuff quickly into production. And then finally, it's gotta be rapid. I said quickly, right? Processes that take days and days to get something into production are a non-starter. Again, we're all engineers, we're very patient types, aren't we? If I have to wait many, many, many hours to get something in production, I won't do it. Or I'll sort-circuit it and I'll do something else. I'll just go right to the production machines and make the change there or something like that. So your continuous delivery process needs to facilitate rapid delivery. And rapid, obviously, is a very loose term. There are some pipelines at Netflix that do take hours. And I'll get to why they take hours and you'll see the benefits of that. But you could short-circuit those particular gates and make it go quickly. But you'll see there's some power of going, or some increased confidence through going through these gates. Two more things, and these are kind of Netflix specific. But I hope that you can learn from these. We make two assumptions going forward with respect to continuous delivery. So the three facets previous to this are given, they're there. There's two more that are kind of, one's cultural, one's technical. And the first one is trust. At Netflix, we have a firm belief or our mantra is this whole notion of freedom and responsibility. Every team at Netflix is free to do whatever they want. There are no, and this is, again, very complimentary to the keynote. Netflix is a very anti-process company. So every team is essentially free to define what continuous delivery means to them. If you remember, I introduced myself as a team building all the platform to facilitate it. Teams don't have to use anything that my team writes. But I assure you every team at Netflix does use what we write. But it's their choice. I said we're largely a Java shop that gave me that little asterisk, right? Any team at Netflix can choose to write things in Ruby, Perl, Python, you name it, Go. But the majority of teams choose Java because of, again, the infrastructure that's there to support, let's say, JVM-based languages, and the tooling there to support continuous delivery. But at the end of the day, every team at Netflix is free to define what continuous delivery means to them and how they can use it. One other thing there that's implicit here, I think it's not implicit, it's explicit, is there are no operations teams at Netflix.  You are the one that gets the call at 3 AM when people in Europe can't watch Breaking Bad. If we're able to determine that, it was your service that broke down. There is a reliability team, it's a very small team, it's very lean, it's about 10 people. They constantly watch production and they're looking for errors and what not. We have a bunch of KPIs that are key performance indicators or metrics that this team monitors and when it dips, they'll quickly assess what's going on and we have extensive monitoring as you can well imagine in an environment like that. So we know at any given point if something's breaking, but that team doesn't fix it. That team picks up a phone and calls Andy or Stu to use the key note. Call Stu and says, your stuff's broken, you gotta fix it right now. And that level of responsibility has some cultural benefits in that, it's an implicit and continuous delivery was that reliability aspect. Software engineers at Netflix are extremely prone to write a heck of a lot of tests and take continuous delivery very seriously, that reliability aspect. Because again, if you're the one who's on call at 3 AM and you don't want that call at 3 AM, you will ensure that your stuff gets out there in a reliable manner and you'll take advantage of the kind of the infrastructure that Netflix provides. So trust is a huge, I think, cultural component of continuous delivery. Let engineers make the decisions, treat people as adults and they'll do the right thing for the company. And second is this notion of judgment, judgments via insight. So as you can imagine, continuous delivery is a series of stages, right? As code is checked in, it goes through different processes or again stages before it gets to some environment. Each stage is a quality gate that can be automated or manual if you choose. And there's a decision, do I go forward or do I roll back? And you have to have intensive insight in that pipeline so that either automation or humans can make those appropriate decisions and whether or not this piece of code should continue out to our customers or we should roll it back and do something else. So continuous delivery, you gotta have a lot of automation but a lot of checks and balances and insight, operational insight into how something's moving through this pipeline and whether or not it should continue to move forward. And I'll share some of the insight things we've built to do that. So I'm gonna generalize how we do continuous delivery but you can largely summarize it in four steps. There's a build step and I will go into detail of each of these steps and the tools we use and I'll leave time at the end so you guys can ask me questions and whatnot. But there's a build step where things are assembled. At the end of the day, the asset that we kind of move forward into production because we leverage AWS. How many people here are familiar with AWS? Okay, so pretty much all the room, awesome. So a machine image, an Amazon machine image. Amazon called it an AMI. Previous to joining Netflix, I called it an AMI. So I'll call it an AMI. The end asset from a build process is an AMI. And that AMI then is pushed forward. So implicit in the latter three steps is a deployment. And the key thing about building the AMI here is going forward. That asset, that service, whatever it is that's gonna spin up in these various environments has no dependencies. When it fires up, it doesn't need to make a call to download some jar from Maven Central or whatever it is, X number of things. It's all done here. So once it goes into this pipeline, it is a self-contained single entity instance that can be fired up in any region across the world and makes no dependencies on anything outside in the world that could be down at that time, and that's a key thing about reliability. Once we're done here, we can spin up an AMI even if GitHub or Maven Central or some YUM repository, you just name it. Any operational aspect that isn't available at that time, which is highly likely it will happen sometime, doesn't matter to us, we're good to go. So the verify step is, again, I talked about that reliability aspect, all the testing. Unit testing is a given, but once you get beyond unit testing, you gotta do a whole lot of testing. This is like integration style testing. Canary analysis, I will go into detail about this, but this is something Netflix I think kind of spearheaded. And this is the idea, and this is, it freaked me out when I first got into Netflix because having been the CTO at a different company, we wouldn't have dreamed of doing something like this. But canary testing is spinning up an instance, an AMI instance in production. And allowing some traffic to go to it, trickle traffic, maybe 1% of all streaming requests go to this instance. And we're analyzing it, we're watching it. And it's not so much analyzing the code but analyzing the machine and comparing it against a baseline, is the machine behaving the same way it behaved in the last release, i.e. memory consumption? You name it, I'll get to more details there. So that's called automated canary analysis, huge aspect for us to understand reliability. And then finally it goes live, and again that notion of live could be anywhere in the globe across various regions of Amazon. So the build process, how do we, what is our build process? Pretty simple, developer checks in code, the code is built and it is tested. Again, here we're leveraging Git and we use Jenkins. Jenkins I think is pretty much the standard kind of continuous integration tool across the world, although we are looking at other tools, we're always looking at what's next. But a big part of that is the testing aspect. Testing phase is obviously here is unit testing. A lot of teams will also do kind of different analysis, static analysis like PMD or find bugs or something like that. And then right after that, assuming all your test pass, we assemble that service into a Debian file. And then that Debian file is installed via something we call the bakery, onto an AMI, or an Amazon instance. And then it is baked, snapshotted. And then that AMI is then now our asset going forward. Remember, no dependencies on anything else in the globe. This thing is good to go. We can put it in any region in the world, fire it up and we know it's gonna fire up and be fine. Well, it'll fire up. Whether or not it'll be fine is we'll find out in a little while. So far so good. So I didn't mention this, or maybe I did mention this, but because we're making Debian files, we are running on Ubuntu. Ubuntu is the standard Netflix OS going forward. We did used to use CentOS. So previous to doing Debian files, we were doing RPMs. All right, verification. I think people are familiar with the Netflix kind of open source stack. Open source to a whole lot of software. Not many, awesome. Guess what? Netflix open sourced a whole lot of software to facilitate all this. I highly recommend you look at it. Go to github.netflix.os s. One particular tool that we did open source, and I will talk more about this as we talk about where we're going, is a tool called Asgard. And what Asgard does is it takes that AMI and spins it up in any of three regions across the world. And since everyone here is familiar with AWS, some details about AWS. The highest level construct in AWS is an auto scaling group. So you have instances and you can group those instances into auto scaling groups. That's great, it's fine and well. But for Netflix, we saw that it's kind of limiting. And we've added a model on top of that. So we have this notion of clusters. Clusters are a group of auto scaling groups. Yes, a group of auto scaling groups. A group of ASGs. And then a group of clusters is grouped together into what we call an application. So we wrote a tool called Asgard that helps manage all that. So if you have a service out there, it's largely an application. That application that has various clusters that can be spread across the globe. Those clusters then can have various ASGs. And then those ASGs have various instances. And again, I mentioned earlier that at any given point there are thousands upon thousands of instances running across the globe. This is how we basically manage all that. So rather than dealing with individual instances, you're looking at essentially auto scaling groups, or you're looking at clusters, or you're looking at the app itself. Again, we wrote that. We have a tool called Asgard that manages all that for us. However, we're in the process of replacing that tool with another one. And of course, once you then have copied this AMI into the various regions of the globe, you spin it up and then you run a series of integration tests against it. So once more, you have this reliability aspect where we're verifying, does the code work like we think it should work once it's running up in an environment? So these are tests like, again, these are coding tests. Selenium, JUnit, Spock, you name it. All these things are being run or higher level tests against whatever service it is. Again, to verify everything's working. Another thing we also have is we have multiple environments. So you can run these tests in a simulated prod environment or a test environment staging. I think everyone here is well aware of different environments and familiar with this. We have the same concepts. All right, this is where it gets really interesting. So assuming that step is passed, everyone's happy. We've run all our functional tests. We'll deploy that AMI into a new cluster, and it's a canary cluster. And so what we call this is automated canary analysis. And there is an entire team at Netflix who has built a series of tools that essentially can monitor your app in a production environment. And monitor the app and the machine around it. And we save all that information, so we have baselines. So let's say we have a service called foo, and it's running a production. Constantly monitoring it and understanding the average memory usage, disk usage, CPU load, you name it, log growth, and that's a baseline. So you can use this tooling to essentially take this new AMI, throw it into this cluster, and then via load balancers you can say, give me 1% of all traffic, go specifically to this cluster. It could have multiple instances. And we'll set this machine at it, or this, it's actually not a machine, but more like an entity, a being, Skynet. And it'll watch this thing behave. And in some cases, you may choose to do this for a number of hours. It may take a couple hours to get that baseline. So this is earlier I alluded to, it's got to be a rapid process. But some teams, it may take eight plus hours and whatnot. In this case, what they're doing is they're again, shuffling a little traffic. So at any given point, you may not know it. When you fire up Netflix, your traffic may be going to some new instance. And this entity is just constantly watching it, gathering all the metrics. And essentially, there's a nice big dashboard and it's giving you a threshold and saying, things look good? Go ahead and just spin this up live, or something's weird here. The memory, whatever it is, you name it. Some operational aspect of this app is behaving differently than it did before. And this is all automated, it'll just roll back. Or if you set that threshold, maybe it's 90%, again, it is a score at the end of the day. And it's below the score, obviously you're going to roll back. It's above the score, you'll go forward. And so, if you go forward, we have a very kind of interesting way of going forward or going 100% live. We call it a red-black push. I believe the real world calls it blue-green, yes. I was gonna say black-green or something, but blue-green. But Netflix calls it red-black because previous to a couple months ago, our logo was red and black, but now it's not red and black. So maybe we'll call it red and white, I don't know. But it's a red-black push and it's a four-step, choreographed deployment. The next slides, I'll show you how it works. But implicit here is that it's taking advantage of the cloud. This is the beauty of the elasticity of the cloud, whether it be AWS or Google or whatnot. And it is, most companies do what's called a rolling push. And red-black is, rolling push is cheaper, right? Because it uses fewer resources. Red-black push essentially doubles your capacity, right? In fact, let me show it to you. And again, this is why you can pretty much get away with it in the cloud. It's kind of hard to do it in data centers. Although there is a gentleman, Brian, later on, talking about Docker. So if you're in a data center and you haven't looked at Docker, Docker can facilitate this kind of stuff, maybe potentially nicer. But so red-black push is like this. You have a, here's this AMI that we burned. So to speak, we've baked and it's live. It's version one, it's running out there. In this case, I'm gonna talk about one AMI. But just imagine this is like a thousand. And it's running in an ASG or a cluster. And so what we do is we spin up version two right next to it. So this could either be, again, one instance or a whole another cluster of instances. And they're running side by side. And then we let a little traffic trickle to the new one. And again, we're still doing the automated canary analysis. That's pretty much going on at all times. And then we essentially turn off the old one. Not turn it off, I shouldn't say that. We turn off traffic to the old one. So at this point, traffic's going essentially, it starts out 100, then it's 90, 10, 80, 20. It's slowly doing that until at some point it's 100 and zero. And then this guy's the new one and this guy can either be left out there, basically disabled or completely destroyed. The beauty here is you can roll back instantly. The anti-pattern that we solved or that the industry has solved with green, blue-green, as opposed to rolling pushes, this whole notion is you can always flip back, right? So in this case, if this guy all of a sudden just dies, you can quickly turn the load balancer and go back to here and you're good to go. This is all facilitated via that tool I mentioned earlier called Asgard. We had to write a whole lot of tooling to do all this. And this is how we deploy. This is the end of the line, so to speak. This is the end of continuous delivery, a pipeline for continuous delivery. So far, so good? All right, it's pretty simple stuff. Okay, challenges, what have we learned? It's not the easiest thing in the world to get this far. So, first and foremost, I'm here to tell you we had to build it all and we open sourced everything. So I highly recommend go look at our open source suite. So when Netflix four years ago decided to go streaming, we were in a data center, like many of you. And we knew that if we wanted to seriously go global, we could no longer be constrained in a data center so we elected to go to the cloud. To go to the cloud at that point was pretty darn early. And then to facilitate continuous delivery into the cloud was unheard of. And so we had to build a ton of stuff, ton of tooling, tooling, frameworks, libraries, and we decided, you know what, let's just open source it all. Because it's not, again, our competitive advantage is to move fast, but what differentiates us is the data. So all the stuff that we used and we wrote to move fast, we give away. So I highly recommend you take a look at this stuff if you're looking at going to the cloud and want to do continuous delivery. Pretty much everything I've talked about today is open sourced or will be open sourced shortly. Another reason for open sourcing it all, of course, was attracting talent. I'm actually fairly new to Netflix. I've been there almost a year and I was well aware of what Netflix was up to because I've been following them on the open source world. So it certainly helps for recruiting. If you aren't open sourcing anything that your business, if you have something that's not business critical is not like your special sauce. I highly recommend open sourcing it. It attracts amazing talent. Second challenge and lesson learned is not all tests are created equal. I can't stress testing at all levels. Unit tests are phenomenal or great. I couldn't imagine writing code without writing unit tests but they fall short of simulating what life is like in the real world. So this shouldn't be kind of news to anybody but testing at all levels is very, very important and especially in a continuous delivery environment where you don't have the luxury of downtime or telling customers to come back in three hours while we do this deployment. You need to have heck of a lot of functional style testing and automated canary analysis in many ways is a higher level of testing that I would include like beyond functional. And it requires a significant investment. Writing tests, tests are not cheap. They have to be maintained but they have to be written too. And they break and when they break you have to fix them. So don't overlook the investment in the testing. A lot of people tend to do that unfortunately. All right, results, what have we learned? What can you learn from us? Well, first and foremost, continuous delivery works. It is the cornerstone of all business initiatives at Netflix. It is an unquestionable sacred thing at Netflix. If you're a team at Netflix that for some reason elects not to do continuous delivery, it would never happen actually but it is just essentially assumed by the business. I've worked at companies where the business would come out and say, we wanna get this feature out there and they'd work with me and my team and we'd be like, all right, it might be two weeks. We'll get it out there in the next release. At Netflix, business says, hey, we wanna make this change. Teams at Netflix are like, all right, do you want it today? We'll push this, it may take a developer a couple hours to do it and it'll be in production later on this afternoon, you're cool with that. That's a phenomenal, I think, difference or a differentiator for IT to be able to go back to the business and say, yeah, no problem. Whatever you want, you'll get immediately. To put it into perspective, we've been doing continuous delivery for, again, roughly four years. I'm new to the company. I was brought in about, like I said a year ago. The name of my team is Delivery Engineering. Netflix, I've built a team of seven individuals so there's eight of us building the next generation continuous delivery platform at Netflix. That's how serious the company takes it that they've gone out and hired eight of us and said, we love continuous delivery. It's working really well for us but we want it to work even better so please go off and make it work better. I think that's a testament to how the business views the benefits of continuous delivery. Again, moving fast is a competitive advantage. Don't overlook that, it's not a startup concern either. There's a famous quote, and I shouldn't say famous. There's a phenomenal quote from the CIO of Walmart. Walmart's the number one largest company on the planet. Fortune 500 number one this year and last year. He's quoted as saying, the only differentiation for a company is speed. And I think that's fascinating that Walmart would say that. You know, we tend to think of speed as like, oh, it's a startup thing. Twitter, no, Twitter's not a startup anymore but you know. But it doesn't matter regardless of the size of your company, moving fast is a competitive advantage. Continuous delivery enables you to do that. Continuous delivery doesn't necessarily mean every change goes into production. What it means is any change can go to production. And largely it's, again, it's the ethos of our culture. Again, kind of piggybacking on Michael's keynote. In order for us to survive and continue to be the number one streaming video content provider on the internet, we have to continually innovate.  So moving fast, it's just part, it's ingrained in the culture. Another thing that's really special I think about Netflix is I told you about freedom and responsibility, but also you are free to fail. There's no problem. There's no kind of finger pointing or firings. If you push something so fast out there and it breaks, while we put a bunch of stuff to make sure it doesn't break, it's okay if it does. Because again, the impetus is if it breaks, fix it real fast and let's learn from it. So moving fast, continuous delivery, constant innovation is, again, it's part of our culture. I think it needs to be a part of any culture if you want to embrace this style of software delivery. Okay. Now I want to share with you three things we've learned and we've definitely made mistakes and you're going to need these three things if you wish to go forward. So first thing, global deployments require detailed insight into metrics. In our case, they're core metrics. If you don't have detailed monitoring in your environment, start there. Then you have to figure out what are the actual metrics that actually matter to you in the business or really to the business, because you should be one of the same. In Netflix case, our core metric is streaming starts per second, SPS. We watch this like a hawk. And at any given point, if SPS dips in any region at any time of the day, people are alerted immediately. That is our core metric, because we know over time, well, it's 11 o'clock here, so Europe right now is watching a lot of Netflix. So SPS is on the rise. If you see a dip, we know instantly something's wrong in the European region. Same thing on the East Coast, we watch the East Coast like a hawk and we watch the West Coast. And so continuous delivery feeds into that, right? Because if Europe right now is in the middle of watching Breaking Bad or whatever it is, a team can elect, let's not do a deployment right now. So continuous delivery requires one that you have core metrics and then the ability to schedule things based on those metrics. Initially, Netflix, when we made a change, we pushed it across the globe instantly. Makes a lot of sense for you on the West Coast. This is a great time to deploy on the West Coast. No one's doing anything on the West Coast. We're still all asleep basically. And wearing flip flops. But like I said, Europe probably starting to watch a lot of TV right now because they're getting home from work and that's that moment of relaxation where we really want the experience to be phenomenal. Because again, if you go to Netflix and you're really happy, you're probably signed up next month or whatever or tell your friends about it. But if you get home from work, you've had a hard day, you try to watch Breaking Bad and you get that buffering error all the time.  So, our continuous delivery processes enable scheduling based on these core metrics. So at peak viewing times, unless it's a critical bug fix, many teams elect, let's not deploy this thing into Europe right now. We'll do it into the West Coast, we'll do it on the East Coast. And then when peak time starts dipping in Europe, then we'll do this deployment in Europe. Not all teams do this but some of the critical ones do this or choose to do this. So, understand your core metrics, watch them like a hawk, leverage them for continuous delivery. Next, this is a, don't forget about the cloud. Use the cloud what it's made for and that's elasticity. Cloud by and large has infinite resources, although I'm here to tell you that Netflix regularly finds out where those infinite things stop. AWS is elastic. For most businesses, AWS is truly elastic. Don't forget about that. One thing that Netflix, and this will be interesting because again, Brian is talking about Docker and I think Docker's fascinating and Docker is something Netflix is keeping on looking at very closely. But one thing that we've done and we think it's a lesson learned and something I wish to impart with you all is that we view an instance in the cloud as an ephemeral thing that spins up, it's got our code on it and it could die at any moment. We don't care about that. In fact, we regularly shoot instances in the head just to make sure that we're prepared to survive an outage. And that again is taking advantage of the cloud. A lot of, in fact, the company I was at before this where we had a heck of a lot of load, we would try to squeeze everything out of an instance. So we'd put multiple apps on an instance so we'd fire up this M3X large and we'd put like six apps on it and we'd be like, yes, we're saving money because we have one machine with six things on it, which is fine. Problem is, is that when the third instance or not the third instance, but the third app on that dies, you gotta actually go to that thing, SSH to it and be like, why did it die? And then kick it back up because you don't wanna mess with the other five apps. Netflix's mantra, you put one app on an instance because again, elasticity, there's infinite amount of instances that you can spin up. If that thing falls over, who cares? Shoot it in the head and spin up a new one. And in fact, production SSH is largely never used. No one really ever SSH to a production machine.  Now if it continues to keep dying, then maybe someone will SSH to it and figure out what's going on. But, so take advantage of the cloud. The true elasticity of the cloud is phenomenal. Spin up resources and then tear them down as needed. Single tendency works really, really well in this model. Obviously in a data center, you're constrained. You don't really have truly elasticity or true elasticity. So something like Docker may help. I think the Docker equation in the cloud gets really, really interesting and kind of muddy. But I will say right now in the Netflix view of the world, single tendency works phenomenally well. Also note, since everyone here rose their hand for AWS, pricing at AWS is linear. So if you want to squeeze six apps onto let's say a six gig machine, you could actually get six machines or let's make math easier. Two apps on a six gig machine is roughly the same cost as two machines with three gigs. Now, it's not always true with like CPUs and whatnot at Amazon, but the pricing model is linear. So let's keep that in mind. Finally, I forgot when we end. 115, oh sweet. This is where we're going. This is a lesson we've learned and it's version all the things. Okay, so what do I mean by that? I told you that from an automation standpoint we have Gradle or just think of a build file regardless of the technology you use. We have a build file that delineates how your software is or how your project is compiled, tested and packaged. And then we have these other tools like Asgard I mentioned and automated canary analysis. There's all these other tools in the pipeline that actually have information about your app and what it should do. If you think about that, that source of truth is spread across multiple instances. You have three sources of truth in that example. You have your build file, you have Asgard and you have this other thing called automated canary analysis. The problem is that anyone can change that source of truth and not necessarily know about it. And in fact, you could go into Asgard and say my particular app now needs, it's got 200 instances. I'm gonna bump it up to 300. What we've discovered is that that's really, that's fine and well, that's great. People can do that and people do that all the time. Problem is we lose the history of that. We don't know why you went to 300. We have an event that we know it went to 300 but we don't know why you did that. But think about it. If you were to go into your build file or into your code and you changed 200 to 300 you would commit it to Git or whatever Perforce CVS. You'd probably leave a commit message saying you need to go to 300 because we've noticed that this thing falls over at 9 p.m. So what we're doing is largely collapsing that model of what it means to be deployed across the cloud, what it means to be analyzed in a production environment and folding that into essentially our build file. And so that becomes a single point of truth. Single entity that defines how this app is realized in production. Think about it. It may sound kind of weird but it's no different than your build file, right? Your build file says how your code should be deployed so why not, or not deployed, how it should be compiled? Why not add all the other information there like it should look like this in production? It should have this many instances. It should have this kind of load balancer in front of it. It should be running on this type of instance. It should have these thresholds when you do an automated canary analysis in that. And all that is stored in one place that engineers can go to and version and when you make changes to it, everyone knows why. The threshold for automated canary analysis used to be, let's say 700. We've now lowered it to 650. We have a record of who did it and when and why. This is very important. So if you, again, from a continuous delivery standpoint, you're gonna end up stitching together various systems and each system will have its own source of truth. That'll become somewhat of a maintenance nightmare as you grow and new people come on, people leave. People are going to lose track of why we made a decision in that upstream system. So we're moving kind of, again, that source of truth down to one place and then those upstream systems will use that place to determine, okay, what am I supposed to do here? Immutable environments is kind of a hot kind of DevOps term these days and this is largely in line with that notion. So version all the things. So last but not least, I did try to leave a couple minutes for questions. That's my contact information. Thank you very much for your time. I hope I imparted some nugget of truth or not truth. Well, definitely truth. Lots of nuggets of truth. Hopefully something you can take back to your company. Companies. There's gotta be some questions. Come on, I'll help you. For the verify stage, and on the topic of versions, how does, where do you get the tests for that particular version of that Amy? Amy, sorry. So the tests are part of the code base? So they're in Git somewhere? Yes, they're part of the code base and then Jenkins and other kind of like, like Meizos, those type of tools are then running those tests against the AMI in that cluster. Are they versioned together separately? Oh yeah, yeah, definitely. How do you manage compatibility between components across the sort of deployment boundary? Components like services? Service interfaces, data storage, the whole thing. Yeah, so, well data storage, at the end of the day, Netflix leverages Cassandra and you go through a series of services to actually get data in and out. The larger issue is service contracts between, like I said, Netflix as a SOA. How we've solved it up to now is every team provides a client jar, essentially. So a binary that, okay, so I have the service foo and you need to make use of foo. So you come to me and you say, let me have your client library so I can talk to you. So every team is binary, kind of independent and they can run together, you know, run at separate paces and the contract is through that client library. That works, but it's being kind of reevaluated. You know, without client libraries, then you're stuck with the URL, right? The RESTful interface and people try to version those. But then the payload could change, you know, I'll change this, Jason, blah, blah, blah. And that's not where we're going. What we've found is that those client jars tend to introduce a whole lot of other dependencies that then can break.  So we're building some tools to understand kind of the dependency analysis between services. So that's how we do it. I'm not saying it's foolproof, but it largely works. So where does a QA fit into all of this? With the huge robust testing suite, where does like a human QA actually fit into this? They don't. So let me explain. The client side teams, like I'm pointing here like there's a Netflix box there, sorry.  And they're regularly basically, that's probably the best job in the world. They watch Netflix all day. And they're watching Netflix in these various environments. But those are the only teams that have QA. There is no formal QA on any of the engineering, like basically server side, all the services that are published. And there's roughly more than 600 services running in production. And not one of them has a QA team that QAs it. It's all engineers. You, again, are responsible for what goes in production. And the belief is that if someone else, if I'm an engineer, I throw it over the fence and give it to QA, then there's this, remember he said something like people are motivated? I forget his quote, but essentially you'll do changes or you'll make changes, depending on how you're motivated. And that's the belief. If I know that what I write today will be in production in a couple hours, I'm gonna make real sure that what I wrote works really well. I'm gonna test the bejesus out of it. Because again, I don't wanna get that call at 3 a.m. So there are really no QA teams at Netflix per se. There is a client side QA team. And again, because you can't automate a lot of that stuff, like someone's gonna sit down and actually watch, do a search and watch and make sure Breaking Bad goes from end to start to end, excuse me. If you haven't, yes, I really love Breaking Bad. Probably say like House of Cards or something, but yeah. Hey, so I was wondering if Netflix is gonna put out like a new service or something like, hey, let's show you a montage of all the videos you've watched over the year or something. How do you establish a baseline or something that doesn't exist? How does the canary testing work for something else? That's great. We basically wing it. A team will wing it and just basically say, all right, we think this is what the baseline is and we'll put it out there in production and find out real quick. So you're right, you largely guess what the baseline is, especially with automated canary analysis. Now, there's a lot of institutional knowledge at Netflix that understands basically the type of app you're building, how much load it's gonna get. I mean, again, we watch our metrics like hawks, so we know that if you spin up a service at this time in this region, you should expect about this much load. So you can largely guess it and then you watch it like a hawk going forward and you refine it. There is a performance team at Netflix that essentially acts as consultants and they'll sit down with teams. Because again, at the end of the day, engineers at Netflix are super awesome at building a service that whatever fires up my little pony quickly on your iPad, but may not understand the implications of the performance of an iPad or the servers that it's coming off of. So that's where that performance team can kind of help in a consultative basis and they'll largely help you to find those initial thresholds. One more question. Is your CD system in the cloud too? No, so the DVD, oh, CD system, I'm sorry. Continuous delivery, yes, of course, yes. I'm sorry, I thought you meant CD as in DVD. No, our continuous delivery system does, the platform is in the cloud. Like I said, right now it's a series of tools. One of them is open source called Asgard. We are rebuilding that. And that tool, that platform is called Spinnaker. It is completely in the cloud. It is completely dog food. So we use Spinnaker to basically continuous deploy. Spinnaker and all the teams at Netflix are using Spinnaker to continuous deploy. But yeah, it definitely leverages the cloud. Netflix is essentially 100% in the cloud. Yes, he said, I'm done. Crane or whatever. Thank you very much.