So good morning. First of all, just an announcement, my lecture slides from yesterday are now all online on the course websites. And today's and tomorrow's lectures will be posted as soon as I get them into PDF format. So if there's any slide that you want to refer back to, you can just download it online and not have to scroll through the entire YouTube video of my talks. As a reminder of where I left off yesterday, we were talking about using supernovae as standard candles to attempt to put constraints on omega and matter and omega and lambda in an assumed lambda plus matter cosmology. As you see, it puts some interesting constraints on the error ellipse that you derive from the fluxes and red shifts of the type 1A supernovae puts you squarely into the accelerating region of the omega matter, omega lambda plane. But as you see, the ellipse straddles the kappa equals 0 line, the one that goes diagonally from corner to corner. So the supernova data in themselves don't tell you whether the universe is positively curved or negatively curved. We're going to need another source of information that contains information about omega matter and omega lambda in some other combination. And that brings us to not standard candles, objects of known luminosity, but standard yardsticks, sometimes called standard rulers, sometimes called standard meter sticks. These are simply objects whose physical size you know. That is, you know the proper distance from one edge to the other of one of these objects. So here's the observer looking at, in this case, not as to God, ruler. It's at a co-moving coordinate distance r, and it's a line so that the angular distance between its ends, as seen by the observer, has some value delta theta. I'm calling it delta theta because it's probably going to be a small angle, because we're going to be looking at things that are far away. So this standard yardstick emits lights. Otherwise, we wouldn't see it. And we're assuming that you can measure its redshift. Maybe it has absorption or emission lines. So the redshift z is measurable. And you're assuming that it's also resolved an angle so that you can measure this angle delta theta from one edge of the standard yardstick to the other. Well, if you know the physical size of the object from one edge to the other, you can compute a function that's cosmologists call the angular diameter distance, because you derive it from the angular diameter. It's just defined as the physical size l of the object in megaparsecs or meters, whatever your favorite units are, divided by the angular diameter of the object, delta theta measured in radians. This function is defined in this way because, of course, in the small angle limits, the angular diameter distance d sub a is equal to the proper distance d sub p in the limit that space is Euclidean and static. So just as the function that we call the luminosity distance is equal to the proper distance in the limit that space is static and Euclidean, this function will tell you the proper distance if space is Euclidean and static. If space is not Euclidean and static, well, if it's space that's described by Robertson-Walker metric, you can use the Robertson-Walker metric to find the distance d sub s between the ends of the yardstick at some fixed time t. And let's call it t sub e, the time that the light we observe from that glowing standard yardstick was emitted. Well, from the Robertson-Walker metric, it's the scale factor at the time the light was emitted, this function s sub kappa, which is equal to the co-moving coordinate distance r if space is Euclidean, smaller than r if space is positively curved, greater than r if space is negatively curved. And you multiply that by delta theta, and that tells you the distance between the ends of the yardsticks at the time the light was emitted. But of course, by hypothesis, this is a standard yardstick and we know that distance l. So we can now compute what the angular diameter distance is for an object in space described by Robertson-Walker metric. It's just the scale factor at the time the light was emitted times the function s sub kappa. Since scale factor at the time of emission is 1 over 1 plus z, the observed redshift of the object, this gives you an interesting relation between the angular diameter distance that you compute for a standard yardstick and the redshift z that you measure for your standard yardstick. And of course, if space is nearly flat and we have observational evidence that we do live in a nearly Euclidean space, then the angular diameter distance is just equal to, well, in this case, the function s sub kappa is just r, the co-moving coordinate distance, which we've normalized. So that's just the proper distance today. We call the real distance, the distance you would measure if you could stop the expansion of space and stretch up perfectly tauts, measuring tape between you and the distant object. So z is greater than 1 in an expanding universe. And so the angular diameter distance that you compute for a standard yardstick in a flat space is always going to be an underestimate of the proper distance to that object. As you'll recall, the luminosity distance that you get from a standard candle is always going to be an overestimate of the proper distance. However, it's an underestimate or overestimate that you know because you know the redshift z of this object. And, well, the proper distance is something that you can compute in a space with known omega matter and omega lambda. And here the plot is the computed angular diameter distance in three Euclidean universes. The bottom one, that's where all the matter is, all the energy density isn't matter. Omega matter equals 1. The top one, all the energy density, isn't a cosmological constant. Omega lambda equals 1. And the intermediate one, that's our benchmark model with omega matter of about 0.31 and the rest in lambda. So you see the angular diameter distance starts out being linear in z, Hubble's law. But it bends over. Notice that in the case of a lambda-only universe, a decidary universe, the angular diameter distance continues to grow until it asymptotes at 1. Here we're measuring distances and units of the Hubble distance. But for a universe containing matter, all matter or partially matter, the angular diameter distance that you measure has a maximum at some intermediate redshift and then decreases again as you go to higher redshift. I have a question. He's asking about clumpy matter. If you have clumpy matter, you have gravitational lensing effects, which are interesting, but not part of this talk. If you are in a universe where everything becomes homogeneous and isotropic when you smooth on large enough scales, if you look at objects at proper distances, larger than the homogeneity scale, then it averages to this. You need a standard yardstick, an object whose physical size you know. A good standard yardstick is hard to find. People have tried using galaxies or clusters of galaxies as standard yardsticks. But there you have evolution of galaxies, evolution of clusters with cosmic time. It becomes really annoying. However, the best standard yardsticks that cosmologists have found are the hot and cold spots on the cosmic microwave background. Remember if you look out in space, in microwave frequencies, you look through the recent transparent universe, and you're basically looking at a last scattering surface when you look at the cosmic microwave background. Astronomers call it an inside out photosphere. When you look at the sun, you are looking at a relatively thin layer where the photons undergo their last scattering from free electrons. And similarly, when you look at the last scattering surface in microwaves, you're looking at a relatively thin layer where the photons of the cosmic microwave background underwent their last scattering from a free electron. Now, I'm going to make an assertion that you will have to accept for about 20 or 30 minutes. I'm going to assert that the typical CMB photon last scattered from a free electron when the temperature of the universe was a little under 3,000 Kelvin around 2,970, and I will, in fact, justify that temperature later on. That's the temperature at which the number density of free electrons drop sufficiently low. That's the probability of a photon scattering from it became less than 1. So this is the result of, as we'll see a little later on, a very well understood physics. It's an equilibrium situation. All the calculations are simple. So this is a temperature that we know quite well. We also know very well the temperature of the CMB today, 2.7255 Kelvin, a number that I am probably muttering in my sleep by this point. And so we can now compute the red shift of the last scattering surface, 1 plus Z sub Ls, the red shift of the last scattering surface, just the inverse ratio of the scale factors, which is just the ratio of the temperatures, the higher temperature back then divided by the lower temperature right now. And it turns out to be 1,090. So although the spectrum of the cosmic microwave background doesn't have those convenient absorption and emission lines, from which we measure the red shift of galaxies, because it involves very well understood physics of recombination, we can compute the last scattering red shift very well. And in fact, the Planck 2015 results tell us that the red shift is 1,089.90 plus or minus 0.23. So we know the red shift part of this equation very well. Now we just need to know the angular diameter distance. It would be convenient if the anisotropy map of the cosmic microwave background were imprinted with nice circular, well-defined polka dots, all of easily measured angular scale. But as you see here, it's actually a random Gaussian field. That's OK because we can break down the temperature fluctuations delta t in spherical harmonics. And for each spherical moment L, we can find the correlation function of the temperature fluctuations. And you saw this or a version of this yesterday a couple of times in the dark matter lectures. Remember, low angular moment L means large angles. And as you go left to right, you're going to smaller and smaller angles. So the correlation function does have a very well-defined peak. So there is a preferred angular size in this problem, L of about 220. This highest peak, the first peak in the correlation function, has delta theta of about 0.8 degrees, or expressed in radians 0.014. So it is, in fact, a small angle. And probably, especially from the back of the room, you might not be able to persuade yourself that there's a preferred scale such that there are, let's see, about 400 dots from side to side. It's there in the correlation function. So the temperature fluctuations, those hot and cold spots on the cosmic microwave background, provide a very useful statistical yardstick. You have to do the statistics, complete the correlation function, but there's that very clear peak at an angular size of 0.8 degrees. So we can measure delta theta from the cosmic microwave background to compute the angular diameter distance. We need to know what physical scale that peak in the temperature correlation function corresponds to. Well, fortunately, as I said before, the physics of last scattering and the production of the cosmic microwave background is pretty well understood. And that first peak in the correlation function results from the presence of standing acoustic waves in the photon baryon fluid that filled the universe before recombination. And I've put up this cartoon version of recombination. To remind you that, on the left, before electrons and protons combined to form neutral hydrogen atom, the photons and the free electrons frequently interacted, colliding through the Thompson cross-section of the free electron. So the photons and electrons were coupled together. And the electrons were coupled to the free protons by the electric attraction between them. And so photons, electrons, baryons, in this case, the protons, are all coupled together into a single fluid, the baryon photon fluid it's called. And this fluid, like any sort of gas or liquid, supports sound waves. So the baryon photon fluid is affected by gravity. It falls towards the minima in the gravitational potential provided by the dark matter. And as the photon baryon gas falls inward, it becomes compressed to higher pressures. It becomes over compressed and bounces back. And then it starts to oscillate in and out. And of course, I have to move my arms back and forth while discussing baryon acoustic oscillations because it's just something that you do. So you have these standing acoustic waves. And well, what is this going to mean for the cosmic microwave background? When the universe becomes transparent, those regions where the compression was at a maximum are going to have a higher energy density, and thus a higher temperature, as seen in the cosmic microwave background today. And those regions where the photon baryon fluid was at maximum refaction, those are low energy density, and will be seen as the cool spots in the cosmic microwave background today. So the first peak in the temperature correlation function represents those regions where the photon baryon fluid had just enough time to reach maximum compression by the time of last scattering and not bounce back at all. So that's where you have maximum compression and the highest temperature fluctuations. And well, that's to go into the details. But if there's going to be a physical size L associated with these standing acoustic waves, well, the ones that have just had time to reach maximum compression will have been compressed over a distance comparable to the sound horizon distance at the moment of last scattering, just as the particle horizon distance is the maximum distance that a relativistic particle, like a photon, can have traveled by a time t. The sound horizon distance, by analogy, obviously, it's the maximum distance that a sound wave can have traveled. So it's the same formula that you use to find the particle horizon distance. Only you're substituting in the sound speed, c sub s, s for sound, instead of the speed of light. And since the sound speed can be a function of time, you put it inside the interval. Now, in theory, the sound speed is a function of time. In practice, by the time of last scattering, the energy density of photons is still higher than the energy density of the baryons. It's less than the energy density of the dark matter, but baryons, of course, provide only about a fifth or a sixth of the total rest energy of the matter. And so at the time of last scattering, photons are still dominant over the baryons. And we can approximate the sound speed of the photon baryon gas as being the sound speed in a pure photon gas. The sound speed slows down a little bit as you get closer to last scattering because of the contamination of those massive baryons. But it's a reasonable approximation. And when you do this calculation of the sound horizon distance at the time of last scattering, it comes to about 0.145 megaparsecs. So this is a result that has some dependence on omega matter because that enters into the scale factor that you're integrating over. But if you do the more accurate calculation, you get a quite accurate physical distance L for the physical size of that peak in the correlation function for the cosmic microwave background. So easily measurable delta theta, 0.8 degrees, a length for your standard yardstick that you can calculate from relatively simple physics. It's all about electrons scattering from photons and nice everyday simple physics by comparison to some things we'll be talking about over the next couple of weeks. So standard yardstick, you can compute what the physical size must be associated with the first peak of the CMB correlation function. You can measure the angular size, 0.014 radians, and you do the division. And it conveniently comes out to about 10 megaparsecs, which of course is a shockingly small distance for something at such a huge redshift. But remember, this isn't the distance today. It's the proper distance divided by 1 plus z, assuming space is nearly flat. And so it's actually the proper distance at the time of last scattering, the time that those photons underwent their last scattering. In fact, they were 10 megaparsecs away from us, proper distance. And then they just traveled along lines of constant phi and theta to reach us today. We have an angular diameter distance. We know the redshift very well. And now the problem is just to find those values of omega matter and omega lambda today that yield an angular diameter distance of 10 megaparsecs. So the very useful thing about using the hotspots of the CMB as standard yardsticks is that it turns out the angular diameter distance is quite sensitive to the curvature of the universe. So using the CMB for a standard yardstick gives you a very sensitive probe of the sum of omega lambda and omega matter today. If their sum is greater than 1, that means we're in a positively curved universe. And positively curved space acts as a magnifying lens. It makes the hotspots on the CMB look larger in angle and thus gives you a small angular diameter distance. So you make the curvature too big or rather the radius of curvature too small. And you are unable to get an angular diameter distance of 10 megaparsecs. You end up getting smaller angular diameters. Conversely, if omega lambda plus omega matter today is less than 1, that means we live in a negatively curved universe acts as a de-magnifying lens. And you get smaller angular sizes for those hot and cold spots on the CMB. This means that you calculate an angular diameter distance that's going to be too big. So the angular diameter distance versus redshift test is quite sensitive to the sum of omega matter plus omega lambda. There are other physical considerations that enter into the calculations. But the net result is this. This very elongated, very skinny ellipse is the 95% confidence interval for the CMB results using that first peak in the cosmic background's temperature correlation function as a standard yardstick. You'll notice it's not lying exactly along the kappa equals 0 plane. There are other, as I mentioned, other considerations that enter into the calculation other than purely the curvature of space. However, you'll notice the red ellipse using the first peak as a standard yardstick is nearly orthogonal to the blue ellipse using type 1A supernovae as standard candles. They're measuring different combinations of omega matter and omega lambda. This is good. And it's the combination of the supernova standard candles and the CMB temperature peak standard yardstick that leads to a spatially flat or nearly flat benchmark model with omega matter of about 0.3 and omega lambda of about 0.7. So CMB alone tells you, yeah, kind of flats. Supernova alone says, yeah, it's accelerating. It's the combination of the two that gives us a model for the universe that is both flat and accelerating. From one point of view, yay, life is good. Everything fits together neatly. In what I call a benchmark model, sometimes people refer to a concordance model or a consensus model. But I like to emphasize that it's a benchmark. It's a standard for measuring things in the universe, like distances and distance measures of different kinds. It's consistent with the various observations we have of the supernova and the cosmic microwave background. And so what's in it? My first caution to you, don't memorize all this. The exact numbers are, I'm certain going to change with the next func data release. But let's look at it as a benchmark. The photons of the cosmic microwave background we've already seen provide just a small fraction of the density today, 53.5 parts per million. The neutrinos, well, important footnote, if neutrinos were massless, then they would provide 36.5 parts per million of the energy density today. Neutrinos do have mass. The neutrino oscillations do tell us that the mass states are different from each other. And therefore, they can't all be zero. And so today, neutrinos have defected to the matter army from the radiation army. But they did so only recently. They count as hot, dark matter. And so back when the universe was radiation dominated, the neutrinos were, in fact, relativistic and were radiation back then. The baryonic matter density, 4.8% of the total energy density today. Again, later in the lecture, I'll show very briefly why we expect this number to be 4.8%. The total matter combination of CMB plus supernova says 0.31 is in agreement with observations. So 0.31 minus 0.048 means that about 26% of the energy density today is provided by dark matter, the bulk of which has to be cold rather than hot. The benchmark model, I assert this is my model, so I can make it perfectly flat. So I assert that my benchmark model is perfectly flat. Everything sums to 1 here. And so omega and lambda is 1 minus omega of everything else since radiation is such a small contributor. We can say that the cosmological constant has omega of 0.69 today. So again, don't memorize this. If you want to refer to it later, my notes are online. But this is the combination of omegas that I'm going to be using for this talk in the next one. So the cosmological principle says there are no special locations. The interest is homogeneous once you average on large scales. And the length scale I use is 100 megaparsecs, a suspiciously round number. I'm just saying with that statement that once you smooth on scales of 100 megaparsecs, the remaining density fluctuations are very small in amplitude, much less than 1. There are, however, special times. Here we have the energy density of the different components as a function of scale factor. Early in the universe, when the scale factor was small, the radiation whose density goes as scale factor to the minus 4 was dominance. Later, the matter density, which falls off as a to the minus 3, was dominance. And now where the scale factor is 1, you'll note we've only just entered the region where the cosmological constant lambda is dominant. So the radiation falls as a to the minus 4. Matter falls as a to the minus 3. So you can compute the scale factor of radiation matter equality. That's just the ratio of the radiation omega to the matter omega today. And for my benchmark model, it's around 3 times 10 to the minus 4. Well, with matter falling as a to the minus 3 and the cosmological constant, by definition being constant, you can compute the scale factor of matter lambda equality. It goes as the ratio of their omegas to the 1 third power because they differ by a factor of a to the 3 in their dependence. And once you take that cube root, you find that this is a scale factor, not all that much smaller than 1, about 0.766 for the benchmark model. Here's the energy density of different compounds as a function of scale factor with the special times of radiation matter and matter lambda equality marked out. You can, once you have models, once you have values for the radiation density, the matter density, and the lambda density today, plug them into the Friedman equation, which tells you how the scale factor varies with time. So now you integrate and you find what the scale factor is as a function of time. Notice, before radiation matter equality, you had a radiation dominated universe. Only the radiation term was significant on the right hand side of the Friedman equation. That gives you a power law dependence of scale factor on time. And it's t to the 1 half power for a radiation dominated universe. In the middle of the matter dominated era, again, you have only one term significant on the right hand side of the Friedman equation. And you get the power law t to the 2 thirds for the growth of the scale factor in the matter dominated epic. And finally, today, we're entering the lambda dominated phase where, eventually, the scale factor will grow exponentially. Notice on this plot, there's a bit of a coincidence problem. Isn't there? Now, t sub 0 is really close, at least on this logarithmic plot to the time of matter lambda equality. And there's been some discussion of, well, is this just a coincidence or is there some deep physical meaning? I don't know. Now, when the universe was dominated by radiation and by matter, the expansion was decelerating. Once lambda became significant, the expansion started speeding up. And amusingly, the period of deceleration almost exactly cancels out, the later period of acceleration. And it turns out that the age of the universe t sub 0 is really quite close to the Hubble time. About 95.5% of the Hubble time for the benchmark model. So in fact, in the very first lecture when I said age of the universe must be twiddled the Hubble time, that was a pretty good twiddle. So I promised that today I would be talking about special times in the history of the universe. We've thrown away the perfect cosmological principle. We allow certain times to be special. And by looking at our three-component benchmark model, radiation, matter, and lambda, we've identified a couple of special times, the times of radiation matter equality and of matter lambda equality. And from the Friedmann equation, we can compute the dependence of scale factor on time. And so for a given value of the Hubble time, like 14.4 giga years, we can compute for the time of radiation matter equality, both the red shift, 1 over the scale factor, minus 1, and the corresponding cosmic time. So radiation matter equality took place at a red shift of about 3440. That corresponds to a time of 50,000 years, a lot smaller than 14.4 giga years. But of course, the radiation-dominated epic of the universe, those first 50,000 years, is when the energy per particle was highest. There were a lot of interesting physics going on in the early universe in the first 50,000 years or so. Matter lambda equality, you compute the red shifts from the scale factor, and it's only a red shift of 0.3 or so. Not very large. And most modern red shift surveys of galaxies can go out to Z of 0.3 or so, no problem. And it corresponds to a cosmic time of 10.4 giga years. So if you compare that to now, where the age of the universe is 13.7 giga years, we find that matter lambda equality was about 3.3 giga years in the past. So you can judge for yourself how much of a coincidence that is. Now, I said the times of radiation matter equality and matter lambda equality are special. They're not that special. Now, so now is not particularly special. The development of intelligent life on one particular planet doesn't have any cosmological consequences. We can produce dramatic changes in the climate here on Earth, and it looks like we're proceeding to do just that. However, we still haven't screwed up Mars, for instance, much less anything 100 megaparsecs away from us. For radiation matter equality, matter lambda equality, they're not all that special. You look at the curve of scale factor versus time, and there isn't an abrupt inflection in the scale factor curve at those times, just a gradual change. If you look at galaxies at z greater than 0.31 and galaxies less than 0.31, there's no dramatic change in their properties at that particular epic. They don't say, oh my god, lambda's now dominant. I have to do something different. But it's a smooth gradual change. And so today, when I'm talking about special times in the universe, I'm going to talk about extremely special epics where you have a dramatic change in properties. The time of last scattering, when you went from having free electrons and free protons to having neutral atoms, the time of Big Bang nucleosynthesis, when you went from having free protons and free neutrons to having atomic nuclei with multiple baryons in them. And I will very briefly mention the epic of inflation early in the history of the universe. But I'll just give a quick motivation. And you'll have to wait later for the lecture series on inflation to learn about the details. OK, last scattering. I stood here and said in my most authoritative voice, last scattering took place at a temperature of 2,970 Kelvin. And so you look only moderately convinced. But we can go through some relatively simple calculations to compute the time of last scattering. And in order to do my calculations, I'm going to make one simplifying assumption that is blatantly not true. I'm going to assume that the baryonic component of the universe is all hydrogen. Last scattering occurs well after Big Bang nucleosynthesis. So there would have been. There was helium back then. However, I'm just going to concentrate on hydrogen. The math becomes a lot simpler. And it captures all the essential physics. So this is the only assumption that I have to apologize for. I'm just doing it to keep things simple. It is useful to discriminate in studying how the universe became transparent. It's useful to discriminate between recombination and last scattering. Sometimes people use them in a very casual, interchangeable manner. But their underlying physical meaning is different. The epic of recombination is that time in the universe when the fractional ionization of hydrogen fell to exactly 1 half. Fractional ionization started out as 1. All the hydrogen was completely ionized. It fell to 0. All the hydrogen was in the form of neutral atoms. So at some points in between, there was one instant when the fractional ionization x was equal to 1 half. That's equal to the number density of free protons. And so p divided by the number density of free protons and neutral hydrogen atoms, h added together. I'm assuming nothing but hydrogen. So the denominator there is the number density of baryons. And for charge neutrality, the number of free electrons per unit volume has to be the number of free protons per unit volume. Again, assuming nothing but hydrogen. The epic of last scattering, I've already defined it. It's when the typical CIB photon last scattered from a free electron. So this obviously has to happen at some point when the fractional ionization is greater than 0. There have to be free electrons for the photons to scatter from. But there's no particular reason why the last scattering has to take place at x of exactly a half. So keep this distinction in mind. And I will try to speak carefully so as not to confuse the instant of recombination with the time of last scattering. They are closely related, but they're not identical. So when did recombination occur? When did last scattering occur? They're not exactly the same question, but in order to answer the question when these two epics occurred, we need to find what the fractional ionization is as a function of time, or equivalently as a function of scale factor. And we have to find, in addition, at what value of x does last scattering occur? Does it occur at x of a half or at x of 99% or x of 1%? We don't know, but hey, we can calculate. So assumption, everything is hydrogen. I'm also going to assume with more justification this time that the fractional ionization x is determined purely by a balance between photoionization and radiative recombination. So here's the relevant equation. Hydrogen atom H plus photon of sufficient energy goes to free proton and free electron. That's photoionization. In reverse, free proton plus free electron combine to form a neutral hydrogen atom, and the excess energy is taken away by a photon. Assuming that the universe, at least its baryonic component is pure hydrogen, makes things simple, because now there's only one energy scale that's relevant to the problem. The ionization energy of hydrogen from its ground state, 13.6 electron volts. Or expressed in Kelvin, that's q divided by Boltzmann constant, about 160,000 Kelvin. I feel a little bit weak, a little bit low in energy, talking about 13.6 electron volts after that cosmic ray lecture. That talked about 10 to the 21 electron volts. But one of the great things about physics is that there's interesting physics going on at a very wide range of energies. So I'm assuming it's all about photoionization and radiative recombination.  There are about 1.6 billion photons for every baryon in the universe. That's true today, even after stars have been cranking out photons. And it would have been true at the time of recombination and of last scattering as well. So collisional ionization by bumping into another hydrogen atom, you can forget about it. It's just wildly implausible. You're more likely to be ionized by colliding with a photon than with another hydrogen atom. So the temperature scale of the problem, 158,000 Kelvin. But I just asserted in my most authoritative voice that the temperature at the time of last scattering is below 3,000 Kelvin. That's less than 2% of the temperature scale q over k that's built into the problem. So why was the temperature of last scattering so very, very low? Well, this is where we do our calculations. I'm going to make more assumptions, very strongly justified this time. Before the time of last scattering, the hydrogen atoms, the free electrons, the protons, the photons were all in a state of kinetic equilibrium, sometimes known as thermal equilibrium. Because before last scattering, all these different particles were frequently interacting with each other. They came to the same temperature T, very useful. You can use the same temperature for all these different particles. And more specifically, kinetic equilibrium says that the distribution of energy and momentum for each particle type is given by either a Fermi-Dirac distribution or a Bose-Einstein distribution, depending on the particle spin. So kinetic equilibrium can be assumed so the photons had a Planck distribution, the hydrogen atoms, electrons, protons, non-relativistic during this epoch, therefore had a Maxwell-Boltzmann distribution of particle momentum. In addition to kinetic equilibrium, we can safely assume that the equation for photo ionization and radiative recombination was in chemical equilibrium. That is, in any given volume of space, there were, on average, as many photo ionizations as radiative recombination. So the arrows were going equally well in both directions in this reaction equation. Now, obviously, this isn't absolutely perfectly true. The whole point behind there being an epoch of recombination is that there's this gradual drift from right to left. So you start out with protons and free electrons. You end up sometime later with all neutral hydrogen atoms. But this transition was sufficiently gradual that in any given instant, you can assume that the equation is in chemical equilibrium. With these assumptions, then the number density of electrons and protons and hydrogen atoms is given by the Saha equation. I was thinking of just slapping down the Saha equation and going from there. But it's important to keep in mind the assumptions that go into the Saha equation. Saha derived this Saha equation in the 1930s, if I remember correctly. And he was looking at the ionization states of stellar atmospheres. So astronomers once again provide us with interesting physics. Notice number density of hydrogen atoms divided by the product of the proton and electron density. It depends upon the temperature T and the ionization energy. Again, the only energy that's built into this problem. As T goes to 0 as the energy drops, this right-hand side becomes increasingly large. And hydrogen, neutral hydrogen atom, dominates over free protons and free electrons. So that's the Saha equation. You can justify using it due to these equilibrium conditions. And we can rewrite it in terms of that, which we really want to know, the fractional ionization x. Number density of neutral hydrogen goes as 1 minus x. Number density of free electrons goes as x. And I've taken the number density of protons off to the right-hand side. So off in the left-hand side, that which we want to know, the fractional ionization. Off on the right-hand side, the temperature, OK, the ionization energy, we know that. And number density of protons. We need to get rid of that factor and sub p. We can do that with a reflection that's the baryon to photon ratio expressed as number eta is constant. And with 1.6 photons per baryon, the inverse of that is 6 times 10 to the minus 10. Really small number. Keep that in mind. So number density of protons, fractional ionization times the total number density of baryons. We're assuming everything's hydrogen. That's eta times the number density of photons. And the number density of photons. It's a Planck distribution. So you integrate up, and it goes as t cubed. Plug that back in for the number density of protons, and you have something that looks good. Because you have that, which we're searching for, the fractional ionization on the left-hand side and on the right, something that is a function of things that we know, plus the temperature, plus eta. Now, I've asserted that eta is 6 times 10 to the minus 10. But you'll notice there's this exponential temperature dependence and only a linear dependence on eta. So if you solve this quadratic equation for the fractional ionization x, it's going to be only weakly dependent upon the baryon to photon ratio eta. And OK, you can solve it. It's a quadratic equation. You take, obviously, the positive roots. Fractional ionization is a positive number. And you get this very nice curve, d-d-d-d-d. Starting out at a fractional ionization of about 1 at a redshift of 1,600. Time goes from left to right here. You're going to lower redshift. And by the time you get to a redshift of around 1,100, the fractional ionization is very small indeed. So when is x equal to 1 half? Well, it's at a redshift of about 1,380, or a temperature of 3,760 Kelvin. And with our benchmark model going from this redshift to time, it's when the universe was a quarter of a million years old, 250,000 years. And I mentioned this result is weakly dependent on eta. Here I'm assuming the standard value eta of 6.1 times 10 to the minus 10. But changing eta only wiggles the curve back and forth a little bit. So we can now pinpoint the time of recombination. You'll notice it's a gradual transition. Going from, let's say, x of 0.9 to x of 0.1 is a sufficiently short time that people talk about the time of recombination. It's actually a process so that at any time you can assume equilibrium. But on a cosmic time scale, it doesn't take very long at all. People ask you, when recombination occurs, it's simply you can assume equilibrium. We can assume the Sahi equation holds true at any time. It starts to break down as you get to low redshifts, simply because, well, the photons start to decouple for the baryon components. And you can no longer assume equilibrium. But at x of 1, half, sure, make those equilibrium assumptions. So if we just find recombination as x equals 1, you can compute when recombination occurred. But we also want to know when the last scattering of a typical CMB photon occurred. Well, you can do this by comparing the rate of photon scattering, the number density of free electrons times the relevant cross section, the Thomson cross section for the electron, times the speed of the photons relative to the electrons, which is, of course, c. So the number density of electrons decreases with time, in part because of the dilution effect. It goes as 1 over the cube of scale factor, and in part because the fractional ionization drops as a function of time. The other rate that we have to compare this to is the rate of expansion given by the Hubble parameter. In general, you have to take into account the radiation contribution, the matter contribution, the lambda contribution. But a time of a quarter of a million years, that's well into the time of matter domination. Remember, matter took over from radiation when the universe was about 50,000 years. This is 250,000 years. So it's a good approximation that the Hubble constant only cares about the matter term. So it goes as 1 plus c to the 3 halves power. So during the matter dominated epoch, the Hubble constant drops fairly gradually as scale factor to the minus 3 halves power. But the rate of photon scattering plummets. It goes as a to the minus 3 times the fractional ionization, which is, as you've seen, dropping quite rapidly in this epoch. And so we can pin down when last scattering occurs. And in fact, it doesn't occur until a redshift of 1,090. This is when things are starting to fall out of equilibrium. So to do it right, as the Planck 2015 result does, you have to do the out of equilibrium calculations. But notice this is at a redshift smaller than the redshift of recombination. And it's when the fractional ionization of hydrogen drops to 0.007, less than 1%. So why does last scattering occur at such a low temperature? Number one, because of all of those photons per barrier, recombination is delayed until all of those photons at the higher energy end of the higher energy v-tail are doing the ionization. And even then, last scattering, because of the different parameters involved, the size of the Thomson cross-section and so forth, is delayed until the fractional ionization is much smaller than a half, when it's 0.007. So the last scattering hangs on until a redshift of 1,090 or so. I said it is 6.1 times 10 to the minus 10, a little diversion here back to the temperature correlation function of the cosmic microwave background. Earlier I mentioned the first peak, the angular size associated with that, is sensitive to curvature. And so it is. Here you're going from that greenish color to that pinkish color. You're going towards smaller and smaller omega, more and more negative curvature. And indeed, negative curvature acts as a de-magnifying lens, moves the peak to smaller angles, and hence larger multiple moments L. However, the height of the first peak is quite sensitive to the baryon to photon ratio. The more baryons you put in, that's going from the pinkish color to the greenish color here, the more, well, the more squishy the photon baryon fluid is. And so with more baryons, the photon baryon fluid is compressed to higher densities and thus to higher temperature fluctuations. So the height of that first peak, it's really a quite sensitive test of what the baryon to photon ratio is. And so here's the Planck numbers, 6.10 plus or minus 0.06 times 10 to the minus 10. You can convert that to a number density of baryons, about one baryon for every four cubic meters in the universe at present. And you can convert it into an omega for baryons. And this is where the number 0.048 for omega baryon comes from. And here the 0.048 plus or minus 0.003. Most of that uncertainty comes from the fact that we don't know the critical density all that well. Cosmic microwave background, got to love it. It's full of information about the universe at the time of last scattering. And I haven't even mentioned the polarization of the CMB yet. You'll have to get that in more specialized CMB lectures. So backward in time, let's go back to the time of Big Bang nucleosynthesis. Remember the early universe was radiation dominated? This is useful, because as we've seen in radiation dominated universes, the scale factor goes as t to the 1 half power. And so the temperature drops as t to the minus 1 half power. And that's a useful normalization. In round numbers, the temperature is 10 to the minus 10 Kelvin at a cosmic time of 1 second. That corresponds to in MEV units, 1 MEV at t of 1 second. Well, when you look at the curve of binding energy for different elements, you find out that the binding energy per nucleon on the vertical axis here is measured in units of MEV.  And in that early epoch, photons were energetic enough to photodissociate atomic nuclei. So at times much before 1 second, you didn't expect atomic nuclei to exist. Just as at times earlier than a quarter of a million years, you didn't expect neutral hydrogen atoms to exist. So sometime between t of 1 second and t of now, the first atomic nuclei formed. Now, today, when you're studying nuclear fusion, you don't have to worry about fusion with free neutrons, because, well, there's not a lot of free neutrons around today. Free neutrons are unstable. They spontaneously decay by emission of an electron to preserve charge neutrality and the emission of an electron neutrino to preserve the electron quantum number and the lifetime of a free neutrino, 880 seconds, about a quarter of an hour. This decay is possible because the mass of the neutrino is greater than that of a proton by about 1.29 MEV, or one part in 730 of the neutron mass. At t less than a second, obviously, the neutrons hadn't had time to decay yet. And also, at t less than 1 second, remember, kt was greater than an MEV. You had photon energies high enough to create electrons and positrons by para-production. So you've got protons. You've got neutrons. You've got positrons. You've got electrons. You've got photons. You've got neutrinos and anti-neutrinos. This means you had some interesting reactions going on. Neutrinos and protons were being converted into each other, because at t less than a second, the cross-section of the neutrinos for interactions was sufficiently high, and the number density of particles was sufficiently high. That's the neutrinos were still coupled. So neutrino plus electron. Neutron plus electron neutrino goes to proton plus electron and the reverse. And neutron plus positron goes to proton and anti-neutrino and also the reverse. And well, guess what? These equations were in chemical equilibrium. So as you can probably guess, I'm trying to build up the parallels between the epoch of recombination and the epoch of nucleosynthesis. That's kT greater than 150 MEV or so, particles, neutrons and protons broke up into quark soup. So you wouldn't have had neutrons and protons very early in the 3D universe. Once you get to temperatures below about an MEV to about 0.8 MEV, that's the temperature at which the neutrinos decouple. But in this intervening temperature range, the neutrons and protons would have been in kinetic equilibrium. It's cool enough that they are non-relativistic, so they both have a Maxwell-Boltzmann distribution of particle momenta. And if they had exactly the same mass, they'd have exactly the same distribution. But since their mass is different, you take the ratio and you find out that, oh, OK, there's a factor out in front, the masses to the 3 halves power. And there's this exponential Boltzmann term that goes as the difference in their masses. So the difference in their mass is 1.29 MEV q sub n. And their masses are small enough. The difference in masses is small enough that the factor out in front you can take is being equal to 1. So basically, you expect an exponential fall off in the neutron to proton ratio. No surprise there. When neutrons are more massive than protons, once the temperature falls well below the mass difference, you're going to strongly favor the lower mass particle. As I mentioned, the equilibrium state only remains as long as neutrinos are coupled. They're the ones that convert neutrons and protons into each other. And with a detailed calculation of the neutrino cross section for interaction with baryons, at temperature about 0.8 MEV, the neutron to proton ratio freezes out at a constant value. So the neutron number doesn't keep falling to zero. Instead, it freezes out at a value. Well, the energy difference between neutron and proton is 1.29. The freeze out temperature 0.8 MEV. And so due that exponential, the neutron to proton ratio after freeze out is about 0.2. So between the time of neutron to proton freeze out, when the universe is about a second old, up until the time of neutron decay, when the universe is about a quarter of an hour old, during that interval, you can take the neutron to proton ratio as being roughly constant, one neutron for every five protons, which has interesting consequences for Big Bang nucleosynthesis. If you are going to build up atomic nuclei out of protons and neutrons, your essential first step is fusing together a neutron and a proton to form a deuteron, hydrogen 2 or the deuterium nucleus. So proton plus neutron goes to deuteron with excess energy being taken away by a gamma ray photon. And the reaction can also go in reverse. The binding energy of the deuteron, mass of proton plus mass of neutron, minus the mass of the resulting deuteron, is 2.22 MeV. A little footnote for those who love astronomy. The sun doesn't have any free neutrinos, obviously. So it has to make deuterium the hard way. Proton plus proton goes to a diproton, helium 2. This tends to fall apart again almost immediately. And that's a decay time of about 10 to the minus 23 seconds, the light crossing time for the diproton. And you can only get out to deuterium if the diproton decays to a deuteron with the emission of a positron taking away the electric charge and an electron neutrino. The decay time for this, well, it's a beta decay. So its decay time has to be quite long, more than 100th of a second. So diprotons are forming in the sun's core all the time. But mostly they just fall apart again into a pair of protons. The alternative decay to a deuteron is extremely rare. And thus the sun has been around for 4.57 giga-years and has only managed to fuse half of its hydrogen into heavier elements. It's a wickedly ineffective, inefficient fusion reactor. In the early universe, though, things went on like a house fire. Proton plus neutron goes to deuteron plus gamma. You'll notice no neutrinos are involved. This doesn't involve the weak nuclear force at all. So it goes like a house fire. Now, proton plus neutron forms composite object plus a photon recombination. Proton plus electron forms composite object with the emission of a photon. Obvious parallels there. So it can make a rough estimate. The main difference is the difference in the energy levels, 2.2 MeV versus 13.6 Ev. And if recombination takes place at a temperature that we know around 3760 Kelvin, then deuterium synthesis has to take place at a temperature that is higher by a factor of the ratio of 2,220,000 to 13.6, or about 6 times 10 to the 8 Kelvin. Now, this corresponds to a time of about 4 and 1 half minutes. As Steven Weinberg reminds us, it's actually the first three minutes. So you are slightly underestimating with this calculation the time of nucleosynthesis. There's actually Iprimitriminute, the first three minutes. So that is the only Italian that I'm going to attempt the entire time I'm in Italy. I should stick to my native tongue. So if you do the real calculation using the nucleosynthetic equivalent of the Sahá equation, you find something a little more accurate. Again, you can find deuterium to, whoops, neutron ratio. And you can define the time of deuterium synthesis as when this ratio is equal to 1. So more accurately, it does come out to a slightly higher temperature, 7.6 times 10 to the 8 Kelvin. If you want to convert that to a redshift, you're free to. It's about 300 million. And the recombination to me, that should be nucleosynthesis, not recombination. You can see where I did a cut and paste inattentively. Recombination, no nucleosynthesis yet, at about 200 seconds or about three minutes. And just as the time of recombination is weakly dependent upon the baryontofoton ratio eta, so too the time of deuterium synthesis is also weakly dependent on eta. But deuterium is not the end of the line for Big Bang nucleosynthesis. You do make a little bit of light helium, helium-3, and tritium, example reactions there. And once you have tritium and helium-3, now tritium is unstable, but its decay time is about 18 years. So during the time of nucleosynthesis, you can treat it as effectively stable. The next steps, once again, undergo more fusions. You go to helium-4. And helium-4 is almost, but not quite, the end of the line for Big Bang nucleosynthesis. There are no stable nuclei with mass number 5, so you can't simply fuse a proton and a neutron to helium-4 and expect it to stick. If you try to fuse two helium nuclei together to get beryllium-8, again, doesn't work. Beryllium-8 is extremely unstable. You do make very small amounts of lithium, lithium-6, by fusing helium-4 with deuteron. You make small amounts of lithium-7, by fusing helium-4 with tritium. And you make small amounts of beryllium-7, by fusing helium-4 with helium-3. So here's the network of reactions that can go on. The beryllium-7, again, like tritium, it is unstable. It later decays to lithium-7 by electron capture. But during the time of nucleosynthesis, you can treat it as effectively stable. So you can go on, get traces of boron and carbon, but these are extremely, extremely tiny traces. Big Bang Nucleosynthesis, it's a race against time. The temperature is steadily dropping. And by the time the universe is about 15 minutes old, that's about it. Big Bang Nucleosynthesis is very efficient until you get helium-4. And after that, you get basically little traces of lithium left out. Now, knowing the cross-sections for all the relevant interactions in Big Bang Nucleosynthesis, you can run your Big Bang Nucleosynthesis code. And you find out that the yield of different elements depends upon the Baryon to Photon ratio. So this gives you another way of estimating eta, the Baryon to Photon ratio, the top panel. As you get fewer and fewer photons per baryon, or more and more baryons per photon, Big Bang Nucleosynthesis starts earlier. It's more efficient. And it runs to helium more and more effectively. So higher Baryon to Photon ratio means more helium. And the fact that it's more effective means less deuterium and less helium-3. Lithium-7 has this interesting dip, because in this Baryon to Photon ratio range, direct production of lithium-7 decreases with eta. And the indirect creation through Baryon-7 that later decays is an increasing function of eta. So looking at the yield of Big Bang Nucleosynthesis as a probe of eta usually focuses on deuterium. As you see, it has a relatively strong dependence on eta compared to the values of helium. It doesn't have this local minimum, the way the yield of lithium-7 does. So you want to find the primordial deuterium to hydrogen ratio, because deuterium is very easily destroyed in stars. Once a star forms from the interstellar medium, its first fusion reaction is destroying that deuterium and building up heavier elements. So we want primordial gas, gas that hasn't been cycled through stars. And so astronomers look for deuterium abundances in, jargon alert, metal-poor damped limon alpha systems. These are metal-poor. Astronomers use metal for everything other than hydrogen and helium. The chemists laugh at us, but so we do. Metal-poor means, oh, metals, heavy elements are made in stars. This is gas that has not been cycled through stars. It's a damped limon alpha system, because it's a cloud of primordial stuff. Hydrogen and helium, with a little lithium, between us and a distant background source, a quasar. And so if you look at the Lyman series, you see absorption lines going from limon alpha, limon beta, gamma, delta, and so forth. Limon alpha is, in this range of frequency, completely black. That's the damped limon alpha part. But you go to the higher order of Lyman lines. You find out that there's this main line, but off on the left here. It's slightly blue-shifted. There's this little supplementary line. That's because there's a small isotopic shift in limon alpha for deuterium relative to that of ordinary hydrogen. Limon alpha is usually quoted as 122.567 nanometers. Limon alpha for deuterium is blue-shifted by one part in 3700. That's equivalent to the shift you would get from a Doppler shift of minus 80 kilometers per second. That's why this horizontal axis is labeled in units of kilometers per second. You can model these lines using different deuterium to hydrogen ratios. And so Petini and his collaborators found that the best fit is deuterium hydrogen about 2.5 times 10 to the minus 5. That's the number density of deuterium relative to that of hydrogen. And I have a question here. I've forgotten the redshift. Nope, it's not written here. It's a relatively high redshift limon alpha cloud with a still higher redshift quasar behind it. You can look up the paper Petini et al. But unfortunately, I don't remember the redshift. But they did get this quite accurate measure of the deuterium to hydrogen ratio for this system. There are other similar damp limon alpha systems as well on the lines of sites to other quasars. And again, going back to the previous slide, you can for a given deuterium to hydrogen ratio as long as you're certain it's primordial. Read off what the relevant value of eta is. And in this case, it's 6 times 10 to the minus 10 plus or minus 0.1 with, again, good agreement with the value of eta found from Big Bang nucleosynthesis. Now, I'm under the impression that I'm supposed to stop talking now. And since it is time for lunch and my voice is giving out, I did have a few comments on inflation. But I'll save them for the beginning of the next lecture tomorrow. So thank you. And please.


So I am David Cantrell and we have Tim Flink here who I'm going to ask to come up in in a minute if he wants to We actually have two talks that are related to each other One is today the other one's tomorrow in the morning. I think 10 30 maybe yeah so what we've been working on is The integration of a tool called RPM inspect into the for the Fedora CI pipeline so the talk tomorrow is gonna be a bit more about Fedora CI and What what we're trying to do there? I was gonna give a brief introduction here and then ask Tim to add anything to it if he wants to So yeah Fedora CI RPM inspect and then how you can help if you're interested in helping there's plenty of work to do Okay, so Fedora CI Please go to the talk tomorrow if you're interested in that a Framework for auto automating tests and providing results. That's really generic, but if you went to the Packager workflow talk or earlier by a meal we kind of talked about the experience that a package or goes through and Wanting to know if builds failed and stuff like that. So this is all tied into Getting to a point where builds that we do of packages can be trusted to put into an OS compose So we don't have raw hide and other releases in an unknown state So doing as many tests as we can Before and there's there's a lot of different aspects to what constitutes a test every package is a little different You know when you test a Python module that's going to be different than testing say the bootloader so How do we get there? Well, we have this Fedora CI infrastructure that's going to provide these runners and triggers and things like that our PM inspect is Just one example that I'm going to go into here. So knowing that builds are tested and working Gating a build before we get there now. I know we have this enabled for raw hide We have it. I think keyed through Bodie One thing that I would like to do is some of these tests move them to before that Which we kind of running right now for our PM inspect We have a few more things to tie together there Because I think a lot of these things need to happen before we even get to that point and not for a high Well, yeah True. Yeah What do you want to add anything else to Fedora CI stuff right now? No Okay. All right. So here's the docs link. Oh, and by the way, my slides are I write them also to be like notes and reference for people later So that's why there's like a URL here So that's that's Fedora CI broadly Now RPM inspect Build deviation analysis tool. Okay, that's kind of a mouthful Let's talk about a little bit of history first So red hat developed an internal QE tool called RPM diff Not the RPM diff that you've seen as part of RPM lent. We like to reuse names so RPM diff began life in The internal one let's say 2003 2004 doesn't really matter a long time ago before avert before containers before the cloud We had no clouds at all And it runs as a service and performs tests on builds that we do internally The service aspect here is key there's no way for package maintainer to really interact with RPM diff at all like it's it's automatically scheduled the results are then post-itting you have to Fix it sometimes often not really knowing what it's testing So what kind of test does it do package policy compliance? So making sure you've named something correctly Making sure you have a an approved license on the package all of that metadata in there We have yeah, the legal checks currently be I verification is a big one for the internal tool Enforcing security policies. Do you have any set you ID route executables in the package payload? And are they allowed? So there's these data files at RPM diff will query to see if You know you you haven't allowed one. So there's about 40 tests that RPM diff does and they're all of Varying complexity This last thing here as well comparisons from one build to the next that's where the diff aspect comes in so For rel packages, this is really important for for compatibility reasons we want to avoid any surprise regressions so Files is appearing new files appearing in a subsequent build a new package appearing or going away sometimes executables would grow Substantially like you know you go from one megabyte to 300 megabytes something went wrong with linking So it'll catch things like that and report it Okay, so how does it do this? internally We use the errata tool developer, okay, so at this point the package maintainer has Completed all of their work. They checked in all of their fixes to disk it they've built it in Koji and now they log into this web-based tool to do the paperwork So this is basically like the internal version of Koji if you haven't used it. All right, so the errata tool schedules an RPM diff job Oh, yeah, I meant Bodhi. Sorry. Yeah Thank you Too many names to keep track of okay, so the errata tool schedules an RPM diff job That talks to the RPM diff hub, which is a Python Server running actually on a physical server It verifies that the build exists. It does this by talking to Koji If the build exists it hands it off to a worker a worker is another physical server There's a pool of them and this is running a Python service as well the worker Performs the RPM diff operation. It copies the builds from it and it an NFS share There's no way around that It then runs a pearl script to unpack the RPMs Then it runs a Python script to actually run the checker program Checker program is written in C and it iterates over all of those 40 tests I was talking about and each one of those spawns multiple pearl and Python scripts and it collects the results and Puts it together in XML format and hands it back up to the hub It's not a great architecture, but again given the age and given the time period it was written. It's kind of understandable It's the service site of it the hub and the worker are too closely tied to the part that performs a test so Runs on physical hardware that would be a little easy to create the biggest problem is the requirement of the NFS share Which is kind of hard to unravel right now It's a legacy code base a lot of technical debt. I am not a pearl expert At all so it's really hard to kind of go through that There's too many moving parts you can't run it from the command line So a developer can't use this tool at all and it only gives you XML because at the time it was written that was you know What everyone wanted yeah Okay, so call graph I love this so the When I was talking about how how this runs the C program the RPM diff checker program This is what the call graph looks like if you're familiar at all with this So this is the entry points all the function entry points in the C program The second row there is basically corresponds with One or more of those tests I was talking about and each one of those boxes is going to be spawning multiple External scripts pearl Python sometimes shell So this is just running it on to Z shell builds and I captured this I thought it was interesting Okay, so back to RPM inspect so Going through all of that code. I thought this is going to be very difficult to make it usable in the fedora space so What are the actual goals here so ensure package reliability Developers aware of the build changes we want developers to be able to modify packages and stay in packaging compliance But we also want to provide data in a CI environment to make decisions. So I Wrote this down got to be able to run it locally got to be able to run it in container Need basically do all the things that that RPM diff can't do directly right now that it relies on external tools to do So the service side that hub and worker aspect that's the part that's being broken out and becoming part of fedora CI and Tim has written that now it runs in fedora And we just need to connect where the results go, which is kind of cool Service listens for new builds and launches RPM inspect runs RPM inspect does those tests and then this hasn't been done yet, but It'll be the next step we get those end of results DB and then we can tie it into the rest of the chain there So Policy checks and then differences. So one thing that RPM diff could not do is just run against one build Which I thought would be really useful for developers. Let's say we have a packaging policy change in fedora and You can just you can check your latest build and say well Does it comply against what the policy is? It does require RPM inspect knowing what that is But we have one point to change and then a tool that can run for developers and then we also have the ability to do The differences between two builds So it does look at sub packages right now. I haven't actually added that that's a good idea actually if if a file by the same name Same location some some kind of a qualifier and it could be a little more intelligent right now All it does at the moment is what RPM diff did which was just identify So when it's when it's doing two builds It'll have the before build and the after build and then it goes through all the sub packages And that figures out the peers and then all of the files and those so anything that's In a before build and has no peer in the after build And likewise so that that's that's where that starts, but yeah, it would make sense to Catch it if it moves because that is quite common Yeah, certainly in fedora, so yeah Yeah, that seems totally doable at least in some capacity Policy check so these are things that I do right now again these ideas came from RPM diff So I check the license tag validate that against the approved license database check header metadata Political concerns is is an interesting one. So this is This is from rel red hat being a US company has to be conscious of certain I Don't know embargoes or or political concerns or market concerns. So there is a the political check in there is a white list for a loud files that would otherwise trigger a warning So an example might be The flag of Taiwan being included in the release and we ship it to China China doesn't like the flag of Taiwan for various reasons so This is a thing that red hat has to be concerned about so it you know Might not necessarily apply verbatim to fedora But there could be other things One of the things I have done an RPM inspect is set it up so that we can easily enable and disable tests per packages So with RPM diff I didn't mention but all those 40 tests everything runs for every build You don't get a choice it's all or nothing so this you can just run one test or you can disable one or two So we have that forbidden language again. This one is maybe not a strict in fedora Red Hat has you know, it looks for English Swear words and some other things Use of macro so we can enforce that and it's pretty easy Java bytecode major version check I was just working with the Java Developers, I forget what things they well I know they do the open JDK packaging but some other stuff too and I was asking them the the origin of this test and For rel we ensure that we don't ship any class files that have a Java major version In the class file header that does not match the version of open JDK that we ship I'm thinking okay. Well that that's pretty good. So I added that in there Also, I also found out that Java class files have the same first Same first bytes as a Mac OS 10 executable Which yeah, it's in hex it spells cafe babe So that was kind of interesting Elf object checks, so there's a lot here especially tied to security enforcement Policies around that and then the set you ID set GID things like that So this is this is what's present now I have a long list of things that I want to add most of them ideas from RPM diff yes So I've built this as a program in a library and the intent is we can just add we can keep adding tests in it Some people cringe when I mention this, but it is written in C The primary reason for that is most libraries that I need to talk with to do the work are written in C I am happy to help you. There is a template inspect function and It's it's pretty straightforward So yeah, I mean you basically if you want to implement If you want to implement a test it's really just One new source file and modifying a header and then it's there so yeah All right, so build types regular Koji RPM builds and also RPM inspect supports module builds RPM diff does not and This is built to extend to other things that Koji might output in the future I know it does more than RPM builds, but let's say at next year's flock We all decide to start using Debian packages, you know, we could modify it for that Very easily it'd be trivial Okay support files so one problem that RPM diff had was it hard-coded basically all of the knowledge it was checking for It was in a mix of header files String constants just it was all over the place. So I started splitting that out into a separate data package And a configuration file to provide even more fine-grained control so RPM inspect comf and then the Product specific data is in the RPM inspect data package now the program itself provides a package called RPM inspect data Generic that is a template package and then I have a separate one called RPM inspect data fedora That's the one that you want to install if you run this on your system It's kind of the same idea as Fedora release red hat release generic release that kind of thing Because there is a separate internal package called RPM inspect data red hat and it is Completely and totally different. Let me assure you Okay, so how do we run this first you want to install it I Actually forgot on that first line to install RPM inspect so you get that in the data package. There is a man page. There's help options But the basic syntax is here Previous build new build is optional. You could just specify one build and then here's an example. So verbose output minus K is I keep all of the temporary files Minus capital T license I'm only going to perform the license test on this RPM inspect run If you run RPM inspect minus L, it'll list all the available tests So you could do multiple ones. They are separated by a comma All right, what can I do today? I've kind of been through some of this here By default it'll output to the console just in like Mostly human readable format. It can also do a JSON output Which we're using to integrate into the CI pipelines And I also added a fetch only mode if you want to use it to just download a build by nvr Discuss I was doing that and it was handy So, yeah, it it does a lot right now Just kind of kind of ran through Mostly all of these. Oh, yeah validate desktop files Make sure you have the disk tag now the way this is set up is as packaging policy evolves as our Build process evolves the knowledge that RPM inspect has will have to change So that's why a lot of this lives in that data package So we don't have to go and change code for these things But there may be new tests that we need to add as time goes on And there may be tests that we can remove that you know are no longer valid So one thing I wanted to do was show So Right now this is So I've got a copper repository in Fedora So every time code is checked into the upstream repo, it'll build automatically. We're doing sent to us Seven and then most recent Fedora's the package does exist in rawhide. I have not Done it officially for previous releases if someone wants to volunteer for that I'm happy to accept that offer The best way right now to use it and the command line is just to enable the copper repo And then if you're familiar with Our build system. So this would be this is Z shell Just listing all the recent builds So I go here when I'm testing it and I I find two previous builds so let's Yeah All right, so I've already installed it because I did not want network connectivity to fail and all that kind of stuff But we'll just run through a couple of things here. So your normal output screen here It'll it'll show you I've well I try to show defaults here And then how to override it Let's see Yeah, the dash Dash-list to list tests format type The test list those are probably the most important. Oh, yeah, and for the let's see product release string So product release string is something unique to RPM diff and I didn't want to carry it over To RPM and spec so internally when an RPM diff job is run It has to be told what release of rel to run for so is it going to run for rel 8 or rel 7 and I thought to myself You know, I can probably infer that from the disc tag So what it does right now is it it grabs the disc tag from your before and after build and if they match That's the release. It's going to use if they don't match It'll stop and tell you to tell me what release to use And we're just going to go with disc tag because that's established And is is well documented So you could take like Fedora 31 builds and say do it against Fedora 30 question But it's it stops and tells you why So, yeah, I know it's not yeah, it's not interactive. It's going to exit yeah, so if I Run it at the command line here, let's see if I want to do the license and metadata tests for Whoops, I've already cashed some builds because again, I don't trust the network I'll do license metadata and then Yeah Koji builds my before and My after build now if you I'm doing I'm telling it the exact path to the build Only cuz I want to save time here But you can just specify the nvr. So, you know your your specification can look like this and What rpm inspect will do is ask Koji it'll say is that a build and if it is tell me where to get it And it'll download it So you don't you don't have to do that separately. It just knows and the rpm inspect Confile tells it where to get that info Okay, so this will perform two tests. This is gonna output in what I call the human readable format, but So that was where it stopped I don't know if everyone saw but I Have the trailing slash there. I Could probably fix that I've actually got a long list of things. I'm gonna fix tonight. Yeah But yeah, so you see it grabbed FC 31. They don't match and it doesn't know what to do It does have a verbose mode. All right, so here we go. Oh, by the way, rpm diff for those Unfamiliar with it We would start this job and then walk out there and get a cup of coffee And talk about how the conference is going and then we come back and it might be done That's that's right. Yeah, so so we've got a big speed improvement here so This is the test name So license test and then it's just reporting text here a lot of this I have lifted from the output from rpm diff Please don't take this as being authoritative. I am Welcome to input of what this this stuff should say question Correct Correct So These are all really good points and I think fedora needs a little more flexibility than what rel does so I would say And this is a conversation that Tim and I need to have to make some decisions on Fedora QA needs to have some input as well My personal thought is that for rawhide the comparison is most useful to the last build that passed And For a released version it would be the last one that we shipped as an update just as a starting point I'm in the fedora CI SIG and Want to have the conversation in that sort of context. So if you want to be part of it I encourage you to watch there and yeah Because again, this is I want input from package maintainers.  I don't want this to be something that you don't get a say in because no one no one likes tools like that No Yeah Yeah, no, I thought about that the main reason is I've got the the peer detection And I have to iterate in a certain way, but I have written it such that this can run Like as many times as you want on the same system rpm diff was very strict in that Exactly one can run at a time It just yeah So I've made sure that that can happen it can run in a container can run inside mock if you don't want to do a container I mean I've tried to make it pretty flexible in that respect So this is the output of the license stuff and then down here had her metadata. So there are some errors here now this Package build host. This is again a test I lifted from rpm diff It checks to make sure that The package build host that gets recorded in the rpm header Internally ends with dot red hat comm so when I created the rpm inspect data package I said, oh, well it makes sense that we probably want to build things on dot fedora project org host Well, you can see this package was built somewhere else. I don't know Hogwarts dot boss red hat calm, but This was not built in Koji Or maybe it was I don't know So but this also I have here suggested remedy and this is another area that I would love comments on from from people for specific tests It tells you, you know What can you do to fix this rpm diff doesn't do that? Let's see All right The entirely possible. Yeah So that's that's one that that may need a little investigation So I wanted to do a Different format here real quick So that's the default output if you're using it as a developer And this is what it would be for that we're using to integrate into the CI pipeline I'm not doing XML because I don't see a need But it's set up such that we can drop in additional output formats if we need them So there we go Alright Okay. Oh, yeah, so the call graph so I explained rpm inspect You saw the one for rpm diff This is what I have right now for rpm inspect And keep in mind that the rpm diff one doesn't download Packages does not directly talk to koji and can't even unpack an rpm Rpm inspect does all that and I try to make code reuse a thing if you can't remember what the The other one looks like there it is and then here's the new one So again trying to make it a little more Maintainable and usable and not have a single choke point for the whole program So Contribution so I have a unit test suite has been started That's kind of the first step an integration test suite is in the work So this is this is one of the huge advantages of rpm diff right now is Despite being a legacy code base largely in maintenance mode for over a decade It has a very extensive integration test suite all of the people that worked on maintenance for it We're very very disciplined in adding in Integration testing time there was a bug so I want to make sure that we have that for rpm inspect just to make sure it's reliable Documenting the details of individual inspection. So what does the license license test do where does it get that data? How can I change that data who owns the data? It's testing against that kind of stuff designing new tests based on the fedora packaging policy or other things So for instance the python test, you know that you were mentioning would be good And then help maintaining the the data package. So this is going to be a moving target just because fedora evolves And I'm not going to know everything about the distribution. So Send pull requests and also Bug fixes and code and just general feedback This is ready to use right now if you maintain packages. I'm interested in hearing feedback like You know, what does it look like? what fails You know, it's not perfect right now But the more people using it using it will help get it to a really good point. I Have a goal of implementing the rpm diff test that makes sense as well as coming up with a process to Take the fedora packaging policy and get that into a test that is easy to maintain through the data package So as the packaging policy evolves, we can update the data package and rpm inspect will just pick up all those changes So we can keep things in compliance Code lives here right now. I do have there's an organization project called rpm inspect that I'm going to move everything to I haven't done that yet I mentioned packages are in rawhide automated builds are in copper and then We've got fedora si on frino poundo sci internally. I'm decantrell Get hub issues is where to tell me about problems and question so actually I made a bunch of notes about that earlier and I do think it would be really useful like it can already do a lot of what fedora review is doing now and the idea of encapsulating the packaging policy in a data file format I Like I think it would be pretty easy and it could potentially replace that fedora review functionality Yeah question So that was a that's a thing from rpm diff. I didn't really explain that so When rpm diff runs internally for a rel build You're going to get a pass or a fail or something called needs inspection If it's on needs inspection or fail you go in and you look at the output of the individual tests and as the package maintainer You can wave that result and explain why the test failed Or if it's legitimate you have to go and fix it and submit it again I don't know if that's really going to be useful data for For fedora, but I it's in there right now and it's in the results It would be very easy to remove or just alter as we see fit for gaining, but that's what that's from the waiver authorization Internally rpm diff has subcategories of tests that the results can only be waived by say the security team or Another team the default is anyone can waive the result. So that's why that was in there So That's a good question. So what we were going to talk about At the end of the week and the packet workshop is incorporating this into the packet workflow So say you have the spec file upstream you get a pull request for a project a build is kicked off We run RPM inspect the results go to results DB and then that's visible through the packet output in some capacity And No, not yet, and I wanted it it is running right now the results just aren't consumable yet But the idea is to have that Completely tied in so the results are visible somewhere so yeah, and and With that we'll be able to make gating decisions. So if there's a lot of failures or Any failures we can say this this package Cannot proceed to the next step This build can't proceed and here's why So that yeah, that's the idea Sorry Right it does not rpm inspect does not perform any build So it just looks at the output of what an rpm build minus BA would be So when I was writing down I was thinking yeah I could do the review and I went and looked at the code briefly and saw that it does that and I thought Well, maybe that's out of scope for rpm inspect, but Yeah, there's some overlap there with policy compliance question Yeah, I think Yeah, I really don't want our payment spec to start doing builds and stuff like that but if yeah, Fedora review can make use of the Policy checks, you know that I think that would be a win so yeah maybe only and and I you had another question and I went around so You can certainly do that. Yes, eventually it'll be automated when you submit a Koji build But yeah, you can do that now if you want to yeah, and actually I'm interested if you do that so like Message me or email me like how that goes because I'm curious What'll happen there? Okay. All right. Yes Define average size Okay, so I would yeah, so the kernel takes a while right now and I'm I'm considering like Maybe I can restrict it like with With mock or with Koji you can Restrict it to a certain architecture and that may be able to allow us to farm things out a little more in parallel Right now there's there's a bug in one test where when you run it against the kernel package It's actually gonna seg fault. I was yeah. Yeah, right. It's really fast. Yeah So the the full suite of like all the tests that are implemented in there right now Which are like 10 or 11? It's less than a minute. So I was talking about that integration test suite and kernel glib-c gcc are once and I'm Running on the side constantly to see what the output is and and where we can do some performance improvements Oh, oh, I could do tech live That's a lot of sub packages Yes What what fedora leases this available for no. Oh, yeah, okay, so So if we're let's say we're doing rawhide's doing builds for fedora 31 right now. That's what the disc tag is so That check is in place to ensure that by default you are comparing builds from the same like release development chain or Collection That again comes from rpm diff design, but there's no restriction to do that you like if you specify a Fedora 30 build as the before one and a fedora 31 as the after build all you have to do is tell rpm inspect Use this as the release string Assume it's fedora 31 or assume it's fedora 30 and that's Literally the only reason it exists in rpm diff, but I think it's a nice default catch to Prevent any accidental comparisons From builds that were done for a previous release. So yeah Any more questions? Yes Yes Yes Yeah, we don't we don't have any of that set up yet. That is Pending yeah, yeah Thank you. I'm sorry. I wasn't too clear on that, but yes implementing it as as a gating test Definitely the main objective here So yeah, I'll be at the packet workshop So if you want more information About rpm inspect or any details. I'm happy to answer those I will be Working to you know help Figure out how we're gonna connect packet with rpm inspect. So that'll be part of the discussion as well Yes, thank you, and I think I finished early All right, well everyone gets an extra five minutes or they can keep asking questions Okay, thank you