morning everybody. So Paolo will be here later today so I just thought I'd put up the list of posters that are at 10.30 this morning during the coffee break at the usual place and so hopefully you've had a chance to have a quick look at the titles and now we'll get going with Raphael's next talk. Okay so yeah we'll continue. So yesterday I spent some time talking about the history of the CMB, why there was an expectation that one should see a cosmic microwave background and I talked in some amount of detail about the the spectrum, why you expect it to be a black body spectrum, to what extent there might be departures from it. Today I'll finally talk about the small fluctuations in the cosmic microwave background so the departures from the 2.7255 Kelvin black body radiation and I'll start by talking about the the kinematic dipole. So the first if you say that there is a cosmic microwave background or a black body radiation around then the first thing you have to check and that's what we discussed yesterday is that it's actually a black body. The second one is it's unlikely that we're addressed with respect to that black body and so you expect to see some some dipole from our motion with respect to the cosmic microwave background and that's what I'll discuss and then I'll say a little bit more about things that you might see because of our motion so there's also a kinematic quadrupole and in principle octopole, hexadecopol and so on and there's also other effects that you can look for and they have even been measured so you can look for aberration from our motion and modulation in the in the microwave background and so after the discussion of what you see the kind of effects you see in the microwave background from our motion I'll finally discuss the intrinsic anisotropies and then I'll briefly give I mean this you already saw in some of the other talks in Rico yesterday also introduced it so I'll briefly introduce the notation I'm using which is fortunately roughly the same as Enrico was using for the for the perturbations and then I'll tell you a little bit about how to compute the angular power spectra today I'll just show you the relevant equations and I won't really have time to show you the analytic solutions to them we'll do that at some later point so this is what I'll try to cover today and yesterday today so always feel free to interrupt me and ask me questions if there's something that isn't clear just feel free to ask me at any time so as I already said so if you have a black body radiation around this provides a reference frame there's a clear notion of what it means to be at rest with that with respect to the black body this is when it has the characteristic black body spectrum and so you can ask what happens if you're moving with respect to that frame and if you have the we know the spectrum we wrote it down yesterday so for a black body a temperature T you can work out yesterday we wrote down the spec the spectrum in the sense that I wrote down the number density of photons with frequencies between new and new plus D new if you think about it in terms of a quantum field then you might let's say you put it at some some finite volume so in in case base you know there's a number of harmonic oscillators so for different values of K so this is say KX, KY, KZ I'm calling it P up there so let me call it P and P and you have harmonic oscillators that you can excite everywhere so this I'm sure everyone is familiar with and what I'm writing here is just the occupation number in in this state for a given one of these oscillators in particular the oscillator that's labeled by the by momentum P so this is the occupation number and if you want to understand what happens when you boost the occupation number is a useful thing because it doesn't really change it doesn't matter which way you look at the state if you had five photons in in this state so and at this this oscillator if it had five photons before you boost then it still has five photons after you boost and so the occupation number is invariant in the sense that the occupation number for the observer that's addressed with respect to the frame and the momentum that he labels by by the label P this has to be the same as the occupation number that the boosted observer sees for the momentum for the same site here but he will label it by in by his own label which is related to this label by a Lorentz transformation so this is the transformation property for the occupation number does that make sense okay so then what you can do is just relabel them so you can this is lambda P so in the same way you can just relabel them so you learn that the occupation number that the boosted observer sees for the mode that he labels with P is the same as the occupation number the observer address sees for the mode that he labels as lambda inverse P okay and so this occupation number we know we know what it's what the function is so let's write it like this as four vectors we know what this function is and strictly speaking it's only a function of the zero component of this so this was only a function of the energies and now what is this so let's consider a boost tool to a frame relative to the CMB with velocity data so then you know that the boost that does it I mean let's say for simplicity we'll assume that it's moving in the three direction and we'll fix that in a second so let's say we're moving in the three direction then you all know the boost by heart I'm sure so the boost in this case is gamma 0 0 minus beta gamma 0 1 1 0 0 0 0 and then minus beta gamma 0 0 gamma and what we're interested in really is the lambda inverse so you just take the inverse by flipping the signs and so now the thing we're interested in is this quantity is the zero component and you see if you act with this on the on the vector e p1 p2 p3 the zero component is gamma times e and then plus beta gamma let me write it the other way gamma beta p3 but this is just beta dotted into p so this is really this quantity and so you can simplify this slightly more you can simplify this to write it as gamma times e times 1 plus beta dot p so this is the sorry this is now the unit vector in that direction so this is the lambda minus 1 p0 component that we need and so now we can plug this into into this formula so you just get that n of lambda minus 1 p is 1 over and then e to the we just plug this zero component in here so it's gamma times e times 1 plus beta p hat over kt this was the t0 minus 1 and you see that this is the same as a distribution with an effective temperature t as a function of p hat this is again the momentum of the photon you're looking at is equal to t0 divided by gamma 1 plus beta p hat and then typically you expand this this out so this is t0 times 1 minus beta p hat and then the this is the momentum of the photon if you see a momentum with photon in the p hat direction you're looking in the minus n hat direction so you get some plus n hat so this is the temperature as a function of n hat and you see that this gives you some dipole but there are in principle higher order higher order terms if you just keep going in the expansion so this is what we have so this I think hopefully makes sense to everyone yeah oh I'm just expanding in small beta so gamma is 1 over 1 minus I mean 1 over the square of 1 minus beta squared so then principle there but they appear at order beta squared so these are the terms I'm dropping okay so this is the basic idea behind the behind the kinematic dipole and this was pointed out a long time ago so this is from 1968 it's a very clear prediction and one more point so the detectors that we have they typically don't measure the the temperature itself instead what they're measuring is really the intensity of radiation that's hitting them and so instead of the temperature field you really want to do this to do an expansion for the for the intensity but the intensity we had yesterday was just 8 pi nu squared well I d nu d nu over the black body spectrum which we just computed so you just expand it with respect to temperature and you get a dipole modulation of the of the intensity in this form the only reason I'm pointing it out is because these fluctuations don't have a black body spectrum so this is the black body spectrum the perturbations always come with a derivative of the black body spectrum with respect to the temperature so this is the only reason I'm pointing out this difference because we'll see that if you were to go to higher orders you get second order derivatives and so on so there's in principle a difference in the frequency different frequency dependence for the kinematic dipole quadrupole and and so on okay so this is something obviously once you understand it's there is something you want to look for and there were a number of early measurements the first measurement of the right ascension was by Ned Conklin which I'm showing you here this was already in 1968 and there were earlier ones that didn't really detect anything so there were upper limits this was the first detection of the right ascension but not yet the declination the declination so you measure it fixed declination and you trying to measure the right ascension you try to see where you have a peak and this was done in in 69 then the first measurement of the right ascension declination and the amplitude was done by Paul Henry so here you see the data is still obviously early days now we've measured with much higher significance obviously but this was the the first measurement of it and all the numbers he gave for the right ascension declination and amplitude are still consistent with what we know about it today you see that's a single author paper so he did everything himself he built the radiometer put it on a balloon launched the balloon collected the data did the data analysis there's some simple foreground cleanings in a way that's very similar to what we do in the analysis today and then he was the the one who wrote the paper and the interesting thing about the paper is that this was what he did for his PhD thesis of this was his PhD thesis the measurement of the dipole of the cosmic microwave background and then obviously as you go on the measurements become more and more precise so there are measurements at higher significance again some of them by the the group at Princeton and then you probably have heard about the measurement of the dipole by smooth and collaborators and here I mean this was measured from a U2 spy plane and these were actually the the radiometers that were used on the on the Kobe DMR so this was testing the DMR radiometers and so here you see a picture of the of the group and here you see the measurement so obviously much higher significance now than the previous measurements so this is the the history of the dipole and now you might ask well how do you know that the dipole you're seeing is actually from our motion with respect to the cosmic microwave background and it's not some intrinsic fluctuation certainly the amplitude is quite large so you don't expect this to be some primordial fluctuation but that's some some theoretical bias maybe but there really are experimental ways of telling the difference between the intrinsic dipole of the CMB and the and the contribution from the motion with respect to the CMB because of the higher order terms that I didn't write earlier but if you write it out and just go to next order in the expansion then you find here the the dipole piece so these are the Legendre polynomials so this is just the cosine that we had before I was just the and then you have here the higher order pieces which as I said involve second order second derivatives of the black body spectrum and in principle this is something that you can look for this is not something that has been measured but you see that in principle here the different frequency dependence allows you to disentangle the the very different pieces it's been difficult to measure it because of foregrounds it also introduced frequency dependence that deviate from from what you naively expect so it's it's not something that has been detected but it is actually corrected for in all the CMB maps that people are using there's a correction applied assuming that the measured dipole is a true dipole and then incorrect for a small kinematic quadrupole and so there's additional effect so this was the effect of our motion on the monopole now I'll say a few words about the effects on the perturbation so if you have a homogeneous bath of photons you're moving with respect to it the only thing you're seeing are these dipole and quadrupole effects if you have fluctuations in the microwave background then there's additional effects and you can again see them by doing these simple exercises so the occupation number now is the depends also on the direction you're looking so you have photons with different temperatures in different parts of the sky this is what I'm indicating here by the T as a function of n hat and then as I said before the n hat so the direction in which you're looking is my is the negative of the direction of the momentum of the photon and what you can do is just so this is the fluctuations at or the occupation number in the address with respect to the CMB but you can again boost so you do the the same exercise you apply a boost and you find that the new temperature that you read off so there's different pieces so there's the piece from the zero of order parts of the zero of order in the temperature and then there's two contrary two new pieces so there's a new piece here so you shift if you're looking in the in the direction n hat and you're flying in this direction you're not actually seeing the radiation from that point but you're seeing it from n hat minus the beta perpendicular perpendicular to this direction and so in other words there's some some aberration so you see the fluctuations somewhat boosted this is the first effect and then there's another effect which looks like something we had before looks like some some form of a dipole but it's different because you see it is a modulation in the amplitude of the temperature fluctuations so you see that if you boosted and you're looking in the in the direction in which you're flying the fluctuations are slightly larger than in the direction you're flying away from so these are the two effects so there's the aberration and then there's a modulation effect and again because this is a second order piece so you already had a derivative on this because it was a fluctuation and then you have another derivative because it's you have this beta it has a characteristic frequency dependence and you can look for these effects and these are actually effects that Planck has detected so these are there's paper and there's a nice paper that you can read about this from the Planck collaboration so this is the reference in case anyone is interested but so there's a number of ways you can see that we're moving with respect to the cosmic microwave background and they're all consistent so we have a consistent picture this these kind of dipole effects are usually referred to as the solar dipole there's another effect from the earth moving around the Sun so there's an orbital dipole and that's actually the one that's usually used to calibrate the maps but so so there's a number of these effects due to our motion from the cosmic microwave background that it can work out in in very simple way so you can go through the the same exercise and just boost this distribution and you'll see these these effects and so in principle we can measure the what are intrinsic fluctuations in the cosmic microwave background and what is are the fluctuations from our motion with respect to the cosmic microwave background are there any any questions about the dipole yeah well yesterday I was trying to argue that there are a number of reactions going on in the early universe that I mean you also know from nuclear synthesis as I was arguing that the universe was very hot and dense early on and in in the very early universe there are processes that both allow you to change the the number of photons we have double content scattering you have brainstorming and you also have processes that allow to to change the energy so you really have all these processes happening much more rapidly than the expansion rate of the universe so you do expect to get into thermo equilibrium if you want you can solve the the kinetic equations and convince yourself but you do end up in in some situation where you have a black body up to these small departures so is that what you're asking or you're asking just let's forget about what happened very early on how would we tell I mean one of the reasons I talked about these other effects is that you really can disentangle what's from the motion and what's from from other effects that that's why I was trying to show you a number of ways that you can see at what velocity we're moving moving with respect to the CMB and that consistent so you can on the one hand you can measure this this dipole piece this was the the first one that was measured this is the around 3 milli Kelvin so it's quite large compared to the fluctuations in the cosmic microwave background which are at the 10 to the minus 4 level so you can you can look for this this is something that's seen and then in principle there's no reason if somehow this isn't due to the motion of us with respect to the cosmic microwave background that you should see relativistic aberration and modulation in I mean with a velocity that's consistent it's it's both the direction that you can measure and the and the magnitude so there's a number of independent ways you can measure our velocity with respect to the CMB so some of it I would say is some theoretical input that you expect the in the early universe to have a black body spectrum and if you have a black body spectrum then you have some rest frame I don't know if I'm answering your question but bird in what sense so I don't know who gets to decide what's preferred but there is some frame in which the spectrum or the occupation number of the cosmic microwave background will look like e to the e over kt minus 1 to good approximation if we're forgetting the small 10 to the minus 4 10 to the minus 5 fluctuations it looks like this in that frame and there's no no dipole so in that sense it's preferred but it's I think the only sense in which it's preferred right yeah I mean in addition I mean you could have argued that maybe the yeah so the the one additional thing that was trying to get across is that you could have argued that this dipole that you're seeing doesn't have to be due to the motion maybe there's just some intrinsic some large intrinsic fluctuation in the temperature maybe the universe just intrinsically is a little harder on that side and a little bit colder on that side it's not something I can rule out just by making one measurement the point is that from that measurement you can extract our velocity I mean you can assume that there is this rest frame you extract our velocity with respect to the assumed rest frame from it and then you check in other places that also have effect I mean show effects due to the motion which are in for example the relativistic aberration so some beaming of the the CMB here you're making additional assumptions you're assuming it's somehow Gaussian and you know what the pattern is but we'll we'll talk more about that but there's the the beaming effect and there's the modulation and the the combination of these two gives you a dipole that's actually consistent with the dipole you get from just the kinematic dipole so it's at least self-consistent I mean obviously there could be in principle some the the additional point I mean you could also say well maybe someone just set up the fluctuations that way maybe there's some some dipoles a little bit hotter here a little bit colder here and then the hot spots in the direction are moving are a little bit smaller than they are in the other direction and in addition they're a little bit hotter here than in the other direction I mean there could be such an initial condition but it's a little bit awkward and in addition you wouldn't be able to set it up in a thermal process because the frequency dependence is actually wrong for that so it's not just thermal fluctuations in that direction but there's actually additional frequency dependence is this function so it's not the first derivative of the black body function it's higher order derivatives so in principle at least there's some consistent picture I don't know if some contrived way you can set it up maybe but it's at least a consistent picture yeah it does yeah it means that I mean if you as you say whenever there's a plasma or some some thermal state there is a preferred frame which is the frame with respect to which or in which this this fluid is addressed so there is that frame so it's broken not in a not in a terrible way so I think when you talk about this goes a little bit away from what I wanted to say but if you talk about breaking of Lorentz invariance there's different ways you could break Lorentz invariance so you could break it at very short distances and this is somewhat this is the the kind of breaking that's problematic and very strongly constrained or you could break it at long distances let's say by by the states which you do here by by putting a chair in the room or by having a plasma in the room so as soon as you have a state that's not the vacuum you typically break Lorentz invariance but that's not a not a problem if you go to shorter and shorter distances you see that the underlying theory is still Lorentz invariance so that's the the sense in which here Lorentz invariance the theory does not violate Lorentz invariance but the the state you're considering certainly does I mean we do when we write down the the metric I mean there certainly is on on long distances when you write the s squared is minus d t squared plus a of t squared dx squared then certainly Lorentz invariance is broken here if you interested in in late I mean if you take into account the time evolution of this but if you probe it on shorter and shorter distances eventually you're not really seeing that this is time dependent anymore I mean in our universe for example we're doing this all the time I mean we're usually when we do field theory calculations we use a Minkowski metric I mean for the for the LHC but you know that really there is a metric like this and there's some let's say the sitter expansion from dark energy there's some some expansion due to the Hubble expansion and it's only if you probe physics on time scales much shorter than that that you don't care about it so on on short distances your restore Lorentz invariance on long distances it's broken the same way in in the early universe in the plasma also also in the plasma really in the same way so if you look at processes that are much faster than the Hubble expansion then you can use the usual Minkowski calculations and in fact that's typically what's done so all the scattering rates that I was quoting yesterday did not take into account the expansion rates of the universe you're always expanding assuming that the individual scattering processes occur on time scales that are much shorter than the expansion rate of the universe okay more questions not then let's move on so let's now look at perturbations beyond the dipole so you take the maps that you get from from the satellites that we'll discuss later and you subtract off the effects that we were talking about I mean usually the aberration and modulation is not really something that's corrected for but the dipole and kinematic quadrupole are actually corrected for and then you get some intensity which depends now on the the direction you're looking and in principle you can also have a polarization filter and I'm indicating the orientation of the polarization filter by some function by some argument psi here and so the way you're defining the orientation can in principle depend on where you are in in the sky or you can yeah anyway so there's some ambiguity in how you exactly define the coordinate system here but let's in this direction we can define an X and a Z and we can define an angle cosine theta that describes the the angle in in this plane in a particular direction and then the intensity will or the fluctuation in the intensity from the intrinsic fluctuations it will have this first derivative from the black body spectrum which sets the the frequency dependence and then you will have what we call the temperature fluctuations so this is the piece that's independent of the modulation doesn't matter which way you orient your polarization filter there's some component that you always see and then there's something that's modulated so there's some polarization in principle and it's of this form so there's a parameter that's called the Q Stokes Q parameter and the Stokes U parameter and they come with a cosine of twice the angle and the sine of twice the angle so this is the intensity breaks up in in this way if you if you're trying to measure it does that make sense I'll be showing plots of the Q and U and temperature a lot so if that doesn't make sense you should slow me down and just ask me again okay so if that makes sense to everyone then let's move on and look at some of the maps so this is really the kind of data you get out of W map plank all the the C and B experiments what they give you are the maps for delta T which is convert so really what's measured is the intensity but it's the first thing they do is to convert it by multiplying or dividing by by this factor this isn't strictly speaking maybe quite what you want to do unless you're actually looking at the C and B but everyone understands that that's the convention the reason maybe it's not what you exactly want to do is because there's also foregrounds which have a very different spectrum but it doesn't matter because it's just some constant you're multiplying the maps by and you can undo this but so typically at least for the frequencies below 353 gigahertz plunk gave you the these temperature maps and then the maps of Stokes Q and then I'm not showing Stokes U but it looks similar to the Stokes Q parameter for the for the plunk measurement so there's these kind of maps you get and then they indicate where the intensity is higher or the temperature is higher you see along the galactic plane obviously it's higher and then you also see that in the C and B there's these intrinsic fluctuations for some some parts that are a little hotter and some parts that are a little colder there's the cold spot and I don't know there's Stephen Hawking's initial somewhere here along the galactic plane so in any way these are the the maps you get from the experiments and we don't usually so in if you do a theoretical calculation you cannot really predict what the map should look like you can only predict its statistical properties so what you want to compute are the the correlation function so you want to compute what is the the correlation between a temperature fluctuation in this direction if it's high here how likely is it that it's high also in that direction similarly you can correlate the temperature fluctuations with the Stokes parameters or the Stokes parameters with themselves and then of course you can also look at higher order correlation so you try to it's a little bit difficult with the two arms but in principle you can correlate three points in the in the in the maps now this is still not what's usually done so for data analysis it's usually more convenient to use multiple coefficients these are the ALMs that you've probably seen before so what you have is delta t is a function on the two sphere so the the natural thing you do in in physics when you have functions on the on the two sphere you just say well I can expand them in terms of spherical harmonics which are the eigen functions of the Laplacian on the two sphere so you can write ALM YLM of of n hat and here I'm just integrating this against another YLM to get the ALM out so I'm sure you've seen these kind of things before so you have these multiple coefficients for temperature and then similarly for polarization you can expand the Stokes Q and U parameters except in this case they're not really functions on the on the two sphere but they are components of a symmetric two tensor on the on the two sphere and so you should expand them not in terms of the spherical harmonics but in terms of what are known as spin two spherical harmonics so these are the eigen sections of the Laplacian acting on these on these symmetric two tensors but otherwise the conceptually it's the the same thing so you just take the map Q of n hat plus I U of n hat and you decompose it in some other set of special functions so this was AT and this is AP LM and these are the spin two spherical harmonics of n hat and then they again are orthonormal so you can just extract the AP LMS by integrating against the spherical harmonics and typically so the temperature multiple coefficients are the ones we're actually using these ones are not typically used because they don't have nice transformation properties under parity to fix that what you can do is define new linear combinations of them which are called E and B because of their parity transfer properties under of the under well because of their transformation properties under under parity so AELM which I'm defining here transforms in this way so it goes like minus one to the L times AELM and this is how an electric field would transform or or a gradient so this would be like a gradient of a scalar field would transform in that way and we'll see later that that's relevant and then for a BLM these are they have an additional minus sign under under parity and so they transform like a B field if you wish so that's why they're called E and and B modes and they essentially correspond to a curl okay so what you then do instead of computing the correlation functions you actually compute the what are known as the angular power spectra so you look at the AELM A star LM and you take an ensemble average so your average here over all possible realizations and then the Wigner-Eckart theorem tells you that it has to be of this form the ensemble average has to be of this form to respect rotational invariance so it has to be independent of m and proportional to delta m and prime and it can depend on on L but has to be diagonal in L and this is true for all of them and you just define the temperature angular power spectrum as the ensemble average of the Tp and the Te cross spectrum as the ensemble average of Te, EE and BB just like they're here so if you if your theory is Gaussian then all the correlation functions are encoded in the correlations in the in the two-point function the two-point functions are defined here and then if you're interested in higher endpoint functions you know that in a Gaussian theory the odd endpoint functions vanish and the even endpoint functions are given by sums of products of the two-point correlation functions and so this is really all the information that you need to specify the statistical properties of the fluctuations in a Gaussian universe and so far we don't have any departures from that at least in for in a from primordial non-gaussianity there are non-gaussianities from a late-time evolution obviously so these so these are the things that can be calculated and we'll see how you calculate them later on you can also measure them from the sky so these angular power spectra are really how you make contact between the the theory and the the measurement now how do you measure them you take the the map we'll see it in more detail in the in the next lecture but you you take the map and you decompose it you compute the ALMs and then this is somewhat schematic this is only true if you had a full sky map but if you have a full sky map you just take these numbers take the absolute value squared and average it over m and this is the an estimator for the angular power spectrum for the observed angular power spectrum and there's two things that one maybe should point out for these estimators so as i already said this is a somewhat simplified version and it assumes that you have full sky maps in practice you to be you basically never have full sky maps because you have to at least cut out the galactic plane and point sources but it's known how to correct for it so we'll see a somewhat more complicated version of this it's some some later point and one thing that's interesting about this estimator is that it's unbiased so if you take if you do a number if you do simulations so you simulate what a sky would look like and then from that sky you do a measurement and then you average over many simulations then it returns for you the the angular power spectrum that you fed in so it's not necessarily true that this is a true realization by realization but statistically you recover the the angular power spectrum that you started with if you average over different realizations so it's unbiased in this sense but there is cosmic variance so if you compute the the variance then you so you compute for each realization you measure the cl's in the way that's described here and then you subtract the theory spectrum that you fed in and and square it if you average this overall possible realizations you get something that's non-zero so this is known as cosmic variance and it's always two over the the number of of modes times the square of the the initial power spectrum so these are the two properties and these are usually also shown in some of the plots as gray bands to indicate the the cosmic variance so this is just something you have to pay attention to because we're only observing the the sky once so we can't really average over it we're only observing it once so it's likely that we're not measuring exactly the underlying angular power spectrum but something that's a small I mean some fluctuation away from it and this tells you how far away you might have fluctuated okay so now we'll look at some of the early measurements of the unless there's questions about it's if there's questions about the questions about this part maybe I should ask yeah so I'm saying you have a theory that predicts for you the cttl yeah this is a theoretical part so here when I'm writing this what I what I have in mind is that you have some some theory so we have a theory and it predicts for you some c tt comma l we'll see how we get there from some inflationary model or from some model of initial conditions plus physics of the baryon photon plasma we can compute this and then now what I'm saying is there in principle from this theory you can simulate so you do so obviously we're only looking at one such simulation but to understand what this assemble average is is you're really averaging over many different realizations so you have a theory and then it gives you some realization of the sky so you get some atl m so this is let's say the the first simulation or something like this and from this simulation you can measure the angular power spectrum in in this simulation so for the first simulation you measure some angular power spectrum which is one over two l plus one sum over m a t l m one absolute value squared and this is the analog of what I'm calling observed here so you do this and then you do this for the for the second simulation a two and so on so you have a bunch of cl so you have the c i t t observed and then for each of them you can calculate this quantity so there's some departure of the one you measured in your simulation from the the true underlying power spectrum and I'm saying you square this and then average this over the the simulations and then you should find this this kind of quantity and it's something you can show fairly easily analytically actually so if you uh I didn't want to go through the the exercise but we can go through the through the exercise later if you're interested but if you just take this estimator so you say you have an estimator c t t l that is one over two l plus one sum over m a t l m squared and now so this is the the estimator and then there's some some average cl so there's c t t l is the average of the cl hats and you can analytically using the properties of these namely a t l m a star t l prime m prime is equal to cl delta l l prime delta m m prime you can use this and this definition to derive this if you want so but this is this was in any case the idea the two factor comes because you have two ways of of contracting so if you look at the at this correlation there will be four a's everywhere so you're squaring this so there will be a l m a l m and then you can the the the the disconnected piece goes away this is what you're subtracting here the subtracts out the disconnected piece and the for the connected piece you can have so you have four a's a a a so that the connected p the disconnected piece so this one goes away you're subtracting that with the with the rest and then there are the two connected pieces and here each one of them has a delta l l prime so you kill one of the sums so in the square you have two sums and each one of them kills one of them does it make sense okay more questions here at so this is the large uncertainty on large angular scales at low l yeah right yeah i mean it's still so the cmb measurements right now or plunk in the temperature power spectrum is signal to noise larger than one out to maybe l of 1500 l of 1800 or so so you are sensitive to this out to fairly large l i mean you just measured it so well the noise is so small by now that you are sensitive out to a fairly small fairly small angular scales but yeah certainly it's it's very small so if you if you look at it in the plots it's clear that the cosmic variance just blows up on large angular scales and that's why we're typically discussing it because we just have a lot of modes out there but in principle this is still a contribution to the to the error bars in different frequencies but there shouldn't if it's true that it's a black body then there should well if you're assuming that there's no foregrounds let's let's ignore foregrounds for a second but then there's no additional information if you measure it at different frequencies because you're just looking at the same same spectrum at different different parts of the spectrum so we had the spectrum which was e to the h new over k t minus one if you know that at one frequency you know it everywhere if you know the the temperature and you've measured it at one frequency you know the spectrum everywhere so there's no additional information in that sense uh is that what you're what you're asking i mean it's the the perturbations again are proportional to the derivative of this but that's the same same argument i mean you know the spectral dependence and if you assume that that's your spectral dependence and you know the temperature you measured it at one place uh then it's it's fixed with foregrounds obviously this is no longer true because then the foregrounds have different frequency dependence from the from the cosmic microwave background and that's why we do measure it at many frequencies i mean plunk for example measured it at nine frequencies make sense yeah for now i'm assuming that the cmb is gaussian yeah and it's a a good approximation in the sense that we don't have departures from gaussianity from from primordial physics in any case so there are some effects that uh introduce non-gaussianity but not not at the level i'm i'm discussing i mean yeah but here i'm assuming it's gaussian and i think enrico will explain to us why we expect it to be at least close to gaussian okay so now let's look again at some of the old measurements and one of the old measurements that uh is from the uh relict uh satellite so this was a measurement that actually uh claims to have detected the the quadrupole it's sometimes it's not always quoted i'm not totally sure if this is just a a cultural thing that uh people from uh europe and i don't know i'm anyway i'm not sure why it's not uh mentioned more this is a russian experiment one objective criticism you might have is that it was a measurement at a single frequency so if you see fluctuations in at a single frequency you may be not entirely sure if it's foregrounds or not but this this was discussed and so this is the the claimed first detection of the quadrupole the measurement of the quadrupole and higher order multiple moments that everyone is familiar with obviously comes from the uh kobe uh spacecraft and in particular for the dmr experiment on kobe that was led by by smoot and that's the experiment that he got the Nobel prize for and here you see the the map the kobe map and we'll see how the the maps evolve i mean this is said relatively uh low resolution compared to what we now have but this was the first measurement of the fluctuations and it gave a quite low uh quadrupole and we're still stuck with a quite low value of the quadrupole so then from there on this was a space mission from there on there were a number of experiments so here the black dots i should say are the kobe power spectrum kobe angular power spectrum out to uh l of about 30 and then the uh additional experiments from the ground started measuring on on smaller scales and uh at this time it wasn't yet clear really what generated the initial perturbations and one of the theories at the time was that the perturbations that have to be there we know that there had to be perturbations because otherwise we wouldn't have stars and galaxies around us but one of the theories at the time was that uh the perturbations might have been generated causally there's certainly a a nice uh nice idea might have been generated causally by by strings and monopoles and then you see the prediction of these models and at the time the statement was that the data does not favor these models i think it's fair to say that if you take the data here seriously it's already ruling out these models these experiments are mad toko saskatoon so they're also experiments that are not usually credited all that much but here you're seeing the the first evidence for the first acoustic peak and then the experiment that everyone knows from uh a few years later is the combination of boomerang and maxima which really had a nice measurement of first and second and maybe third i mean that's maybe a little too ambitious but certainly the first two acoustic peaks from from boomerang and maxima from from 2001 and then as you fast forward so i don't want to go through all the experiments but as you fast forward to doubly map nine so this is the nine-year doubly map data you see how the measurements really quite rapidly become better and better and you see you clearly see the first three peaks and here the gray band is what i was mentioning earlier that so the gray band indicates the the cosmic variance that we were discussing so you see that it's it's dominating on large scales as you would expect because it goes like one over two l plus one and then here you see the measurement that enrico yesterday was referring to the sign of the te cross correlations you see this negative sign which really tells us that these these perturbations were generated even before the universe was first filled with a with a hot and dense plasma so there was some process early on that generated these perturbations and we're seeing these adiabatic perturbations today in the in the c and b and the the reason you cannot draw too much of a conclusion from the temperature perturbations as we'll see later is that temperature perturbations can be generated at later times as well whereas polarization you can only generate when there's free electrons around which happens at realization but that only contributes at large angular scales and then during recombination so here really getting a measurement of the velocity potential as enrico was also showing from the from the c and b in a fairly model independent way and then here five years later are the the measurements from punk and you now see seven acoustic peaks so it's a very nice nice measurement and in what follows i'll describe how or the underlying equations that are being solved how to compute the angular power spectrum and then the next lecture i'll say a little bit more about how these angular spectra are actually measured so you will see from from both sides what this plot is showing so eventually we'll try to understand the lambda cdm prediction that's shown here in red and we'll try to understand where the where the blue dots come from okay to understand how to compute the angular power spectrum we have to go back and do some more perturbation theory general relativity perturbation theory and what we've seen before was for the homogeneous case we had a line element of this form so minus dt squared plus a squared the x squared and then a stress tensor that had a zero zero component which is the is measuring the energy density and then a spatial component that's measuring the the pressure you have some some perfect fluid if we now want to describe the the perturbations you have to do perturbation theory around the solution in in general relativity and rico already discussed it so this will be somewhat brief but what you do is you just perturb the metric so you write the metric as the the background metric plus fluctuations which i'm writing as h zero zero h zero i and h i j and then in a similar way you perturb the the stress energy tensor so you have some perturbations to the energy density you have some delta t zero i and you have some some perturbation to the to the spatial part and and rico went through this so i will just show it but under an infinitesimal coordinate transformation so you send x to x plus epsilon mu of x you can show that the the metric perturbations transform in this way so there's the zero zero perturbation of the metric shifts by the time derivative of the zero component of of epsilon and then you have these transformation properties for the zero i component the i j component this tells you that you can gauge some of these away so you can use the epsilon zero for example to gauge h zero zero away then this is something that's fixed but you can use then epsilon i to gauge h zero i at least the the scalar part to zero this is what's called the synchronous gauge and that's what i'll be i'll be using this uh yeah so this is the the gauge that i'll be using and rico will probably use different gauges but you're just really changing uh changing coordinates so there's nothing i mean obviously in in detail changing gauges and so on can be tedious in using one gauge rather than another gauge can be very convenient but conceptually there's nothing nothing deep about it so you just change your your slicing and change the coordinates on the on the slices so in synchronous gauge then if you gauge away the zero zero perturbation and the zero i perturbation the line element just looks like this it's nice and simple and the the perturbation to the stress tensor has a piece that's the perturbation to the to the energy density and then you have a a velocity of the perturbation to the velocity for velocity of the fluid and then you have a perturbation to the pressure some the remaining metric perturbation and then there's something that's referred to as anisotropic stress i don't think i'll say too much about the anisotropic stress but in in principle it's something that's there it's it's relevant for relativistic degrees of freedom like neutrinos degrees of freedom that free stream in principle in the equations it's also there for the for the photons and so as Enrico already discussed you can decompose the perturbations into scalar vector and tensor perturbations so for example the perturbation to the to the velocity you can write as a derivative of a velocity potential plus something that's a transverse so di of delta ui is zero and similarly you can decompose the spatial part of the metric in terms of a perturbation to the trace and then di dj times b something again with a vector that's that's transverse and then the transverse traceless gravitons so this is the way you can decompose them and at linear order they really because of rotational invariance they don't mix the same decomposition works for the anisotropic stress and i'll mostly be interested for now in the the scalar perturbations so these are the the perturbations in the the energy density in the in the plasma these are the ones that we were looking at for the temperature perturbations and for the te cross spectra obviously later on will be interested in the the tensor perturbations or the primordial gravitational waves that you can look for in the c and b through the the polar the b mode polarization okay so yeah so as i already said this and riko said it so i don't have to spend time on it but rotational invariance guarantees that these don't mix and you can just look at the scalar sector separately the vector sector separately the tensor sector separately and the vectors if they're not sourced they rapidly decay so usually in inflation we won't really talk about the vector perturbations we talk about the scalar perturbations and the the tensor perturbations and now one point that i briefly mentioned earlier and this is something one can do obviously no more rigorous way but earlier i said the transformation properties of a elm correspond to a gradient the transformation properties of the ablms correspond to a to a curl that you cannot really make a curl at linear order perturbation theory out of a scalar so there's no ablm in the in a in a theory that has only scalar perturbations and so the scalar modes really only generate temperature anisotropies te cross correlation and e power while the vector modes and tensor modes generate in addition to that also a b modes and so the the vectors as i said they decay so we won't really talk about them anymore but the this this simple fact following from the transformation properties of the e and b modes tells you that if you see a b mode polarization you're actually looking at tensor modes in at least in the context of inflation it changes if you have something that sources them like cosmic strings and so on okay so now let's look at the equations that we have to solve to compute the angular power spectra and to work them out we have to first figure out when we should begin our calculation and yesterday i already mentioned that i'll really be interested in temperatures below 10 to the 9 kelvin so for the most everything or most things happen in thermal equilibrium and we're certainly for the for the c and b at temperatures above 10 to the 6 kelvin we have a nice black body spectrum and so we should start above 10 to the 6 but not temperatures too high that makes our life difficult so we'll start at temperatures that are low enough so electrons and positrons have disappeared and all we have in the universe are some helium nuclei protons electrons photons that mediate the interactions in the in the plasma the neutrinos maybe a cosmological constant or dark energy but this is mostly relevant at late times and then a dark matter so this is will start just below 10 to the 9 kelvin and with this matter content so we'll try to write down the equations of motion that describe the universe from 10 to the 9 kelvin to the present with this matter content and use them to compute the angular power spectra does that make sense okay so then for the electrons and protons we know that they they scatter very efficiently and we can describe them as the as the baryon fluid so this is what i'll call baryons even though electrons clearly are not baryons but in cosmology electrons are part of the the baryons for cold dark matter because it's very non-relativistic you can also describe it in a in a hydrodynamic way you just have to keep track of the the energy conservation for the neutrinos because they they're very light they they free stream and have an isotropic stress and so we usually describe them by a bolsman hierarchy now i'll introduce this thing next and for the photons certainly if we want to keep track of their polarization which we do want to keep track of because we're trying to compute the e c l e e c l t e so the the power in in polarization then we also have to describe them by a bolsman hierarchy and so i'll try to go through those those equations so before i don't want to write down the the full equations right away because they look a little frightening maybe if you haven't seen them before so it's some some large a large ish system of equations so instead what i want to do is look at a somewhat simplified toy model so we're imagining having some a gas of of massless particles it's thermal and so the only simplification i'm doing here or there's a number of simplifications so first of all i'll work in flat space for now i will not take into account the expansion of the universe and for now i'll ignore the the helicity of the photons so i only essentially describe a scalar particle and so instead of keeping track of all the particles what you want to do is you want to describe them in terms you want to coarse grain the system and describe them in terms of the the phase space density so the the phase space density looks like this so you're just summing overall the particles and have a bunch of delta functions where the particle is localized and where its momentum is so this is completely classical obviously so but so we we have this this phase space density and then what we're trying to understand is what is the equation of motion this satisfies we know what the equations of motion of our particles are if you have a particle that's so these are massless so they're moving at the speed of light so the x by dt is just the direction of the the momentum and there's no if we assume there's no forces no collisions then then we just have the momentum conserved so it's just a bunch of free particles that okay just a bunch of free particles and so the question is what's the equation they they satisfy and it's easy to actually if i only have five minutes maybe i should ask for questions instead because yeah okay well it's just that i'm starting something new in a sense so i'm saying either way so i can i don't know if there's any any preference if there's questions you can ask questions otherwise we'll look at this for a little while longer maybe but okay well maybe let's try to get to the through the flat space apart and then generalize it after the in the next lecture all right so what you can do is from this from this phase space density you can compute dn by dt the partial derivative with respect to the with respect to time and then you see that it will act on on this thing here and this so if you have the sum over r and then the derivative of this with respect to the acting on the delta function that has the momenta just a zero so the only piece you have to worry about is this piece and you get a derivative acting on the on the so so this is well d by dxr xr by dt of the the delta function and then d by dxr i can trade for a minus d by dx which i can pull out and here's the delta so this is x minus x r p minus pr of t and then this is just the the direction of p hat r so this is minus d by dx sum over r and it's pr hat times delta x minus x r delta of p minus pr of t and so this again because of the delta function i can set equal to p and so this is minus p hat dot the the gradient of n once i pull it out i just get the phase space density again and i get some collisionless Boltzmann equation that's satisfied by the by the phase space density and so as for the for the photons so we should keep in mind the the photons obviously we shouldn't just study this system but as for the photons the intensity and the the temperature are related in in this way so the temperature perturbations are related to the intensity perturbations again just by doing a Taylor expansion of the the intensity and so if you imagine doing a measurement that measures the intensity perturbations over the whole range of frequencies then you can convince yourself that the integral over delta i nu is just for delta t over t times the integral over the intensity and this should be clear because what you're in what you're computing here so you know that role with for the for the gas it goes like the temperature to the fourth power and so if you compute delta row over row this is just for delta t over t so this is what i'm essentially writing here and so this is the quantity we're interested in and so it's it's natural to define this object so this is the perturbation to the intensity so this is the perturbation away from the thermal distribution for my for my gas of particles and i'm integrating overall the frequencies so i multiply by another p to get the intensity from the number density and i have i keep track of the direction so this is the temperature perturbation if you want or the perturbation to the intensity add some position x in this in this gas for contribute and the contribution for particles with momenta in the p hat direction or if you have a device there then again it would be more natural to replace this by by minus n hat so you might want to know what's the the temperature perturbation in the gas looking at at some point if you look in the n hat direction so this is what this this quantity measures and so instead of the the Boltzmann equation for the phase space density will now be interested in deriving an equation of motion for the for this for this quantity and it's very easy because it just inherits the its equation of motion from the from the Boltzmann equation i mean if you do a perturbation theory you just put deltas here so this is the equation that you that you get so the the delta satisfies that same same equation and you have an equation of motion for the for the temperature perturbation at a position x in in the direction p hat that looks like this so you get an equation of motion that looks like this and now what you see is that this equation is translationally invariant this is something that Enrico already also explained to you that if you have translated you probably know it anyway but if you have an equation that's translationally invariant and it's linear then it's very convenient to look for Fourier solutions because it diagonalizes this this operator and in particular so what we're doing now is we're taking a superposition in terms of plane waves of this form and by rotational invariance so this is rotationally invariant it should only depend on the magnitude of q and the the inner product between q hat and p hat so does that still make sense so before we were looking at so before we were looking at the the contribution to the density perturbation at position x from from photons or from particles with momenta p hat now what we do is we Fourier transform this and so we have some some wave crests if you want so there's we're looking at particles with given direction so all the particles here so maybe you have some over density somewhere so there's a direction p hat and then you have the Fourier momentum that's perpendicular to the plane waves so you have a setup that looks like this for the so this is what the delta of q and q dot p hat is is describing does that make sense so we're Fourier transforming the initial density perturbation and we we look at a given given Fourier mode which will be plane waves in some in some direction that makes sense okay and so then the the reason i'm drawing it is because it will be kind of obvious what this what the solution to this equation is the the so if you look at this so the particles move in the p hat direction if you have p hat perpendicular to q so if the particles move in in this direction then obviously this quantity doesn't evolve at all they're all just moving along the the wave crests and this is what you're seeing here when they're orthogonal the mu which i called q hat dot p hat is just zero and this quantity just doesn't evolve on the other hand if you have them parallel so you have q dot p parallel then you just get one here so you have a plane wave that goes like e to the minus i q mu times t this is the solution and so you just in the case where this becomes one you just have a plane wave that's moving at the speed of light which also makes sense because now the particles are moving in this direction so you get these plane waves traveling at the speed of light so these are the solutions of these equations it's just describing these simple cases where you have these plane wave solutions so does that make sense these are the basic physics of the Fourier transform of these things okay and so what we're interested in is the the temperature perturbation at some point in the in the plasma looking out in some direction n hat and this is just the one quarter we saw before so it's just one quarter of this is essentially delta row over row of the of the plasma and then you have you're interested in it at the at the origin let's say or where you're sitting and you're interested in it not so p hat again is minus n hat so the momentum of the photon is minus the the direction you're looking and what you can do or what we're interested in for the c and b as we said are these multiple coefficients and so you just Fourier not Fourier but you just integrate against the spherical harmonics and you get multiple coefficients that look like this in terms of these gadgets that are obtained from these ones by expanding in terms of Legendre polynomials so does that all still make sense there's a lot of gymnastics but we'll eventually be interested in as you can see these are useful because now if you take the angular power spectra they're directly proportional to the delta tl square so these are really the pieces the underlying pieces in the in the C sub l so these are the quantities you actually want to compute that's why i'm going through this gymnastics and so what we'll do next is just generalize this to the expanding universe and include interactions but maybe i'll stop well let me show this one still so you just plug this into the underlying equation so we had a very simple equation to begin with and then we Fourier transformed and we expanded this in terms of the Legendre polynomials and these guys satisfy a system of equations the reason they're coupled is from the q mu term that i was just describing to you a second ago on the previous line so there's this q mu which will couple the the different pieces and so you get a system of coupled differential equations and these are called the the Boltzmann hierarchy and we'll extend it to the case where you have perturbations in the in the metric so in an expanding universe and to include interaction so i i think maybe i'll i'll stop here and then we continue with that in the next lecture thanks


As of Tuesday December 15, 2020, St. Nusha has recorded a total of 278 confirmed cases of COVID-19. A total of 17,801 tests have been processed to date. We also have a total of 243 recoveries, 31 active cases and 4 total COVID-19 related deaths. There are no critical cases in care presently. There are no new cases reported today, Tuesday December 15. We take the opportunity to acknowledge the communities, the business sector, the faith-based organizations, the hospitality industry and other partners who continue to support the national effort in the management of COVID-19. The Ministry of Health and Wellness continues to monitor both the global and regional situation and analysis as we assess our risks and make recommendations. We note the steady rate of increasing cases in most countries as they experience their second wave. We continue to monitor closely and we learn from the best practices implemented as we strengthen the existing protocols. As countries open their borders and social and economic activity increases, the risk for transmission also increases. St. Nusha, just like the rest of the world, remains at high risk for the introduction of COVID-19. Our main risks include the increased numbers of returning nationals and visitors from areas of high incidence of disease, the illegal entry through the borders, the non-adherence to protocols, the bridges to home quarantine and the hosting of mass crowd social events. Every visitor returning national and illegal entrant must be managed as a possible positive COVID-19 case. It is imperative that we remain vigilant at this time to keep ourselves and our families safe. As we celebrate the Christmas season, let us all remain on alert and continue to play our part to protect the health and safety of our family, friends, colleagues and neighbors. We ask all to minimize their movements, avoid social gatherings and other activities which will bring about greater exposure to the virus. Let us continue to practice the infection prevention and control measures to minimize the spread of COVID-19. Wash your hands during the day using soap and flowing water, wear a mask in public places and ensure it covers the mouth, chin and nose and to maintain a six-feet physical distance from others. Use a bleach solution to sanitize frequently touched surfaces and frequently handled objects. Avoid contact with other persons who have flu-like symptoms. If you are experiencing flu-like symptoms, keep away from others and seek medical care at the closest community respiratory clinic. The Ministry of Health and Wellness will continue providing updates as new information becomes available.


I believe you can answer your own data analysis questions. Do you? If you do, stick around for another episode of Code Club. I'm your host, Pat Schloss, and it's my goal to help you to grow in confidence to ask and answer questions about the world around us using data. I love starting new projects. Everything feels so simple, so obvious. Then weeks, months, years after starting, things have gotten complicated, confusing, and tedious. We're ready for it to be done already. With each new project, I tell myself this is going to be different. Well, today is the day we're going to start a new project. We have the goal of answering a question that is important to those of us interested in analyzing microbial communities. To what degree do Inter and Intra genomic variation limit the interpretation of Amplicon sequence variants, also called ASVs? Along the way, we'll learn different elements of what it takes to make an analysis reproducible and how to use a variety of tools that will help us out. We'll get started in this session with using the command line to set up our project directory. Even if you don't find the problem we're studying interesting, you'll hopefully find the approach generalizable to a variety of problems that do interest you. I'm a microbiologist, and one of the things I spend my days and my career doing really is sequencing a gene called the 16S ribosomal RNA gene. This is a gene that's found in all bacteria in archaea. There's a gene called the 18S ribosomal RNA gene that humans and other eukaryotes have in their genomes. This gene is very popular because it has a signal with it that tells us something about who the gene came from, and so it's useful for doing taxonomy of bacteria and other organisms. Over the past 30 years, this gene has been used extensively to better understand the types of bacteria that are in different communities. If you've heard about the human microbiome project or anything related to the microbiome, that's largely been driven by sequencing the 16S gene. Now we have a few different approaches that people have taken to analyzing these data. The first is to compare the sequence to a database. We have a database of sequences that come from perhaps genomes or cultured bacteria, and so we can take my sequence that we've perhaps never seen before, compare it to the database to figure out what approximation who it's related to. This is what I call phylo typing. Other people maybe call it closed reference, but this is what I call phylo typing, where we're trying to figure out, so to speak, a name to put on that sequence. So the second approach that we've been using for about 20 years is to take our collection of sequences, and for a given sample these days, we might have five, 10, 20,000 sequences in a sample, and because we know the databases are limited, what we'll do with this approach is we'll look at the similarity of the sequences to each other within our own sample, and we'll bend the sequences together based on how similar they are to each other. This is what's called an operational taxonomic unit, or OTU, and generally we're happy if we have a collection of sequences that are not more than 3% different from each other, or you could say a collection of sequences that are more than 97% similar to each other. There's a long back history to this, people kind of think it relates to a species definition, but that's pretty wobbly. It's an operational definition is what's important to know, and the other thing that's important to know is that we're assigning sequences to a bin, this OTU, based on how the sequences relate to each other, not to how they relate to a database. The third approach that's more recent, and say the last 10 years, and it's really picked up steam in the last couple of years, is what's called amplicon sequence variants, or exact sequence variants, or oligotypes. They go by kind of their abbreviation of ASV, ESV, and oligotype, and the idea here is that we're binning sequences, again based on how similar they are to each other, but instead of having some room for a variation within an OTU, these do not allow any variation. The name exact sequence variant means that the sequences within that bin are identical to each other. There's no variation. Now, that's a little bit overstated, and so they've kind of relaxed the definition, and so the algorithms people use generally allow for a couple of differences in the sequences. So these sequences that we generally use are about 250 nucleotides, up to about 1500 nucleotides, and so a couple of nucleotides is still going to be less than 1% different. So what we'll talk about are called amplicon sequence variants as kind of a catch-all. I think that's kind of the term that's in use these days, but know that if you've heard of terms like oligotype, amplicon sequence variants, or ASV, or exact sequence variant, ESV, I kind of refer to the same idea. And so there's been a lot of bluster, I think, in kind of very strident positions put forth that ASVs should replace OTUs as the biomarker level of choice of what we should use, right? And so keep in mind that phylotypes, OTUs, and ASVs, they're all kind of a form of an OTU, and so our language is a little bit sloppy, but an ASV is an OTU with a much more constrained level of variation. And so again, this kind of strident take that ASVs should replace OTUs, like I said, I feel like it's a little bit over the top. And I've written a few reviews of manuscripts people have posted to bioarchive, and I've put these reviews online to make them public. And there's a number of reasons that I worry about the, this kind of adoption, the strident adoption of ASVs and getting rid of OTUs altogether. In my group, we use all different levels. And again, I could go into that, but that's not really what I'm interested in here. What I'm interested in here is a particular concern I have with ASVs. And that's that one of the challenges with using 16S ribosomal RNA genes is that an individual genome might have 21 copies. It might have one copy, things like E. coli have seven copies. And so it varies, again, across the phylogenetic tree. There is some conservation within each group. And so that would cause, that would only cause problems perhaps for quantifying the relative abundance of different bacteria. But the added problem is that these different copies in a genome are not identical to each other. E. coli's seven copies, if we look across the full length of the gene, there's six different sequences across those seven copies of the gene. And so when we talk about looking at Amplicon sequence variants or exact sequence variants, so to speak, we run the risk of splitting E. coli into six different bins, six different OTUs. Is that good or bad? I'm not convinced that it's so good. Alternatively, there's also the risk that if we look across genomes, say, at different species of the same genus, that there might be ASVs that are shared across species. And so the ASVs aren't as fine scale as we might think they are. So that's the question that I have, and that we're going to spend the next series of Code Club episodes working on to answer. Now, we could pick any question. That's a question I work on professionally, this area of microbiome research, and it's one of the questions that I've had over the past few years as more of these methods come out and as we see these methods being applied. As I hear people getting comments from reviewers saying OTUs are dead, long live ASVs. So I think it's worth checking out this type of question. And as I said, as we go through, we're going to highlight different concepts and practices related to doing reproducible computational research. To answer this question, we're going to leverage the great resource that's the RRN database, the RNDB, that's housed at the University of Michigan and actually maintained by my colleague Tom Schmidt. He and I share a lab space and our offices are pretty close to each other. This database is quite large. The last time I'm aware that somebody's done some type of analysis like this, looking at inter versus introgenomic variation of the 16S gene was in 2013. And those researchers only had a couple thousand sequences. The RNDB currently has about 16,000 sequences. About 15,000 of those are for bacteria distributed among about, I think, 4,500 different bacterial species. And there are also archaeal sequences from a few hundred archaeal species. So the database is much larger right now. And will really allow us to look at these unique questions as they relate to ASVs. Even that 2013 study wasn't really thinking about ASVs when they set up their analysis. So for today, what we're going to talk about is something that might seem very boring, but is the area of project organization. I've had the problem where I open up a directory from myself or one of the people in my research group, and I just see this pile of a thousand files and there's no organization. So how do we organize our data to make it easier to navigate and work with our project data? One of the papers that's really motivated my thinking on reproducible research in general and project organization in particular is a commentary that was written in Plos Computational Biology by William Stafford Noble. In this, he talks about a variety of factors. But again, what I want to focus on today is project organization. I would really encourage you to go into the notes for today's Code Cook Club episode and to dig up his paper and give it a read. It's really great. So you'll notice that he's a computational biologist and what he's recommending is based on his experiences and the types of projects he's worked on. So I've adapted his system of organization based on my experiences and the types of bioinformatics and microbiome projects that I work on. And so that's what we're going to talk about today. As you proceed with your career, you might find other ways of organizing data and organizing projects that are more effective for the type of work you're doing. And so you might learn from Noble, you might learn from Schloss to make your own improvements that again work for you. But there's a few principles that I hope you keep in mind as you do this and that we've tried to bake into how we organize a project. So the first is that anybody that looks at your project directory should have a pretty good sense of what's going on. They should know where things are. If you want to find the raw data, where would that be? If I want to find the code, where would that be? If I want to find any figures, where would those be? And so those types of questions should be obvious from looking at the directory structure. It's a form of documentation. Keep in mind that the person looking back at your directory structure to make sense of things might be you many months from now. It could also be your PI who's trying to figure things out long after you've left the lab. A few other principles that help you to get to this goal include things like making sure that all the files and data for your project live in one self-contained directory. Imagine giving me that directory for me to put on my computer. Would the project hold up? Would it be cohesive? If you're a file geneticist, you might think of this as being monofiletic. Another thing to keep in mind is that the different types of information should be kept separate. So your data should be separate from your code. Your raw data should be separate from your process data. Your references should also be separate from that. Next, we should also assume that any code that's going to be run on our project is run from the top of that directory. We might call that the project route or the project home. But any operation is going to be done from the root of your project, not from within some sub-directory. And the reason that's important is because if you run your analysis and you of course run into a bug, as everyone does and that crashes out, well how do you know where to restart from? So if you're moving directories as you do the analysis, then it gets kind of tedious to try to figure out where you are. Meanwhile, if you keep everything as if you're running it from the root of the project, then you never really have to worry about where you are. Another thing that I like to do for my own consistency and kind of my own style is that for the most part, any directories, any files in my project should be in lowercase and not have spaces. So I don't really care about how you capitalize things, but do not put spaces in your file names or in your directory names. They only cause problems for bioinformatics analysis. Now, if you need a space or feel like you need a space, instead of using a space, use an underscore or if you have to use a hyphen. With these ideas in mind, let's take a look at a generic project that we'll use to form our own project directory. What I'm showing here is a generic organization structure for a project that we might do. You'll notice across the top that the directory that we're in is the project directory that we'll call lastname underscore project underscore journal underscore year. So the last year would be my last name or whoever's running the project. So it might be Schloss in this case, project would be like a one word or maybe two word blurb describing the project. So I might call this RRN analysis, all kind of concatenated together as one word. Journal, well, that's a little bit aspirational at this point, we're just starting. I hope this could be publishable, but who knows. So at this point, I might just put a couple of X's in there because I don't really know what journal I wanted to go to. And then finally, the year would be the year that we're in. Again, the hope is that this does get published or whatever project you're working on gets published. And so if you've got a series of projects, we could look at the last name, the project, the journal and the year to easily find that project that relates to the paper. You could argue with me that perhaps the year should go first. But whatever what's most important is to have some type of structure to how you name your projects. This again is something that I have found to be useful. Within this project directory, we need a directory called code. This is obviously where all our code is going to live. We'll need a directory called data. Again, where data is going to live within that. I'm I generally have maybe four or so different subdirectories. So one is going to contain my reference files. One's going to contain my raw sequence files, perhaps one is going to contain the files generated by the mother software package. And one might contain processed files or output files that are perhaps more consumable by someone that wants to get access to our data. Okay. We also have an exploratory directory. This might be a directory where I put notes related to different types of analyses I've done, kind of like you might use a laboratory research notebook. And finally, again, hoping that this might someday be published, a directory called submission, which is where I put everything related to writing a manuscript that I would ultimately submit to a journal. And finally, there's two files at the root of this project, one being a license, which indicates how other people can interact with the data. And then the final one is a read me document, which again, we'll put notes in there about the status of the project. And we'll come back to all of these later. The read me file will be very useful down the road, as we start to put more information in about the project, different dependencies, different versions of software we're using, and perhaps we how we do different stuff, I could generate this using finder on a Mac, or using Windows Explorer on Windows. If you're using Linux or something very similar to this as well, I'm spacing on what it's actually called help develop our skills, working from the command line. What I'm going to go over today is how we're going to set up this structure from the command line for our own particular project. Again, being familiar with some of these command line tools will be very helpful in quickly manipulating files, quickly setting up analyses, and then down the road, running analyses from the command line. When you start your command line interface, you'll get an interface that looks a lot like this. The actual syntax that's here on your command line interface might be a little bit different. Mine has my user ID for my computer, the name of my computer, this tilde, and then the dollar sign. The dollar sign tells me that I'm at a prompt, and that the command line interface is ready for me to give it a command, ready for me to tell it to do something. What I can do from here is I can type ls and this will list out the different directories that I have within my current directory. You'll have different directories, but what you should notice in here are things like applications, desktop, documents, downloads, dropbox. I don't know why I have so many dropbox directories. And again, if you're on a Mac versus Windows, you're going to have different directories, but you should get the gist of this. Also, if you look at your Finder or your Windows Explorer window, you should see content of this directory that look a lot like what you see in your command line interface. So for now, this graphical interface to look at the directory structure is a helpful tool, but hopefully we'll kind of develop our strengths with using the command line interface so that we don't have to rely so heavily on using this graphical interface. Okay, so ls lists out the contents of the directory. What I'd like to do then is to change directories that I'm in. So if I do cd documents, you'll see that at least on my computer the prompt changes from this tilde to documents. Now that tilde indicates the home directory. So if I type cd tilde at any time, I'll go right back to my home directory. But for now, I want to be in documents. Okay, so if I want to see what's in my documents directory, I can type ls. If I do the same operation using my Finder window, I go to my home directory, documents, I can see the different directories and files that are in my documents directory. And I can see that those are here as well. One other thing to point out is I can use the command pwd. And this says print the working directory pwd print working directory. And this will then print out the directory I'm currently in. You'll notice that it has forward slash users forward slash pschloss forward slash documents. This tells me where I am right I'm in my documents directory off of my home and my home was users pschloss. The other thing to note is that this path as it's called starts with a forward slash that initial forward slash is the root of my directory structure, the root of my computer, you can think of your computer as being a lot like a tree where it's got a root at the very bottom where there's one directory the root, and then within the root there's multiple directories and as you kind of go out the tree grows. Within my documents directory, I would next like to create the directory that my project is going to live in. To create directories, I can use the mkdir function. It's actually a program. Anything we type here is a program, but they seem so simple that we just want to call them functions, or I do at least. So I'm going to make a directory that I'll call schloss underscore rrn analysis underscore, I don't know what journal underscore 2020. Okay, so this is going to be the name of my project directory. And so it exists. If I look at my finder window in the documents directory, I now see that I have this schloss rrn analysis project directory to move into this directory, we're going to change directories. And so we'll use the program CD. So CD schloss. And this can be kind of long. So if I hit the tab key, it will automatically complete the rest of the directory name for me. So again, I could do CD schloss, and then hit tab and it'll complete it for you. You could also type it all out. But as you'll find, this is very sensitive to typos. And it's very easy to introduce typos when you're doing any type of data analysis. So tab is your friend. So now we're into our project directory. And if I want to see what's there, again, I can type ls. It's empty. I'd like to start filling this with different directories. So I'm going to go ahead and use mkdir to create these directories. Now I could do mkdir code, enter, and then all the others, or I could put all the directories on a create on one line. And so I'm not going to do all of them this way because I want to show you how to do something else down the road here in today's episode. So I'll do code exploratory, exploratory, submission, and data. So ls shows us that the directory is empty. But again, I want to put in directories into my project directory so that I can have this organization. And so there's four directories that I want to have in my project. And I could do mkdir, and I could create each of those in separate mkdir function calls, or I could put them all on one line. And so that's what I'll do here. So I'll do code, exploratory, submission, and data. And now if I do ls, I see my four directories here in my project. I can also create files from the command line. These won't have anything within them. But they'll at least create an empty file that takes up space in my project directory. And so the function that we'll use to do that is called touch. So I just think of it as touching the file and touching it into existence, kind of like a fairy godmother or something, right? So the two files I want in the project route are license.md and readme.md. Okay. So if I type ls now, I see my four directories, and I see my two text files, the license readme, the license md, and the readme md. Now, when you look at this, it might be a little bit hard to know what's a file and what's a directory. So if you want to make that clearer, we can give ls a flag, as it's called, hyphen capital F. And what that does is that puts a backslash or forward slash, I'd say, at the end of our directories, but doesn't change our files. And so it's much easier in this output to see what things in our project directory are directories in which are simple file. As I mentioned earlier, PWD gives us the path to where we currently are, to our working directory. It prints our working directory, PWD. And so this is an absolute path, because it goes all the way back to the root of our computer. Now, I've been calling this Schloss directory our project route. And so what we want to do is do everything from within the project route and relative to that project route. So the path that's relative to the project route is going to be called a relative path. And what we really want to be doing is working with relative paths. We want to describe everything relative to the project route. And so as an example of that, I could do I could make these like the raw data directory, I could change directories into data and then make raw, but I don't want to do that. I want to stay at my project route. Again, because I don't want to change directories, because I'm lazy, maybe, but more so to kind of emphasize how we work with relative paths. So we do mkdir data forward slash raw. And if we then look to LS, we see LS generates what we've been seeing. But if we do LS data forward slash forward slash, we see that it now has a raw directory within that. I guess I can make that more clear using the dash F, right? So I also have other directories that I want to put in data. So I could do mkdir data forward slash references, data forward slash mother, data forward slash processed. And again, if I do LS dash F data, I see the directories that are within data. And this is all doing it without leaving my project route directory. If we wanted to get a sense of the overall structure of our project, we could use LS dash R, which is short for recursive. And this will then output the contents of all the directories. So we see at the top of the project, the project route, we have our two files and our four directories. Code has nothing in it. Data has four directories in it. Each of those four files, I'm sorry, each of those four directories are also empty. And then exploratory and submission are also empty. Something you'll notice is that before code and data and exploratory and submission, we have a period forward slash. So that period forward slash indicates that we're in the project route or doing it relative to the directory we're currently in, if that makes sense. So again, we could do LS, or we could do LS period forward slash, we get the same output. And so that's what the period forward slash means in that output using the dash R flag. We can also combine flags, we could say LS dash F R. And this will add the forward slash the end of our directory names, as well as give us this recursive output of what our directory structure looks like. So the final thing I want to tell you about, and I worry about giving you this power is the power to delete things, right? And so we have the ability to delete directories as well as files. So again, with great power comes great responsibility. There are people that have, I've probably even done this myself, that probably everybody has run into this problem, where you might accidentally delete an entire project, or your thesis, or very important documents, when you delete something in the command line, it doesn't go to the trash, it is gone. And so there's really no, there's no coming back from, from using these commands. So you want to be really careful. In the next code club episode, we'll talk about using version control. And one of the benefits of tracking your project with version control is that if you do it right, you'll always have a backup of where you currently are. So if I want to remove a directory, I can do rmdir, and perhaps I'll do data raw. So I'm not going to have any raw data. I don't know what that would be. And so what that then does is remove data raw. And if I do ls-fr, I see now that my data directory no longer has that raw directory. So we could, but I want that back. So I'll do mkdir data four slash raw. And it's back. Okay. So rmd will delete a directory. Now the thing that perhaps as a saving grace is that directory, if you use it the way I've described rmdir, it has to be empty for you to remove it. So to empty a directory to remove a file, we can use rm short for remove. And I could say, let's remove the license. And again, if I do ls-f, I see that I no longer have the license file in my project root. Now, of course, I want that there. So I'll do touch license dot md. And again, at a future code club episode, we'll fill in the contents of that license. Okay. So again, if I do ls-fr, I can see where I'm currently at. And I see that even though I deleted them, I brought them back and we're good to go. All right. So now I've got three exercises as usual for you to engage with. Go ahead and pause the video. And what I'd like you to do is work through these exercises. And once you've finished, go ahead and release the pause button. And I'll come back and show you how I did these different steps. So the first exercise is to go ahead and quit out of the command line interface that you've been working in. You can do this by typing exit at the prompt. And that should close your window out. Restart your command line interface and go back to your project root. Prove to yourself that you can get back to where you need to be to be doing all this great analysis that we have in store. The second is I'm asking you to put a readme.md file in every sub directory of the project. Use ls with the right flags to confirm that you got all of the readme files where they need to go and do not use the cd function. Finally, I'd like you to think ahead about what questions you might want to answer using this data set from the RNDB. This question of inter and introgenomic variation, what would you want to know? Or what would you want to ask as you think about how you would apply these or think about these ideas? For the third question, what I'd like you to do is think ahead about what questions you might want to answer with the data we're going to download from the RNDB, specifically how they relate to our thinking about the use of Amplicon sequence variants. For the first exercise, I want to be able to get back to my project root directory. I've reopened the terminal. I'm at my home directory. I know it's my home directory because I have that tilde sign. It's ready for me to give its instructions because I'm at the dollar sign prompt. I remember that it was in my documents directory. I can do cd documents. Perhaps I forget what it was called. You could type ls and I look through all my different directories here and I see that, sure enough, there's a directory called Schloss RND, RND analysis. I'll do cd Schloss. Again, if I hit tab after just starting to type the first few letters, it will auto-complete the rest of the directory for me. Hit enter and ls and voila, I'm right back to where I want to be. For the second question, I asked you all to put a readme.md file in all of the directories of the project. So if I do ls-fr to get a full sense of the directory structure, I can see that the code directory does not, that these are the directories basically, that do not have a readme file. So what I want to do is I want to do touch code forward slash readme.md touch and I'm going to repeat this and what I'd like to do is I like to hit the up arrow to get back to my previous commands, raw and references. I can also do touch exploratory readme.md touch submission readme. So again, now if I do ls-fr, I should see a readme file in each of my directories. And so again, I could have written this as one touch command where I just space out the different file names, the different directories where I'm going to put readme. But this works also to kind of keep things separated. But again, we've achieved the goal of putting our readme file and it's an empty readme file in each of the directories of our project. I guess I don't have a readme file in data. So I'm going to go ahead and do that. So touch data readme.md. And so now if I do fr, yeah, so I do have a readme file here in my data directory. For the third exercise, I asked you to think ahead about the types of questions we might want to answer with the data from the RNDB. And so general question I might ask is that if we assume that the data in the RNDB represents some fictitious microbial community, how many different taxa would I see if I looked at the kingdom level, phylum class, order family, genus species, an ASV level? How many and how do ASVs correlate, if you will, with those different taxonomic levels? Are there many more ASVs than strains or species? Or are they comparable? The second question that I might want to ask is whether or not ASVs are specific to a strain. So if I see a set of ASVs, are they particular to a strain of bacteria? Or are they particular to a species of bacteria? Or are they particular to a genus of bacteria or higher up, right? So kind of getting at the resolution and asking what is the resolution of an ASV and where do we run into problems of lumping versus splitting one genome into multiple ASVs or one species into multiple ASVs? And then also for both of these questions, I might ask how does the answer change if I'm looking at a full-length 16S sequence or a subregion of the 16S gene, like we typically do when we're doing like aluminum IC sequencing? Similarly, I might ask, well, what OTU threshold would I use if I want a limit splitting of a species or strain or species into different OTUs? And we can again look at different taxonomic levels there. And I might also ask, because we've got archaeal data here, how the copy number varies between archaea and bacteria. Do archaea have as many copies as bacteria do? Or are they much fewer? And perhaps we don't need to worry about this so much if we're thinking about archaea over back here. It may not feel like it, but we've actually done quite a bit today. We introduced a new project. We talked about project organization and the different principles you want to keep in mind as you go through the kind of keeping track of your project and keeping it organized. Organization is not something we do merely on the first day, but it's something we're going to be developing as we go through our project. Thanks again for joining me for this week's episode of Code Club. Be sure that you spend time going through the exercises on your own to help reinforce your new skills. Even better would be for you to take the ideas we've worked through today and think about how they relate to your current projects. I'd love to see what you did. Please feel free to drop a line in the comments below to tell us how you organize your projects and what you like about your approach. Also, please let me know what types of data analysis questions you have and I'll do my best to answer them in a future Code Club. Be sure to tell your friends about Code Club and to like this video, subscribe to the channel, and click on the bell so you know when the next Code Club video drops. Keep practicing and we'll see you next time for another episode of Code Club.