OK. Thank you. Sorry that I might go over time. I'll try not to, but say what I need to say. If you need to leave, leave. Leave now if you need to leave. So I thought I'd start by advertising. We have a special year in number theory at the Institute, the CRM, Sonto Arecious Mathematique in Montreal. There are four conferences in analytic number theory in the autumn semester. Those are the dates. Those are the titles. The third one is on the boundary of analytic number theory and arithmetic geometry. I mean, beautiful. Nowadays, about the work of Baageva and several strong people like Venkatesh and Ellenberg is that there's not a lot of difference between for them algebraic methods and analytic methods that all come together. So this will be a celebration of that in November and perhaps more obvious topics in the other three conferences. The first is going to focus large extent on counting points over finite fields, maybe size of Selma groups, this sort of thing. OK. What do you celebrate? Sorry? What do you celebrate?  Who knows? We'll find out soon about fields metals. So yeah, if you look on the website of a CRM, you can get information, information how to apply if you need funding. Also, there may be some depending on where you're from. That's not my prejudices. That's to do with who supplies money to us. OK. So let me get back to my topic. I want to prove Hall-Essers theorem today and prove Linux theorem today. Or a strong indication of Linux theorem. I'm writing you some of the things we've done. Here's the definition of distance. Here's the connection of distance in the size of a Dirichlet series throughout today after some multiplicative function. We have Perron's formula. We have Hall-Essers theorem, which is what we're aiming to prove. I thought I'd start today by talking. Somebody asked me about Siegel's theorem, which I don't exactly want to talk about, but Siegel's zeros. But it's pretty connected. So what is a Siegel's zero? If you have a quadratic character, then it's typically written this way. Actually, I guess it's typically written with beta. That you have some zero of Dirichlet L function where beta is close to 1. And here, chi is a real character. And if you just think about the graph of his 1, there's a zero very close to 1. And you can do something about bounding the derivatives, so the L function can't grow much. So it actually is equivalent to L1 chi being small. Q is a conductor, log Q. And actually, what you can typically do with these things is truncate. Well, at least what I've been sort of saying all long is if you look at sort of all the product, if this is small, i.e. it goes down towards 1 over log Q, there's some sort of predominance amongst the chi's to be, remember, there are any plus or minus 1 in the chi p's. So here, there's a predominance of minus 1 over plus 1. There's some sort of bias. So this is the same thing as saying, and you can make this precise, that chi p is predominantly minus 1, which is, of course, mu p. So that's the same thing. And our language is saying mu pretends to be chi, or is chi pretentious. And actually, rather than most specific, I put all these f in only if I'm not sure why, because these are all very poorly defined. Let me be a little more precise is that the distance squared between mu and chi, and now I'm actually going to restrict myself. So this was p up to x, but let me change this definition a bit and say, if I put q and an interval qx in here, I mean the primes between q and x. So between q and e to the 1 over 1 minus beta, remember beta was the seagull 0. It's very close to 1. So 1 minus beta is very small. 1 over 1 minus beta is very big. e to the 1 over 1 minus beta is really big. If you look at the 1 over log q, oh, the seagull 0 is also defined to be within a distance 1 over log q of 1. So there you see this is sort of a non-trivial interval. And this is the same thing saying that this is uniformly bounded. So there's a direct link between seagull 0s and pretentiousness appropriately written out. So I'm going to be a little bit careless because I don't want to write this down every time. But there's something about distance over a certain range where mu and chi are quite close together. So chi pretends to be mu in some sort of interval. Now one of the great results of analytic number theory is repulsion results. So that says that, well, some of us you may well know well, that you can't really have two 0s of a zeta function very close together and near to the one line, or even two different zeta functions or l functions, which have roughly the same conductor can't have 0s too close together. Now if we were to say have two seagull 0s, suppose you had two characters. Suppose chi 1 and chi 2 had seagull 0s. And the conductors are close together. So they're roughly the same order of magnitude. So this is a Bourguin approximately, a bigger one, to the bigger one. And as I said, this is known to be impossible from analytic methods. There are these various techniques to look at repulsion. So let's have a look using our way of thinking of things. This would mean that the distance between mu and chi 1 is small. And it would also mean the distance between mu and chi 2 is small. So both of those are small because of the seagull 0. But then all I have to do is say use the triangle inequality. And that's greater than or equal to the distance between chi 1 and chi 2. That would also be small. And then just changing things around, that means the character chi 1, chi 2 bar is very close to 1, which essentially means that, well, if we look at what sort of thing does this mean, it means that chi 1, chi 2 bar typically points towards 1 for the primes in that range. And that's going to mean when you form the sum out of this that this thing gets too big. So it's not hard to show that L functions at 1 are smaller than log q, or I should say log q 1, q 2, q being the conductor for chi 1, chi 2. And so you get a contradiction. And so this is how we can prove repulsion using pretentious methods, which is a lot easier than the repulsion from classical methods. And this lemma has a name, which is pretentiousness is repulsive. So this is a strong technique, because it allows us to show that things can't be too close together. Let me give you another example. Suppose you had some, well, all right, let's just do this one. Suppose d mu chi, suppose you have a seagull 0, but you didn't happen to know chi was real. Then if this is less than, less than 1, if I just take twice this, then this ends up being, sorry, greater than or equal to, by the triangulantility, d 1 chi squared. And for the same reasons, this means that the L function of chi squared at 1 has to get big. There is one L function that gets big at 1, and that's the Riemann's data function. So that implies that chi squared must be principal. So in other words, chi is quadratic. So all of the sort of standard results about zeros of L functions in A to 1 can be recovered very easily in this context just by such manipulations. In, we will, I hope, get to the proof of Linux theorem today, or sketched. And so Linux theorem says that for all, so there exists a constant L. I'll write it over here. Linux, we've got a lot of mention today. We learned a lot of great work in different fields, of course. There exists a constant L such that if A and Q are co-prime, then there exists a prime P less than Q to the L, such that P is congruent to A mod Q. So this is less than, less than. So there's a small prime, smallish prime, in every arithmetic progression. And the proof, in all cases, comes to two cases. So in the proof, there's the hard case. Hard case. Let me do the easy case first, or talk about it very briefly. Easy or easier case, which is when there is a seagull zero. So as Terry pointed out this morning, we'll do, maybe I'll write that out specifically later, is that when you're asking for things, objects in arithmetic progressions, you can break it down according to additive or multiplicative characters. An analytic number three is traditionally with multiplicative characters. So if I'm looking for psi x, QA, the primes congruent to A mod Q, I break it down as 1, V, Q. Well, I'll just write it down now. The sum, chi, different ways to write this. Well, for now, I'll just write it easier. Chi bar A psi x chi, A mod Q. Here psi x chi is the sum of chi n lambda n. Oh, thank you. Thank you. Thank you. And what we get is that from the principal character, this is, well, essentially x. And the seagull zero character, so this is the quadratic character, minus x to the beta over beta from the seagull zero beta. And now what I said and I raised, no, I didn't raise. This is my diagram, very meaningful diagram. But if you have 1, 0, it repels all the other ones, the seagull zero. So it means all of the other terms are small. Well, there's some work to be done to justify one about psi. But basically, you can forget them. So this formula only becomes 1 over phi Q x minus chi of A x to the beta over beta. So again, chi is the quadratic character here. Well, obviously, this is big if chi of A is minus 1. But let's focus on chi of A plus 1, chi of A plus 1. So when I say equal, I mean approximately equal. So when chi of A is plus 1, this guy becomes 1 over phi Q x minus x to the b over b. So the beta is very close to 1. So the 1 over beta is more or less 1. And so this thing can be written as, let's take out the x.  Yeah, it's a little bit of algebra, x to the 1 minus beta. It's e to the minus 1 minus beta log x. As long as x isn't too large, this looks like x. So 1 minus beta, this thing in brackets looks like 1 minus beta log x. So we get 1 minus beta x log x over phi Q. And so actually, sort of amusing to note this, the number of primes up to x mod Q is a constant multiple of x. So if you have a single 0, at least in the appropriate range, and it's the same appropriate range from Q to e to the 1 over 1 minus beta, you actually get a positive proportion of the numbers in the arithmetic progression of prime. You have to observe when beta is close to 1, very close to 1, you're going to gain more on the rest of it. The rest of the rest, there are other contributions. You say you can dismiss that. Yeah. But there, you're going to get less also. Ah, OK, OK. So let's assume 1 minus beta is a moderate size. Well, yeah, it is tricky if you if it is possible to do it, but you have to work. Yeah. You can bound with 1 minus beta involved in the error term. Yeah. So this looks a bit like a sieving result.  Am I taking away from your lecture on Tuesday? I'm trying to. OK, Jolly. I haven't finished yet, Henry. OK, so yeah, maybe I'll leave it like that for now. So this is the relatively easy case, though, as Jean pointed out, I was a bit exaggerating how easy it is to dismiss all the other terms if 1 minus beta is really quite small. By Dirichlet's class number formula from the connection I put up there, 1 minus beta is bounded below by 1 over root Q or pi over root Q or something. So it can't get incredibly small, but it can get pretty small. A lot smaller than 1 over log Q. The hard case in the traditional proof is when there's no seagull zero, so you can get a lot of zeros interfering with life. And then there are very, very delicate estimates that need to be proved. And Linux dedicated a lot of time for this and there are other places where you can find work on zero density results that allow you to prove Linux theorem. It's not the approach we're going to take. So OK, so let me go now. We're going to prove how less is theorem, more or less. And then we're going to come back to Linux theorem. So what we said last time, or the first time I talked, was we have Perron's formula. And there was some advantage to breaking the f of n up into two parts, into f of n log n over log x, and then the complementary part. And this complementary part, we noted that the value of the sum would be irrelevant, the fn 1 minus log n over log x. So this is the important part. And that comes out in this integral that when we do 1 over x, the sum n less than or equal to x of fn log n over log x. Then by Perron's formula, we get that's approximately 1 over 2i pi, the integral minus f prime s over log x, x to the s over s. And then our tactic was to pull out f of s from the integral. Actually, we'll write it on here. So this is approximately, we're going to do absolute values. So this is less than, less than.  So this is less than, less than m f x t. This is how big the Dirichlet series gets. Times the integral between minus t and plus t of f prime over f of c plus i t. There's an x to worry about.  So that's just pulling out the f and taking absolute values. And as we said, this loses. And I made some remarks about Cauchy-ing, got a wintz from Terry. But now I'm going to do the Cauchy-ing. But this is Cauchy-Schwarz. So if we Cauchy-Schwarz this integral, we've got two parts. We've got the integral of the 1 over 1 plus t all squared. Well, that's easy. That's a constant. And we've got the more interesting one, which is the integral of f prime over f squared. And as I said, squared, you have a nice thing is you can fold it out. So the other thing I should have said here is that we'll assume, again, that f of p to the alpha is 0 if p is greater than x. We're allowed to do that when we're constructing f. And that's convenient when we look at f prime over f because we make it into more or less a finite sum. Well, actually, if I just put p to the alpha greater than x, we make it into a finite sum. OK, so what we want to evaluate is the integral of f prime over f. And that, well, let me work with essentially the case where f is, well, I don't want to say this. So if we look at f prime over f of s, then where is this going to be supported if f has an Euler product, as it does because it's a multiplicative function? It's going to look like something that's only supported on the prime powers. So over n, where n is a prime power. And I guess the easiest way to write this is lambda fn over n to the s. So it's a sort of similar, I guess, one minus sign to make it all look completely analogous. So with zeta, that would just be lambda of n. So when f is all 1s, this is only supported in n less than root 2x and on prime powers. And the other thing I wanted to say is what happens at primes, which is really going to be the important thing, is lambda fp. When you just work it through, it looks like fp log p. So to do that, you just think of the Euler product. You've got the 1 plus fp over p to the s is how it starts. You put f prime on the top, f on the bottom, and then you see the first coefficient is that. So this is the thing that we want to take the integral of. So it's essentially lambda times something. And at least for log p squared, it's a little more complicated here, but I'm not going to get into that. So what we're really interested in is something like the following integral. So it's from minus t to t, the sum over n up to x of lambda n over n to the sigma plus i t of times a of n. Some constant a of n, which is, in this case, at least the primes f of p. Sigma is c, but later I'm going to want the flexibility. I'm doing a little more. So let's do this dp. And what we need is a good upper bound on that. So what I want to do is see how to get an upper bound on this. Now there's a slick trick to do well here. So let's take a function phi, which is even. So that phi hat has compact support. Actually, that must be wrong. I need to get this right. Well, what I want is for phi, well, what I'm going to do, I'm very good at these things, is extend this from minus infinity to infinity, the same sum. We'll worry about phi in a minute. We'll get something that recognizes at least the integral between minus t and t. OK, so it's probably going to look something like this. Maybe even flatter at the top. OK, so we have a nice inequality. And the idea now is just to expand this. And then we have this integral n2 over n1 to the it phi t over t. So this thing looks like it's very transformed. I mean, you can put the e to the it log n2 and 1. So what we end up with here is phi hat of t log n1, n2. This is what I want, actually, just supported in minus 1, 1. So what would that mean? That would mean that times t. Thank you, Henrik. Yeah, we just changed variables by putting over t and over t. And so there's another t. So if it's supported just in minus 1, 1, what does that mean about the phi hat t, blah, blah? So if t log n1, 2 is less than 1, well, I guess this means that this is less than 1 over t. And so we're going to get n1 minus n2 is less than n2 over t. I'll just put that in absolute value. So it restricts the range of n1 and n2. So let me take the particular case for sigma equals 0 and just work this out. So sigma equals 0, we get the sum n1, n2. I want to do one other thing, which will be what I'm going to do here is fix one of the variables and make the other one very in a short interval. So it's convenient just using an inequality like an1, an2 in absolute value is less than or equal to what? Twice or half the square plus square. Yes, that'll do. OK, so if we just plug that in, then there's a symmetry. We can choose one of the variables first, the other second, and then it's twice that. So it's going to be less than, less than with n1. So we have an1 squared lambda n1 t, the sum over n2 of lambda n2. Just substituting n. But now, actually I put n2 up to x, but I've been wiser to put that n2 was in the short interval around n1. So now we're trying to count primes in a relatively short interval. And well, we can use Bruntitch-Marsh and Selberg's treatment. It's nice and elementary. So there's an upper bound here of less than some constant times n1 over t. So when we plug this whole thing in, the t's disappear rather magically. Now, what have I done wrong? I'm sorry? You are working. It's sigma equal 1. Lambda n, yes. I feel like I put an extra n here. Yeah, because I think it's sigma equal 1, not 0. It's the same. Sorry. You know when you write notes on scraps of paper. OK, well, let's continue with it and not worry too much. So the value of sigma doesn't greatly affect things. I was just trying to do one example where we'll work the thing through. You have to make sigma equal 1, not 0. You have to divide by n everything. All right. Let's try sigma equals 1. I'll do what I'm told. OK, so we get over n1 over n2. But then this is in a range around n1. So this becomes 1 over. OK, and now we get this this way round. It's certainly one option. And then, OK, if the a's, for instance, are like f, that they're uniformly bounded, then this is going to be, and this n is going up to x, this is less than less than log x. OK, so such a technique is going to work for different sigmas. You just have different estimates. And if you put the right thing in, the right thing comes out. So anyway, my point, yeah, I guess in this case, that's what we wanted. Sorry, you were sort of making the right point. I wasn't listening very well. So we have s equals 1. So it's lambda n over n. It should have been sigma equals 1. We get the log x. It's of the square. So we take the square root. Here, our upper bound is bad by a square root log x and a log t. But if we take t to be a power of log x, then the key problem is the square root log. That's really what things are going wrong. So we say to ourselves, OK, if integrating once by parts one is halfway there, then integrating twice by parts is going to get us all the way there. And that's what you might think, but good luck. So in fact, in Selberg's element, to improve the prime number theorem, you can interpret that he did that, that he goes to essentially f prime prime. But it's a hard road to get from there to the proof of the prime number theorem. It's not so easy after Selberg's formula. So when Harper came to us with a new proof of Hollath's theorem that was elementary, and the question was to try and reformulate it in a way that was more analytic. And the key to it, in a way, was that we actually got two f primes over f, not an f prime prime. So eventually, I think it was sound who came up with the following formula, which will just prove, works. And the idea is to have two f primes over f. And it's a rather peculiar generalization of this, say the least, but it works most importantly. So here we go. We're going to add in two new variables. We'll integrate over our favorite line. We're going to add a new variable alpha. And we're going to add a new variable beta. So what are we doing here? So I'm quite finished yet what we're doing. But we're going to take alpha and beta being positive real numbers. And then rather strangely, as s goes up this vertical line, up the line c, so here's s. Then in the first integrand, f prime is shifted along a bit. And in the second integrand, it's shifted along a bit more by alpha and by beta. So we somehow are simultaneously going up three lines, which is a bit of a peculiar generalization. Let's see why this is useful. What we're going to do is integrate over all possibilities. Let's assume that things are nice and convergent. And I'm just going to compute this thing. So firstly, let me integrate over beta. So if I integrate over beta, well, I'm left with integral over alpha. Well, I'll bring that in actually. Integral over alpha. Let's just see what happens when we integrate over beta.  So we get f of s plus alpha plus infinity, so far to the right. So what is f evaluated at infinity? As you go off to the right side of a plane, the only term left is the one. Everything else disappears. And when beta equals 0, you get f of s plus alpha. So that's rather cunning because it multiplies through in there. So I guess we're left with f prime of s plus alpha minus f prime over f of s plus alpha when we do the beta integral. Now we do one more integral. And, well, again, it's the same trick for the f prime.  So f of 1, f of infinity, which if you like, f of 1 minus, I hope I've got the right way around, f of s.  The only thing left is what happens at 0. And so we're going to get minus plus log of f of s, x to the s over s to the s. And so I think I got a minus sign wrong somewhere. But let's now just use, I'm sorry, here. And now it looks prettier.  So the first couple of terms of expansion of e to the log f of s. So now let's just use Perron's formula, assuming it went to infinity. And we get the sum for n up to x. The first term gives us f of n. The second term gives us a minus 1. I'll worry about that in one second.  And then, in fact, if you think about it, the 1 just knocks out the first term. So you get this. So this guy is not going to be important. It's going to be small. And so you've really got what you want, which is f of n. We've got an extraneous term, but it turns out to be small. And what we have in the middle here is what we want is 2f prime over f, or once we pull out an f. So the idea here is simply to, well, OK, we've got to do a couple of things. One of which is we've got these integrals going from 0 to infinity for alpha and beta. But it's not hard to imagine that, well, if you think about Dirichlet L functions, as you go further and further to the right, especially f prime over f, I guess, as you go further and further to the right, they're going to get smaller and smaller and not relevant. So for a small price of a small error term, you can cut these integrals down to mean very short. It's about length epsilon. And the important thing is that, when certainly, it more or less goes through the same maneuvers, but you end up with 2f prime over f. So I'm not going to be very precise, and you can see that already. But what we end up with is something like... So one of the things we want to do, actually, is slightly move the line of integration because we've put things so far across. So let me just say we'll cut alpha and beta down to a nice finite length. So that's without any real cost. And then we're going to do a slight change of variable, which is we're going to move a line of integration as follows. Which is not a big deal. And when we do that, and we do things slightly carefully, we end up with 1 over 2i pi, the integral from minus t to t, of the sum n up to some point x, lambda fm over m to the s minus beta over 2. So you see, if we move c to c minus alpha minus beta over 2, the first one, the line goes to c minus beta over 2. And in the second term, we get a sum over n. And then we've got an f term here, and we'll want to take absolute values. So this is f of s. So I guess one does this before moving the contour. And so let me take this out. Yeah, absolute values. Thank you. So forget that. We've got x to the c0 minus alpha minus beta over 2 t over c minus alpha. And then times a maximum. Well, you can be more precise than this. And now this maximum over f is happening in a box. It's no longer just happening on a line, but instead the maximum of f happens on some sort of box. It's the maximum height minus t and width 1. And so how do you find the maximum for such a thing? Well, you can use maximum modulus principle, if you like, and know that the maximum is going to occur either on the left-hand side or the right-hand side or the top. The right-hand side, it's far enough away that it's going to be bounded, so not very difficult, not very harmful. These are not going to be important, so the maximum ends up being on this line. So sorry, it's a bit technical, that. But you end up with, well, you can see the structure is emerging that we want. We're going to get these two lambdas. There are some technical issues. It's convenient to make sure these primes don't get too big and that they don't get too small. So one wants to kind of cut in a bit. And this will all be written out carefully in the book. One wants to cut in a bit to avoid some issues that come being right at the extremum. To do that, one alters a little bit the identity that I proved up there to sort of big and small prime factors dealt with in a way that allows one to avoid any of these difficulties. With this, when we take the mean square using the lemma that we kind of proved, well, you need the extra flexibility of a sigma, but when you work it all out, it all nicely works out. So we have this one issue about, well, actually probably I'll just leave it at that, so if I don't want to go too far over time. So this is roughly how proof might go. I mean, at least you've seen the main elements. You can see there's a little bit left over with the 1 over 1 plus t term that gave us our log t before, and I said it was like log log x. If one gets bound suitably here, sorry, for here, and I mean there's bounds that come out of this maximum and there are also bounds that you shift more to the right that are just bounded by the zeta function, in fact, then you can manipulate that to change that log p to a log the maximum value. I mean, it's a technical thing, but it can be worked out. Okay, so that's roughly the new proof of Halas's theorem. It takes a little bit of effort to get your head around it, and I certainly haven't done a great job of it, but I think when you read it, you can get to growth. So the main idea I don't think is so hard is just that these 2f prime over f allows you to use the mean square. You win the root log x twice, and that's enough. I'm sorry? Ah, you have a point. I haven't explained this x minus alpha here. We'll give you another log. Yeah, he gets the details from this. Thanks for that. Okay, let me move on to the idea in the proof of Linux theorem, which you'll see comes fairly naturally now, and I only want to start rubbing out from here, I think. I'm going to leave that sum up there. So now what we said last time, I guess, is that when you're trying to do the proof of prime number theorem, instead of using lambda n, if you're a multiplicative function person, you look at the sum of mu n for n up to x. And similarly in arithmetic progressions, when you want to prove a prime number theorem for primes in arithmetic progressions, you can look at instead this thing. And then with suitable convolutions, if you like, you can elementarily show that if you can prove that these are all little o of x over q, then that's going to imply for you the prime number theorem for arithmetic progressions with primes. So this is what we're going to work with, but I want to do the same thing I did in the case of the prime number theorem is I want to go more generally and work with an arbitrary multiplicative function fn inside the unit circle. So we're interested in, and I'll put the phi of q here just to, I don't have to write it out very much. Okay, so this is what I want to estimate and think of f as mu. And then I'm going to decompose it with the characters, and so I'm going to get the sum over the characters mod q of, so let me round, chi of a is the sum of f of n, chi bar n, n up to x. Okay, and what you notice is that this thing is a multiplicative function. Product of two multiplicative functions inside the unit circle is a multiplicative function inside the unit circle. That's good. So we can apply the technology we've created there, and it's going to be convenient to apply that technology but not necessarily using every character in here. So, you know, what I should have said earlier, I meant to say was when we were proving this repulsion principle by showing that, that there was that, well, I've got several things to say. How do I want to say this? So let's just, what I want to say is suppose that one of these is big, okay? Suppose that one of these sums is greater than greater than x. In a minute, we'll apply the technology, but let's just suppose one of these sums is greater than greater than x, then by Hollas's theorem, that means that m of s must be, so Hollas, that implies by Hollas that m f chi-bar x t is greater than greater than one. That's the only way from Hollas's theorem, that can be the case. That implies from over there that there exists a t less than or equal to t, such that the distance from f chi-bar to n to the i t up to x is bounded, okay? Which I'm just going to note, the same thing is the distance from f to chi-n n to the i t. Now, let's suppose two of these terms were big, okay? If two of these terms are big, you'd have this small, and you'd also have it for a chi-2. But then we can just use the triangle inequality again, and we'll get the distance between what? One chi-2 bar, n, and n to the i t2 minus t1 up to x is also small.  Okay, so again, this repulsion principle through pretentious and triangle inequality says two of these, only one of these sums can be large. And I guess this is Terry called it orthogonality, but it's the same thing. The difference here is that you're sort of more particularly working with one modulus than in the large sieve. So, yeah, so now we know that there can be at most one big guy here, okay? So let's pull the big guy over to the other side. So we're going to have the big, there is a big term we'll put it over on this side, and then we've got the sum of all the other terms on the right-hand side. So, let's, so e is all of a moduli mod q minus perhaps someone where that character sum is, where the sum of fn chi-n is big if it exists. So let's, let's try and bound this using the same techniques. So we want to bound the sum, all right, one over phi q, the sum over chi and e, which is almost everything, e for everything, almost, not exceptional, of the sum n less than or equal to x of f of n chi-bar. Now, what we do here is we just proceed as above, as the bit I've just rubbed out, but let me, you'll see that it's very similar. So this is less than, less than essentially one over phi q, the sum over chi's. Forget the alpha and beta integrals, the one over two i pi, just the integral from minus t to t of something like the sum m lambda fm times chi-bar of m over m, well, when you work it out, one minus beta over two sum m n lambda fn chi-bar n and one plus beta over two, okay? So these were the two f prime over f's we had, but now they're twisted by chi in each case. That's a dt that comes in. So plus i t, thank you. So now we're going to Cauchy, but I think maybe you can see what's going to happen when we Cauchy, we're going to Cauchy putting this sum of chi inside, okay? And so when we Cauchy, we get less than, less than, and actually with this whole thing inside, we get two terms of the form one over phi q, the sum chi in e, the sum m lambda fm chi-bar m m to the one minus beta over two plus i t squared dt and then a similar term over here. It should be a big square root, I guess, to the half. And now when we expand, rather like we did up there, we have something extra going on. We have the sum, oh, and we can extend this to all characters, I should say, extend it to all characters. We have something else going on. We not only have the same expansion, but when we expand to all characters, we're going to have m and n in the same arithmetic progression. Yeah? So that's what's going to come out from the part of a sum that's the sum when I expand that chi-bar of m, chi-n one over phi q. That part of the expansion will restrict me to m congruent to n mod q. So the calculation I did before when expanding that, I ended up wanting n two in a short interval around n one between n one of times one minus one over t and n one plus n between m times one minus one over t and m times one plus one over t. And from Brunt-Titsch-Marsch, I got an upper bound on the number of such primes, like m over t, well, with the lambda weight. But now I'm adding in the extra condition m congruent to n mod q. But you can also apply Brunt-Titsch-Marsch with this additional condition, as long as the length of the interval is bigger than q to the one plus epsilon. So when you do that, you win an extra factor of q, phi q. So winning that extra factor of phi q when you go all the way back ends up giving you something like the following, is that, I think I will finish on time, gives you something like that this thing, I don't think this will stay in the middle, will it? It's my big discovery of, oh, maybe we'll, okay. So what we're getting then is that the sum over, in our set E, is going to be essentially less than, less than up to some logs, but the most log of m f, let's just say, the maximum with chi and E, and let's just write this way, the maximum with t less than or equal to t of f twisted by chi one plus i t over log x. So that's when we restrict to some subset of a moduli, there should be also a log of this thing as well. But the point is, winning that one over phi q means at first sight you think, well, if we apply Hollas, we just have to sum the result from Hollas over all of these moduli. But the orthogonality condition with the characters means actually we only have to take the maximum of all of them. It's a remarkable win. So now let me just go back to the beginning, is that we had the sum of fn up to x, we took out the plausible big character, the one that may be big if there is one. So let's just say if all of these f chi's over log x are small, then we've already got Linux theorem. Oh, there's an x, sorry. If all of the f chi one plus i t, if none of them get as big as log x, then we've already won. We've already got a good error term. But it's possible that one of them is big, we proved you can't have two of them are big by repulsion principles. So if one of them is big, we put them on the other side and we get a good error term. So what if we proved, we've proved something like, the following is that some n less than or equal to x of, oh, this is of mu, of course, sorry, with mu of n, or f, I think you've landed in my head, is like 1 over phi q chi of a of some n less than or equal to x fn chi bar n. And in fact, you can win an error term of the form x times something like log q over log x, if you do this carefully, to some power which is 2 over pi, 2 over pi as it happens. So we haven't quite proved a theorem. So, okay, right now I've done it for general f, you'll notice in error-affirmative progressions. So the real question then is, if what we want is if f was mu, we'd want this to be little of x. So we want to know that for, that there's no character for which this is big. So let's suppose that the pnt for error-affirmative progressions fails. And here notice this error term is good as long as x is a big enough power of q. For x equals q to a big power. Then, when we work through this, we have something like, sorry, there should be an over q here. We have a sum n less than or equal to x of mu n chi bar of n, phi of q, sorry. Then this guy is going to be greater than greater than x.  That's from Hollas's theorem. But at the start, I pointed out that that's equivalent to having a Siegel zero. So what we've proved is, unless you have a Siegel zero, Hollas's theorem, just using the Fogonality mod q, gives you Linux theorem, and we're reduced to the Siegel zero case. So just to finish off, how do you deal for Siegel zero case? Well, I told you the classical way. Let me tell you another way. This says that the quadratic character mod q, for most p up to x, the quadratic residue symbol, q over p is minus one. Remember I said if you have a Siegel zero, then a positive proportion of the numbers in arithmetic progression tend to be prime in the appropriate arithmetic progression. So how can we exploit that? Well, there's a nice way in the book of Friedlander and Van Jets in the Bible, the opera of the Cribra. I'm going to give you a slightly more naive way to do it, which is, and this only works in certain cases, but it actually motivates what they did. So if you look at a binary quadratic form, ax squared plus bxy plus cy squared, which has discriminant q, then it's elementary number theory that p divides at implies q over p is zero minus one. Sorry, zero plus one. So let's see. If you look at the values of this binary quadratic form, most of the values, well, most primes cannot divide them. In fact, the set of primes that can divide the values of this binary quadratic form are very sparse. In fact, they're so sparse that when you do the most simple sieve method, the inclusion and exclusion, you can more or less prove the right number of primes using that. If you use a little more technology, you can get a very good estimate as they do in their book. But it is a simple case to do this exceptional sieving. And so anyway, so that concludes the proof of Linux theorem. If you can read it legally using Hollass' theorem, when you can see the ideas, the main ideas that you can get, the details, well, hopefully they'll be cleaned up in the book. Okay. So questions or comments? You start with the formula out there. That means you will never have a reminder better than one of the log x. Well, in Hollace's theorem, as I pointed out before, there's certainly cases where you can't get a better error term. In very special cases, I mean, so in some ways, what we're trying to do in just writing this way is that this is a technique that works for general multiplicative functions and error type of progressions. When you specialize to mu, then you can expect a lot more. And we saw in the proof of a strong form of the prime number theorem, there are techniques to exploit this sort of thing to get better error terms. Whether or not those can be combined, here, I don't know. So perhaps the next question, which I'll pre-empt, is how good a Linux theorem can you get? What's the value of L you can get? Well, that's for someone here to do. The proof has to settle down, and then one can look for a constant. But there is some hope, perhaps, but by combining these two methods, you could do well on the constant. But the current record is 5.2. What's the current Linux record? It's 5. No, it's not. I'm not going to keep it down. And now look at all the things I'm going to do. They're going to get a better result right now. It's engineering work. It's engineering. Anyway, so there's engineering work that gets you down to 5. something. And that's an amazingly strong result. And to do that by any technique is extraordinary. If you look at Heath Brown's paper, well, you'd be blown away by a number of techniques that are incredible in that paper. So I don't think there's any great hope in the very near future we're going to be competitive. But who knows? Somebody has a clever idea. Who knows? So what do you have, 10 pros and cons? I wish we had 10,000. If we had a number you could write down, that would be a good start. I don't know. We haven't tried. We had a previous proof that was somewhat more difficult than this, with the old version of the proof of Halas, and trying to write down a number was a struggle. I mean, it's one of these things you really don't want to do that until you kind of cleaned up the proof. But when you've cleaned up the proof, it probably won't give a good constant. So then you've got to clean that proof, and you've got to look for the bits that are really damaging the constant. So it is the work for somebody energetic and good technique. I'll set as a problem, Terry. I think. I have a comment. It's found that the question is kind of very favorable. This is something that came up in trying to use the theory of the multiplicative functions in connection with the Galois abstractions for the flight of certain questions. So basically, what it looked like is that it would be very interesting to have this called beyond the integers when you are in, say, looking at the algebraic bits. So somehow it could not stand in the literature any kind of work that can be done. I mean, for instance, if you want to extend the proof with your native subprime number here, and say in algebraic number here, then automatically you would run into a problem of understanding is multiplicative functions beyond the integers in such a sense, I would imagine. So I don't know if there is any work that can be done on that. I don't know that there's any particular problem. I mean, class number one probably doesn't agree. No, but I mean, you would deal with ideas that are a number of things like that, right? So you can introduce a distance the same way where you're going to take the problem, etc. So this came up naturally in trying to make some results stronger about Galois function. I think if we really would have needed this kind of theorem in number of fields. So is there anybody who has looked at that? No, and it's the first time I've actually been asked that. So no, it's a good idea. And I would hope that we'll could identify what issues come up because of a class group. Well, we presume the ideals wouldn't be a big issue. Okay, I mean, it's just that it's not what we need to see. No, it's certainly not written. I don't know, have you ever seen anything? No. Also, you'd like to have a structural theorem and a kind of stuff, right? So if you want to put the prime number of theorem in a number of fields statistically, automatically you would, I guess, you would end up with that kind of stuff. One would assume that there would be an analogous theory. It's just that it's not the form of the theorem. Yeah, I mean, if you're using multiplicative functions, it should work. So yes, that was actually my question, I guess. So I think one way of trying to do some of what you're suggesting would be if the theory works for functions which are non-message-bounded by one. Okay, so this is something we've sweated some blood over only to come to a painfully simple conclusion.  This is the pretentious dimension. And so anyway, pretentious density, whatever. So when you bound like this, then it turns out this is the right, it seems to be the right thing to do. The proofs naturally want to behave and Demetrius and Sound are developing some of that theory at the moment. So one thing we did do recently was do the analogies in function fields, partly to try and motivate how to simplify some of this. And it works out very beautifully in function fields. So Perron's former is replaced by Cauchy and Cauchy behaves rather nicely. I mean, it's fairly simple. The other generalization that's begging to be done for obvious reasons is to go to gl2, l functions, which I hope somebody will get going on in the near future. It's not very obvious how to proceed, but there are some ideas around of how to do gl2. We see there is no pretentious character, no pretentious problem on the platform. We'll see, we'll see. There will be a result that, okay, we may not be talking about exactly the same thing, but I'd like to hear afterwards maybe what you're talking about. Also to answer your question, I think a number of things would be more interesting for the analogies, not in terms of the character, but in terms of the progression of the machines, but in terms of the discriminatory. I mean, for example, what's that smallest prime in the queue in terms of the normal? Actually, I was going to say, of course, we would expect this, I mean, sorry. Smaller prime means that prime gets its fleets. It's probably a step forward to extend these two number things to the same extent that there is an expression. It's how much, to what extent, one could get a theory which would be independent of the degree of the exception. So it would be uniform. It sounds like you have an application in mind. Actually, a former student is quite interested in the splitting question. He keeps on threatening to work on it, but he hasn't done so yet. So maybe I'll encourage him. Any other suggestions for more work? Yes, okay. What about some sort of, in the pediatric version, it seems like Hollis is there and says these things of cancellation unless you're an unrunned 5G L1 character?  I'm just wildly speculating. What do you think? Good luck. No, it would be nice. I mean, I don't think there's any, you know, I mean, one of the things I've tried to bring out somewhat, but not fully in these lectures is that there is an algae for classical theory. And one can hope where ideas of the G1 L functions have generalized in different directions. And a lot of a theory is after all, you think of G1 and you do something suitable elsewhere, that hopefully the pretentious stuff can follow suit and allow one a little more flexibility. So I mean, it's perfectly reasonable, something might work, but we don't know of obstructions, but it doesn't mean they don't exist. I mean, Henrik feels he has one, so. Let's say I can do it again.


Hello everyone, welcome to the Jenkins Weekly Infrastructure Team Meeting. Today we are the 4th of April. Around the table we have Isef Damien Duportal, Marc Waite, Isef Stefan Merle, Bruno Verarten, Kevin Martin, Okul Kumar and Sonali Rajput. I am pronouncing you down correctly. It's right, it's right. It's Sonali Rajput. Rajput, okay. Sorry, I'm trying my best with my terrible French accent. I'm really sorry, Rajput. Okay, so welcome. Yes, Ymer Velmer is joining us just when I was telling his name. Hello, Erwe. Erwe is the last one. Okay, so let's start with the usual announcement. So today's weekly is not out yet. Due to the current DG cert issues, the release for today is blocked at the verify state. So the one file has been generated, tagged and pushed, as usual. It used the new GPG key that we added last week. However, it's still using the old DG cert expired certificate. That is why the verify step fails. Now we'll have different options. We will discuss later the go we have, we might have to trigger a new weekly release or not. Or we could finish the packaging depends on what we want to achieve. That's a discussion for later. And once the packaging step is done. And we have decided to use fully the new DG cert or not, then we can proceed to the usual delivery bits. Is that a nice quick summary as part of the announcement I propose that we discuss this option a bit later. Yes. So the second announcement is that we have the new DG cert certificate. And that's a good news. It should be okay for the tomorrows LTS release. So the next week's signing certificate is there. And should be okay to use with tomorrow LTS. Do you have other announcements? Okay, I don't. So I let I propose we go to the upcoming calendar. We might not have a weekly could be next week or maybe tomorrow or eventually later today, but we might or might not have a weekly soon before the LTS, which mean the next LTS release will be tomorrow. So the next week is again to have that LTS release to have both code signing with the new DG cert and repository packages signing with the new GPG key. So everyone will stop seeing expired key once you will have imported the current one. Any question. So, okay. About security adversaries, we don't have any mail on the security mailing list. So, and a no security advisory plant. Next major event as usual, we have silicone and devox, I will copy and pass from last week, unless you have a new event where you can meet Jenkins community. Okay. So, that has been a really, really packed week. Thanks. Thanks, mainly every and Stefan for the huge work you did on all of these tasks and Mark for taking care of the certificates. That's the, the greetings that are required for because that week was pretty, pretty, pretty hard. So, the task that we were able to run and to finish the Robo Butler service has been sunset. Remove from Puppet removed on the virtual machine cleaned up and its repository and image have been archived. Haven't seen any error. If we see any rock due to that. Then we have to trigger an issue. I think that the edamame machine is now empty from any service. That's an OSU OSL sponsored machine. So we should be able to add that machine in. We have different way of doing that but as an agent for CI Jenkins. Food photo. CI Jenkins is now having an automated certificate renewal using Azure DNS and let's encrypt it's only and it's everything is managed as code and it's working. So, what we did a few months ago with a certificate bound to expire the 11 of April is now renewed every three months, even though that is a private instance. So the opportunity to clean up the DNS as Stefan and Hi so there were not only we had the current DNS record for search CI that we have to migrate to the new child zone where the permission are restricted. But there were former not used or let's say legacy records that have been removed. We heard anyone from the security from the security team and no one is ringing at my door to hang me. So I assume search CI is working as expected. Trusted CI and search CI now have all the permission and Azure credential required to spawn virtual machine agent in Azure. All of this credential are now restricted only to the resource group or where it's required and everything is managed as code. So thanks Stefan for that work. That includes expiration date for this credential directly inside the code. So if we don't have the time to manage to switch to manage identity that will remove the need of having a credential at all. We can have something that check our code publicly and send it alerts before it expires. So again food for talks new ideas. It's ready to be worked on. Digital ocean we have a token that should have that should expire in a few days that has been rotated. Maybe it won't serve. But at least we have a new three months valid token for the technical infrastructure and everything is working. Someone proved that they weren't a spammer. So we created the account with no spamming issue. I assume that the spam message come from their IP that was Marcus from a public IP Marcus abusing. Because I'm not sure why us but the anti spam will be triggered for this persons since I was able to create the account on my home. So the only difference I reuse the information they gave so either they had something on their web browser that was sending spamming requests to accounts. I'm not really sure. Anyway, the account was created so I let the user. To the thing. Social link updated on GitHub organizations that's a new feature thanks Alex and team and their way that has been updated on our GitHub organization page. Miss so someone was trying to create an account. They weren't doing the correct email. I purposely avoided to give the correct email later that person gets the correct email or create a new account. I don't want to risk any account takeover on that area. In the end of the presentation, every again we were able to close the introduce artifact caching proxy for CI Jenkin say oh, so it was reopened due to issues that are tracked on another issue, because the work for the ACP is there. We finished all the tasks that were to be finished last week. Now it's not creating a new service. It's finding why the service in some edge cases is not working as expected, which is different topic. Another account issue account show log status the user was able to find and fix the issue so that's closed. We added a new repository on CI Jenkin say oh, to be built coverage model as per someone requests. So we have now a number when we need to run a scan organization on the wall, Jenkin see organization for the plugins, it takes one hour and 20 minutes. It succeed. And if it's not running and at the same time of 10 or 12 other bomb bills, then it succeed without any problem. The main challenge here is that it has to pause because we have a rate limits imposed on GitHub API. So it needs a 30 minute pause in the middle. That way it takes so much time otherwise it should be more 40 45 minutes. There might be solution but I haven't found any about all making it even using different GitHub app. I'm not sure, theoretically we should not have any API rate limit with the GitHub app for this kind of permission. But it sounds like a GitHub branches plugin inside Jenkins limitation. I assume it come from the time we were using token instead of GitHub app. But I don't have enough knowledge on that area. So it work as a general matter of fact. Let's try to run this without having a bomb bill running and letting the other that you run this because CI Jenkin say you is using a lot of CPU when doing this. That might be the reason of the rate limit not because of GitHub API but because of the CPU usage. The account issue the user account was created. We had issues with agent instability a few weeks even month ago but jumps never answered our question so I took on me to close the issue, especially the instabilities looked word you assume that it was out of memory but it wasn't based on the metric we were able to get. And now we don't have enough information, especially since we changed and shifted or the workload from EC2 to Azure and Digital Ocean and we are back so I propose to close this one and see if it come back because it wasn't reproduced after the pool request was merged. So that could have been network issue or something else. Any objection on that because I don't mind reopening and but I never had feedback from Jim so you assume it's closed. No objections from me seems like a reasonable response. Maven 3.9.1 was released and deployed. So thanks everyone involved in that step. It's generally available and we sent an email to the mailing list. Thanks a lot for helping the Jenkins security team and specifically taking that thing alone. So you were able to not only grant access to two new GenSec team member to release.ci, but also you were able to restrict or at least clean up some part of the air back model. We might need more restriction there. But thanks Daniel for raising that and there we for taking care of that step. Hervé, were we able to restrict all the groups or do we have some following tasks on this one? We have some following tasks, but we should open an MDSK issue to ask other users what restricted permission should be applied for other groups. May I ask you to take care of opening that issue and starting the discussion so we can ask Daniel for a double check. Is that okay for you? Yes. Cool. Don't hesitate to add it to this week milestone. I haven't created it yet. But yeah. Thanks. I haven't heard back from the security team so I assume it's okay. I believe you checked with at least Kevin, right? And Yaroslav as well. Is that correct Hervé? Sorry? Were you able to check with Kevin and Yaroslav if they were able to reach the CI? I've checked with them in secret code access. So private token and release CI. Cool. Thanks. So subsequent issue for our back restriction on release.ci. Okay, we were able to close the big issue as well. Congratulations Hervé. We now have a brand new private gates cluster. So private Kubernetes cluster in Azure which is hosting in FRACI, release CI and our bots that doesn't require public access. It completely manages code so now we can iterate and it's able to run fields on Linux, on big Linux machines and on Windows and using two different subnet for the agents to avoid any security issues. So great work. Just a question again, Hervé. Were we able to clean up the former cluster or do we still have that manual task to run? As indicated, we have some public services to migrate to the new public 8S cluster. Yeah, but we're thinking about the temp private one. Temporal that one has been removed. Oh, you cleaned up everything? Oh, cool. Okay, so thanks. Next, someone asked, okay, there was an account question about deleting an account. Alex was using dummy accounts that has been done. Quite tricky issue, disable pull request merge mode everywhere on CI Jenkins. That was a long due request, but that one like the trigger the full organization scanning and put CI Jenkins in outage. So the lesson learned for everyone is that before triggering an organization scanning, we must let the other know in advance. And if we accidentally trigger one, no stress, just cancel it before CI Jenkins or your become a responsive, especially when we have already more than 1000 build on the queue. There might be improvement there, of course, but that's a status for today. The good thing is that that's change is now going to decrease drastically the amount of builds for plugins because now we won't have pull request triggering a build each time the main or destination brand changes, which was the case before. Now we rely on GitHub branch check that say, oh, your branch is not up to date and it's up to the developer to either update the branch which generate a new commit or new history and trigger new build or accept that it will stay as it. Thanks, Jesse and Tim for taking care of that one that will help on the billing port. Finally, we finish the work around as a credential for CI Jenkins so you so that's worked it by Stefan and I know the that's the same as the other controller as a credential require to trigger virtual machine from CI Jenkins so you in Azure are now managed as code and have a clear expiration date in clear. So we should be able to track it next time. Okay, so that was all for the job that we did. We closed a few issue that were out of out of subjects that has been closed as not planned. And finally, we can go on the walk in progress. So first of all, we had an issue with ACP in certain case, only only when using digital ocean and the bomb builds that are two to 300 parallel steps running on each one on the different pods. When we have that we our cluster are scaled at the maximum and after 90 minutes of that maximum workload. We start having word issues. Only on digital ocean so it's not even sure that it's the ACP set that we have. Because why, why don't we have this on AWS or Azure that sustained the same kind of workload that is for AWS. So we are trying to search the differences that could be related to really low level topics. There has been multiple, multiple areas. Thanks Basil for pointing that it could be related to the amount of maximum connection, despite what the matrix says it's still important to check. We delivered earlier today a set of engine x tunings. Because we were able to reproduce the issue of a spend quite some time during the past days on this one and we spent some time earlier today. I asked you help to retrigger a build forced on digital ocean using the ACP with the new settings. Where did you had time to check the result of this one? Yes. Did it fail or did it succeed? Because of time out for the first time and I restarted it but I did not have time to check why it failed again. Okay. Okay, so we don't. Okay. So right now there is no blocker. First, because thanks to Elvis walk during the past day, I think it was Friday. When the bills are running on digital ocean, then we use G frog. So it's a bit more bandwidth, but that allows developer to run the big amount of bombings that we're having. So currently not blocked. No ACP. When on deal. And work in progress on enabling ACP deal again. So thanks for checking this one. And I think that's all there has been a discussion trigger by team about why are we using engineering instead of our own artifactory instance. So the discussion is healthy. It sounds like there are different solution. I'm not sure if it's a willingness to change the world setup or to just ask question to think I'm not really sure right now. We are going to check this one because digital son might need that's a subsequent subject might need to be stopped in a few days. So maybe we won't have to dig dive on that one. So I propose that outside the bill that they just mentioned that we are checking with the new engineering setup. I propose that we don't spend too much time here. Time out again. Time out. Oh, yeah. Okay, good. Gotta check the time out reason then. Thanks. GPG key expires on March the 13th. So that issue is only there waiting for tomorrow LTS and the end of all the issue. Because right now the weekly Debian repository and read that repository are signed with the new key, but not the LTS one. That will be okay tomorrow and that issue should be closed so it will mechanically move to next milestone. Any question. Okay, so I'm really sorry for the users that are having issues and that are frustrated by that problem. If you just have to wait tomorrow and the problem will be gone. Yeah, and I have the action item from the board meeting on Monday that be sure we convene a retrospective to figure out what we do to prevent that in the future. We may want an admin monitor months ahead and etc, etc. There are lots of things to improve my my apologies we made a bunch of mistakes on on that PGP expiry and on the code signing certificate will get better. Thanks Mark. So let me write this down incoming post more time to improve for next time. Closed after LTS release tomorrow. So digital and the credits are almost all exhausted. Because of the usual activity on the bomb is that we had last month. But that you knew you know the activity seems to be more and more than normal activity. And the easy to issues we had and the shift of traffic and workloads will we consumed eight K instead of one point five K during the past month in digital ocean to handle the big amount of bills. So we only have a few days left of credits I think we should be under the $1000 and that's my credit card on this one. So I will stop the service once we will have depleted all the credits. Hopefully every was able to communicate that to digital scenario today so we hope we should have an answer. And we should do that the same time arm. So the question is are they okay to extend the credit or grow. We'll see. Maybe that will be the end of the digital ocean platform. I don't know. If it's the case we will have to search for other sponsor and if not the case we still have to search for the sponsor to extend the increasing amount of bills from the developer. That's a good if time for the community but that's quite the nightmare for us we have to search for findings on that area. Any question on that topic or a date or things I could have forgot. Okay, so let's wait for answer from the oh for adding credits again. We are really thankful to digital ocean for helping them for helping us because yeah we we really really had the opportunity to shift our workloads directly to digital ocean with almost no pain that was quite easy. The only issue and it's not even sure it's digital ocean related is the ACP. But yes, the API and doc was quite nice. So say again thanks for what you did for the Jenkins projects, I hope we will be able to continue. We have a user saying that the password reset email is not coming. So we tried multiple time they don't receive the problem that the Jenkins in for discovery is that. Accounts Jenkins I use send email through send grid SMTP. We have that mean we have a working account and send great I've asked Kosuke for accessing that send green account but it gave me access to a mail girl mail gun account so I might have missed something. So from now on we have to solution wait for Kosuke to tell us if he has access to send grid and if he can change that account. I don't even know if we pay for that so mark I might need your help on that area. It's in send grid is definitely in the list of infrastructure that we use. I'll have to we'll have to read that page to be sure. Okay. The main goal is can you ping Kosuke just to be sure that you see two different person asking him to check send grid because I might have missed something but yeah I need help to have an answer. I'll ping Kosuke yes. I will communicate to the user but right now we cannot analyze why the user is not receiving email. My proposal is that we check we wait until the end of week and if we don't add any news from Kosuke or anyone else that could help us get access to send grid. I propose that we shift account up to mail guns since we have access. So at least we should be able to monitor the state of the emails that could have been refused by the remote user map servers. Does that make sense. Wait until end of week. If no sun grid access, then shift to mail gun. So we can. So we can diagnose such. So I'm going to tell that to the end user that we don't we have an issue accessing the emails SMTP so we cannot check. Either they can use another email address or wait for us to get access or change the. We have work in progress so on using Azure IRM 64 virtual machine so as a reminder Azure announced in December 2022 that virtual machine with IRM CPU was GA so now we can use them. That's a good news because they found last week last month three shifted all the easy to virtual machine workloads on our private controller to Azure VM controller to decrease the AWS bill and to have less bandwidth and to have a centralized management inside Azure. But we still are using IRM 64 for building images so we need virtual machines linear virtual machine with IRM 64 right now we keep using easy to for that. That issue is about doing the same for Azure. Do you have a status report on that one, Stefan. Yes this morning I was able to switch from open to 2222 because thanks to the work of everything. So it seems to to work better I am waiting for another request to add that 22 channel to the gallery in Azure to be able to spawn those VM. So it seems good. Azure. There are some for gallery management. Okay. Exactly. Cool thanks. So we'll have to check this one. Is there any question on that area. I don't. Same thing. A question that we had privately. It was just an idea and we are going to open that more and more. We were thinking about being able to use this air and 64 machines on both AWS and Azure for the workloads that should then that shouldn't be concerned by the kind of CPU. We thought, I think you mentioned, you asked about the bomb. The question is, do we is the bomb, the whole PCT steps these 300 steps that cost us lots. Are there some that could run on IRM 64 or even all because the costs is 20, between 15 to 25% depending on the cloud and the instances. So it's not the only thing we have to do but at least gaining 1015% of the cloud cost for this bill could be interesting. I'm not aware of anything in the, the bomb or PCT build in the bomb builds right now that is dependent on Intel architecture on AMD 64. I would expect it at least the plugins I maintain run happily on ARM 64, just as well as they run on on AMD 64. So that could be interesting for this one and eventually for for the plugin CI, but not all plugin. So that could be also something proposed to the developers saying by default we might want to shift to IRM by default unless you need a specific Intel binding in that case you should. You should we should provide something on a pipeline library for build plugin. Well, and we certainly could already provide an argument in pipeline library that says that allows you to opt in to arm 64 or opt into platform independent maybe arm 64 is the wrong choice but saying hey as far as I know this plugin is platform independent pick anyone and it would allow us then on occasion to run on system 390 if we wanted. Of course building Jenkins itself and running the ATH won't work on IRM 64. In the case of the ATH not only because acceptance test you want to run the real life, we could add specific acceptance test only for IRM. But you have to know that most of the Docker images that are used in the acceptance test harness are using Intel image and the required work for ATH will be clearly too high for now. So that's why ATH is out of question. And in the case of Jenkins the question can be raised, but I think the Jenkins score itself might be a bit complicated. That's also something for us inside in for to think about all of our workloads should be able to run on IRM to decrease our build Kubernetes management Terraform project. Most of the tools that we're using are statically compiled using Golang so IRM is one of the targets and we can use binaries for this platform. Is that okay or do you have question objection things that make you think. So in terms of net cost savings there are other things that are higher priority in terms of cost savings right. There are other things that we think will could save us more more more cash but I like I like platform portability and the cost savings is nice. I see that one as a not top priority but still important because we have to do it step by step. And as you say if you start with opt-in we can do a little bit of this task every weeks until we have something working. The task itself on the notes is mandatory because we need to be able to opt out of AWS at any moment so we need the same feature parity between both clouds. Next task is the same kind. It's a long running task we have to upgrade all of our Ubuntu instances out of 18 and ideally out of 20. So that is to update everything we can in to Ubuntu 22. So thanks survey. You did the first release of the Packer image using that new Ubuntu version. We are currently, we are going to roll out first that version on our private controller verify that we can run Docker and our tools without any issue. Then we should be able to roll out to CI Jenkins. So I propose if it's okay for you that we deploy as soon as we can to infra CI. And we we my proposal arbitrarily if it's okay for you is to roll out to CI Jenkins with an announcement to developer Thursday. So tomorrow Wednesday will be LTS and Thursday will be Ubuntu 22 for most of the agents. Does it make sense for you? Is it okay for you? Cool. And the next step obviously that that will be from the team so it can be your way but it can be someone else. It will be planning to migrate all of the Linux node pool that we use on all of our Kubernetes cluster so the underlying machines are switching to Ubuntu 22. We can control that for sure on Azure and AWS. No, I'm not sure AWS I think we're using Amazon Linux so that we shouldn't be concerned, but at least on Azure. And I don't know for digital as well. That's okay for you folks. H2. So we have Packer image to roll out to CI.G Thursday. Testing in infra CI. We have Kubernetes node pools. At least Azure. Maybe do. And the next step will be checking our Docker images. Checking Jenkins. Docker. I think we have at least the open VPN that is still Ubuntu 19 so that should be increased. And we got the VMs. Yeah, but afterwards. Yeah. Two next issues unless you have another point on Ubuntu 22. Okay, so document code signing certificate renewal process and renew the signer certificates. So this one are quite the. Are quite the eats right now I propose that we will give a summary of what what we did today Mark Stefan RV. And what are the upcoming next step that we could go for the release is that okay if we discuss that now. Yeah, sounds good to me. So document. I've shared with we have chaired a four of us in a private channel because the sensitivity of the data that we exchange we have chair. We were able to upload the new digital search after fighting with transforming it from different open SSL certificate format but it sounds like Azure accepted the certificate and was able to pour seat and determine all the meta information which is a good direction. So we should be able to update soon the documentation once it will be finished there were just a few missing elements compared to what we have it wasn't enough but it was good enough for us to get started. So now we need to be sure that the data we have added the new digital search inside the vault is able to be used on two elements. First, to be able to generate a sign MSI installer for Jenkins. You want inserted in Azure. Need to check MSI generation. And the jar generation by Maven tricky parties and for could we could we change that were generation to signing, because really the thing that we generate the charges we generate the war file just fine. It's it's that we want to be sure we're signed. Yep, that's a signing the correct row. Thanks Mark for using this one. Yeah, otherwise it's confusing. Today, the weekly has been generated and signed with the former certificate the verify steps which is the last step of the release pipeline failed. Now it's time to decide what we do with the pack. Do we want to package to start packaging this version. Let me write this down. We have today's weekly is released with old cert. The packaging is currently configured to not generate MSI signed or not node MSI at all for today's weekly is configured for no MSI at all. The goal was to avoid a stressful message with that MSI because it would have been signed by the old certificate and every user downloading it would have been a red war red big red error message on Windows which is pretty scary. We don't want that. That's why we say better not having MSI at all. It's a weekly release user can wait or take last week release or LTS. Now, do we want to trigger the package build as it or do we want to rollback Mark change that disabled MSI generation and add a second change that will use the new DG cert and start and checking that we can produce MSI with the new certificates, meaning we can validate that the action we took in the past hours are okay with the new certificate at least. So the jar will be signed with the older one not verifiable but MSI should be okay. So alternative to be sure that I captured the alternative so alternative one is continue the packaging as it packaging as is no MSI right alternative to is and after that then then revert well no that that's okay alternative to continue or revert revert or let's say this way add MSI to packaging continue the packaging. Add MSI to packaging and new certificate. Right and new cert signing. Exactly. And new weekly this one. No, no, we could we could continue the existing because it hasn't run the packaging step right and new signing. So this is current weekly includes MSI and signed jar right or signed war file. Exactly. The danger is that if, if we, there may be some surprise in there where we say oh whoops, something failed, and alternative one might have detected that failure. But for me alternative to as a much more attractive choice alternative three would be a full new weekly with the new shirt. Right right I think so alternative three new weekly build using the new certificate certificate right with the risk of failure because we missed something somewhere. Right and and then we're stuck in the mode of guessing of trying to have diagnosing what failed and having diagnosed what fails then we've got to or how do we fix it. So we may choose between alternative one and two and add to the three one. Correct alternative three is possible, even if we choose one or two. That's correct. And really we could do one. And then we could even attempt to right but the problem with attempting to after one is no no I take it back we can't. We choose either one or two and then three. We cannot do we cannot do both one and two because as soon as we've performed the packaging step if it's successful, it will not repackage again that same release. I would choose two and then three. My personal of the way. Yeah, the same there, because the two include one. The same there my goal on proposing the alternative to right now is to have an intermediate step to validate as we say it's the certificate state is it correctly encoded usable the new certificates. Well, and for me, alternative to is more attractive because it's also much faster. Because when we choose alternative three we have at least a two hour window before we get any feedback whereas the packaging step will complete in 30 minutes 30 to 45 minutes. And if there's something wrong with alternative to we can stop think find and then do three. Right. Exactly. Anyway, there is a risk that even with alternative to walking as expected. The way packaging consume the DG cert certificate is different than all Maven consumers. So we could still have a bad surprise when Maven try to read the file, but I assume that. Yeah, at least we have a smoke test with alternative to if it smoke it's bad. And then we will Maven will fail immediately before generating any data, as far as I can understand. I remember we can we can start it again. Exactly. The only thing is that we are time constrained because we must finish everything before the LTS tomorrow. That's a request from me as Jenkins infrastructure officer because I don't want the LTS bill to discover bad surprise in last minute. That would be great to finish between before midnight. There isn't a fixed timeline tomorrow for the LTS. So as soon as we take decision we can say don't start the LTS process because it's a human that will trigger it. So we can ask that human to say hey wait for the weekly to be finished, even if it delayed a few hours. Is that correct mark that is correct LTS is triggered by the release lead. So in this case Chris Stern, and he'll trigger. All we have to do is ask him hold please don't trigger until we give the go. Perfect. Okay for everyone. Should we ask the team you come the release officer for a last stump on this one for last. Is it easy okay. Is it okay if I ask him directly on the. I think that makes yeah, we certainly want him informed about what we're doing and, and yeah, asking asking for a pause on the launch of the of the of tomorrow's LTS. There's a good one to ask and copy Chris Stern on it. Okay, and probably should copy Alex Brandis as well just because Alex has been a frequent release lead. And what are the, the, let's say the annoying part of triggering a new weekly in an unexpected way like this. What will be the additional work that it will generate for everyone here. Documentation as I understand maybe more changelog things. Sorry, ask, ask your question again Damien. What are the consequences that I don't foresee of triggering a second weekly, which is, yeah, we would have to we would have to create a new changelog but the changelog is is pretty easy to do and that that's a very low risk item to do it. Yes, people will wonder why the changelog is so brief, and we'll put a banner on it that says this changelog is so brief because we were verifying the MSI. Great. Okay. Okay. Cool. The alternative to is not is not creating generating a new weekly version. It's, it's packaging the one that we got with the same version. So the only one is the alternative three, which will trigger a new version but this one will be packaged correctly with the, everything with the new search so it, it's not only to check it's, it's the real good one with the new set not expired. So, so if alternative to is successful, it will result in a signed jar file or signed war file and assigned MSI if alternative to is, or no, I take it back. That's that's wrong alternative to only validates MSI signing not war or file you're correct. The so so one of the changes in alternative threes changelog as you said Stefan would be the war file is again signed. Correct. Correct. Yeah, it's signed with. I'm not sure that the signing even applied the that expired certificate so I suspect what we will see on the current war file is that it looks unsigned. Okay. Yep. So for that. I'm taking on will ask validation to team before proceeding and we'll let now Chris turn as the LTS lead. We will just before proceeding that the three, the two we don't really need it. Or you or we need because the two is just packaging so they don't really care. I think. Yeah. Yeah, absolutely. Before proceeding for the three. So as you said mark we will have so the board ask for a post mortem for both GPG and code signing, which I believe will be must do it earlier. Yes. Well, I mean there, there are a number of other things hiding in it but the most obvious is do it earlier. Yes, absolutely. So a set of different user complaint on specific details like changing the key name or not that okay we can discuss that that's healthy to raise them. But yeah, I see this proposal for user just a frustration from someone not being able to finish their day to day job which I can understand. Over. Yeah. There are consequences of doing as of not doing of not saying no to end user sometime. But okay. Do we have other elements on that area about the signing certain everything. Okay. Any question any objection any things and clear that you want to bring on these topics. Nope. Now, next issue realign repo Jenkins CI or mission it's easy nothing was done. I started the cluster and then we had a lot of other tests so I didn't have time anyone interested in helping finding an idea available at that that we can run on virtual machines or Kubernetes is absolutely welcome to help here. I still have that m chart to test locally. Finally an issue I saw just before the meeting you were able to send a pull request survey on that topic about to allow us to close that part it's about the disk space of the container agent use for the bomb builds that triggered issue we don't have issue today but it's an improvement, especially for performances. Can you remind me the status of this one I've absolutely forgot. So I have opened a pull request to move to the TMP and the home Jenkins folder as empty their volume as they are currently moved as overlay. Okay. Are not overly that's a good thing. Okay. Will you be able to sync up on this one after or tomorrow maybe I'm wondering about the testing process. I believe we should be able to test it. Not only. So yeah, is it okay or do you want to discuss it there. Okay, just to let you know that we might have to select an annoying path a slow path, annoying because slow to be sure that we don't break all the agent at the same time. I will give you elements because I feel like I wasn't clear and I didn't add and neither took the time to explain the testing path there. So I wasn't expecting the test path to be there. So the goal is to discuss and learn all together on how could we test this kind of element in production without breaking everything. I just see your message about digital as an answer. Survey. What did they say give us the primer. See approved extension and the renewal of the sponsorship. And Oliver will help us in a day or two. Great. Okay, so does that mean they are going they're extending again or just putting what you requested. I've just the surprise I will see we forever in a day or two. Okay, so at least that means we won't have to delete digital ocean. We didn't renew the token for nothing. And that means we will have to continue our analysis with the bomb. Excellent. The message from them looks like they're willing to continue funding us at least at the, at the level that we were funded before other than the one extraordinary month. I don't have any idea about. Okay, I just read you almost a response and I don't have any idea of how much. Well, thanks to digital ocean they and thank you for asking them that's great. If it's okay, I propose that we decrease a bit just as a safety measure for my credit card. If it's okay for you. Yeah, that's why I will if it's okay for everyone I will open a poll request that will decrease the maximum amount of builds back to the normal so we won't have any possibility but it will still be used is that okay for everyone. Yes. For you for your dream in nice. Yes. Fair. A few new issues if it's okay for you. So we received an alert about the update center as certificate that need to be renewed. That was done one year ago. We have two months but I propose that we do it in advance. So if it's okay. Thanks for Stefan for changing the alert the calendar alert to an issue. I'm going to add this one to the next milestone. We'll have to work on that. I started already. I sent you some stuff. Cool. I'm looking at the new issues. So that one will be on the new. Let me add it in the notes. New issues. I will format the notes afterwards. So we have this one. Digitalism credit depleted that one is already on. This one my great trusted CIG and can say you from AWS to Azure. That's the next big thing for you, Stefan. We have free virtual machines that allows to to see I trusted CI to run currently on EC2. We want to move that workloads to Azure so we can have a centralized management and less bandwidth costs. We might need to only move to virtual machines depending on the let's say network security solution we will select. The third one called bounds is an SSH passion. We can have the same pattern. We can have an Azure bounds passion because they provide cloud resource we could use VPN. That's something to be discussed, but at least starting the Azure Terraform resources for two new virtual machine with the associated data disk and install them with Ubuntu 22. Then that could be a great start inside a private network of their own. So is that okay for you to take this one Stefan. Yes. The sensitive part will be the permanent agents. Once you will be able to manage with Puppet these two machines with Terraform and Puppet, they are empty inside the private network as a first step. We will have to think carefully with Daniel and Daniel is here the bus factor about the way update center generation works because the cache is using hard links. We'll have to her sync and migrate this but using hard links, not sim links and hard links are really sensitive things. If you delete the hard link, it deletes the point in the associated file. That's the most tricky part. No risk for us because we will only run her sync from a machine to the other, but still be careful. I was about to say, oh, we have been fighting at yours, but no, I keep it. So Daniel will be our source of knowledge for that part. We already asked him elements. There will be plenty of way to test it without any risk, but it's just okay. That's a chair knowledge sharing. The goal is to control cost on AWS. And upgrade to Ubuntu 22 for that machine. That part will be the easiest part because we only run Docker images or GDK builds. Any question objection on this one? There is a third one that I've added that we need to start. It's not top priority, but it's really important. We are spending more than 1000 bucks per month just of outbound bandwidth of CI Jenkins IO controller. There are two source. We have identified two potential sources of outbound bandwidth. The first one will be for people browsing CI Jenkins IO. At first sight, I'd say just a few web pages, but as some people there that already mentioned, yeah, downloading the logs of a bomb builds. And I'm sure that both Mark and Derwey can confirm that. Clicking on console output, full console output takes some times because it's 30 to 40 megabytes log output for bomb builds. So that one could be a source. It looks like we have Apache logs parsed on Datadog. So we need to go on Datadog and run whatever magical request or put it on SQLite and run whatever. I'm not neither a Python or SQL person. So I will use happily Datadog interface or shell. The other source that could be one of them also quite big. It's the stash and stash. Let's say for a bomb builds, we are stashing what is called a megaware. It's a war file with the set of plugins that are being tested as part of one of the 300 builds. So it's generated one time, stashed one time during the preparation phase, and then it's unstashed for each of these two or 300 steps. So if you generate it on Digital Ocean, you stash it sends to Azure. So stash is only inbound. It's okay. But then for every AWS or Digital Ocean bomb PCT builds, you have to unstash it two or 300 times. That is outbound bandwidth from the controller. And that one is a lot of data. So one of the solution is using the what we call external artifact manager. These are alternative implementations that instead of zipping the file on the agent, sending them to the controller, which are the stash or archive artifact, they do the same. And then unstashed is sending it back to agent through the same inverted protocol. Instead it say, oh, let's use that S3 bucket. And so the agent copy the data on the S3 buckets and the other agents will be able to reuse from the same S3 buckets. The part I don't know here is can we do that on S3 bucket inside Digital Ocean and another inside AWS. I don't know that's the part that should be studied. But that is that will avoid a bunch of stash and stash directly to see a genkid say itself less pressure on this one as well. Yeah, or could we stash to the artifact caching proxy. Right, as as another angle rate my mind. It's, we've got an HTTP server sitting there. Right. But the problem then is permissions to write to that thing and that may not be what we want. Yes, that could be a code not be used. I don't know that will require. I don't think it has been built for that use case. Right. But yeah, that's that's an option. Can I ask you to add you to comment on the issue. Yes. So that one that budget will allow us to ask the trusted CI virtual machines without any overhead cost if we're able to decrease the bill here. Any question on this one. Any objection to edit to the next big cost for us. Not important issues but just wanted to mention them I'm going to move them at launchable to agents basically is working on launchable. And require a tooling the proposal and he created that issue just to keep track it's not priority now it's low priority but we can do it if we have time. So install launchable command line tools on style inside our information inside our machines and templates. So that won't be required to install each time launchable is called so it's optimization. That one we had thanks survey for adding create script to lock and lock as your resource group. I was almost deleted see I didn't can say you in production last week due to a human mistake on terraform. And as you provide a way to lock some critical elements so the, let's say the direction we are all going the consensus is having an external script, where we identify the sensitive element and we lock them. So, any change on terraform once it's code manage or even today when it's manually manage. We are all deleting and recreating the resource because whatever change we want to do accidentally will be blocked and forbidden, even if the user has the permission. So, I think it's a bit too early to spend time on that we already have enough, but yeah, I, let's keep that in mind that should be done. So, let's go back as the backups, the more operation we are doing the more critical it start to be so better planning for it instead of suffering for not having had the time for this one. Then we have an issue. That one is also for later. So no priority it's removing credential and using workload identity management, we can do it for search CI and CI dot g today. But yeah, that's, let's say bonus. The rest is for later. I don't have any more issues or recent changes or things to speak about. Do you have other topics or is it okay for you folks. You, you don't want to speak about the discussion you have to get budget for from CDF I think for Amazon. I think, I think you may be referring to a request to adjust the settings in the Azure account to allow us greater flexibility. As far as I can tell that's that's resolved at least resolved in the sense that the question has been asked and clarified with Fatih did you did you mention of CDF. Yep. You're right. Oh, correct. Thank you. Thank you. Yes, good point. We received an answer from Docker open source, despite us closing the issue because they, they went back and they want sunset, the free team. They confirmed that Jenkins in front Jenkins forever our part of the open source program. They were, they weren't just added with the label on the advertising publicly thanks for the reminder good point. I confirm that that's what we want. Because there is no need and I will say it's even safer for us for not advertising this publicly. Not because it's secret but because these two organization are using images for our own usage in the case of Jenkins. So I don't see a point. The goal is not to publish that for external usage, because the effort for helping and supporting people on our use cases is worth nothing. That's up to the debate. We can always ask them to have the badge. So as I survey checked, we have more than 50 million pools on some of our images on the info. So that means these images are popular. That's, that's bothering me.  Out of the open, I mean, we could run some Bitcoin miner and gain some money for the sponsorship, right? Sorry, I don't believe it's coming from your mouth. It's sponsoring problem now. Oh, good. The richest open source program in the world, right? So juxtasites, no, that's, that's bad for them. But my proposal and again, you can disagree, but think about it. I don't want us to be advertised as, Hey, let's reuse your tools. We don't even have time for supporting our own use cases. So supporting our use case on someone else's machine is anyway, that could be a way to get some contribution though. So mixed feeling about that. But right now, yeah, proposal is that we keep it things as it's and we think about deleting this organization in the future from the Docker hub and using something else for Jenkins forever. Please don't advertise this element. This one, this one must not be advertised. I can accept for Jenkins CIA and fraud, but if you do Jenkins forever advertising that, yeah, yeah, let's do it. You will have the Jenkins security team that will come for you. They will find your address. They will find you. And we will never heard about you again. Now Jenkins forever is only untrusted workload. Please don't, don't advertise it's only for testing on short time window. The content here cannot be unsure that it's safe to run. We can say for Jenkins CIA and fraud because we use it, but not Jenkins forever. Other topics. Okay, so before I close, I just want to thank everyone there for the huge work you have done during the past two weeks, all of you. Honestly, that has been really, really two challenging weeks with a lot and a lot of things. And I'm really happy because alone two years ago, or Olivier was suffering and he did that for years. And yeah, I know the amount of time and the difficulties we all have, we all have a lot of work. We don't all work all together the same way we all have different ways of working. But that's I'm really proud of the team and the amount of tasks and things we do. And personally, that's a relief because I know I've known that when I'm not there I can be back and I know that all my tasks are reviewed by you folks. And you have the ability to say no to challenge what I said and it's really, really a safe place for me. So really thanks team because that makes the project really healthy and really, really efficient. So ability thanks for all the work you do. And on these good words. See you next week for everyone watching this and for the other. See you later tomorrow.