Thank you very much for this opportunity to speak in your web seminar. So yeah, there's a lot here, a lot of topics, and I put together a little graph to show the connection between these various topics, both, how to say, existing connections and some new ideas, some new connections. So the main object was to study gaps between primes, in particular the large gaps, and there are connections with something called the interval sieve, probabilistic models, which are somewhat familiar to a lot of people, the Hardy-Littlewood K-tuples conjecture, and the existence of z equals zero. Now I've drawn a bunch of arrows here showing the connections, and for some of them, I've written names of people who have done the work to make these connections. And then there's a brand new one, which I threw in just recently because Andrew Granville shared with me a pre-print connecting z equals zeros with the interval sieve, which we'll come up later in the talk. So everything written with the word new is something new, which I'm going to talk about in this lecture. So we'll start with what's known about large gaps between consecutive primes. There's an upper bound, which is a little bit worse than square root of x due to Baker-Harman and Pence. You can improve this a little bit, assuming the Riemann hypothesis, this is due to Cromere, but that's the best it's known, even conditionally. I think you can do something with the log if you assume RH plus something about the pair correlation conjectures. So there's work of Goldston, Heath Brown, and others on that. But again, this is about the best we know how to do, even conditionally. Well, the best lower bound is a little bit bigger than log x. In recent work I have with Green Conyagin, Conyagin tau. So this fraction does go to infinity, but very slowly. So what is the truth? Well, the first person to make a guess at the truth was Cromere. And I'll talk about how he arranged this conjecture based on a probabilistic model of prime. So he thought that G of x should grow like log squared x, at least in the Lin soup sense, Granville modified Cromere's conjecture, overcoming some flaws that it has, which I'll talk about later. And same order, but his prediction was a slightly larger constant. So how does that compare with computations? The computations show that at least up to 10 to the 18th, the ratio here is, well, it doesn't quite reach one and it certainly doesn't get above one. And you can see these more clearly on this graph. So the horizontal axis is a log squared scale, which turns these predictions into straight lines. So the black step function is actual data of G of x. The green line is the asymptotic for Cromere and the red line is the asymptotic lower bound that Granville predicted. So you see there's, it's not quite reaching these. It looks like at least it's growing at a rate of log squared x, but the constant is not really evident from the calculations. All right, so the first thing I wanna talk about in detail is this, the connection between gaps and Ziegels zeros, which is something I came up with last year. So the first idea is the Jakob Stahl's function. And this is, this was first done by Vestentius and then improved by Erdesh and Rankin in the, all in the 1930s. So if I let U sub T be the set of integers that have no prime factor less than or equal to T. So this is performing the, say the Sib of Eratosthenes up to the prime T. Then if this set has a large gap, say a gap of size Y, then that implies a gap of size Y in the primes. Well, at least, so if I let Q be the product of the primes up to T, which is about E to the T by the prime number theorem. So between capital Q and two capital Q, there must be a gap of length Y in the set U T because the set U T is periodic mon Q. And since that means that there's a set of Y consecutive integers, all of which have a prime factor less than or equal to T. And since the number itself is much larger than T, the numbers have to be composite. So it's a very simple idea. And in fact, it's the beginning point of every paper that gives a lower bound on prime gaps. Now the second idea I wanna talk about, which the relation is not completely clear yet, has to do with the effect of Ziegels zeros on primes and progressions. So a Ziegels zero is basically a zero of a Dirichlet L function with a real character, which happens to be extremely close to one. So I'm not gonna say exactly how close, but the gist of it is if we know that this zero exists then primes in progressions to where the residue class is a quadratic residue or where the character is one. So in particular, the number one will satisfy that, are going to be very deficient in primes. So depending on the size of Delta as a function of Q, the right hand side here will be much less than what you would expect, which is X over phi of Q log X. And X, so a theorem of Gallagher says that you get something like this as long as X is at least a fixed power of Q. So we know by Ziegels theorem that Delta cannot be extremely small. It must go to zero no faster than any fixed power of Q. Unfortunately, the constant implied here is ineffective. So that's one of the biggest unsolved problems in number theory is to make this effective or improve it or show that Delta has a very specific lower bound. Anyway, so the theorem, if we assume that this zero exists is a lower bound for G of X for a particular X. So you take X to be, so the exponential of a power of Q, and then you'll get something like this. Now, without the one over Delta, this is worse than what we know, but of course now depending on the size of Delta, this could be better than what's known unconditionally. So let me give you some examples. That's the theorem I was talking about that gives a lower bound on G of X infinitely often if there is the Ziegels zero. So for each Delta, for each Ziegels zero applied to the modulus Q, there's a particular X for which then G of X is large. Here are some examples. So if Delta is smaller than a negative power of log Q, we get a lower bound infinitely often of log, double log to the power K minus one. And already if K is two or larger, this is better than what is known unconditionally. So if you write it, the unconditional bound in this way. And if we have a really bad zero, so of this type where epsilon Q goes to zero, let's say extremely slowly, then we'd have a lower bound of log X to the one plus Delta where Delta goes to zero extremely slowly. So again, way better than what we can do unconditionally. All right, and the, so go to page seven. So again, there's the theorem. Not really going to go over the proof in detail, but the idea is that it's a very short proof. Once you have the tool that the primes in the progression one mod Q are very small, then what you're doing is via the Jakobsfall idea, you take a residue classes AP satisfying this congruence QAP plus one congruence to zero mod P. And so what happens is if you have a number between zero and Y, which is not covered by these residue classes, then QN plus one has to be prime. So, and then how many primes are there up to X? Well, if the zero is quite strong, this is small. And then there's a very trivial argument for covering the rest of these numbers by the primes in here. So one prime for each of the remaining uncovered. So then what we get is that we have a set of residue classes for the primes up to T that cover the interval zero Y. And for an appropriate T and Y, this will give a lower bound on the prime gap function. Okay, so I don't want to, if you don't understand the whole slide, that's fine. I will be posting these together with the talk later. The idea is it's very short, it's very short proof. Okay, so now I want to talk about random models, Kramer's model in particular. So let's go to page eight. So Kramer's random model was very simple. So knowing that the prime number theorem says that the density of primes near the number N is like one over log N. So we choose a random set by picking N to be in the set with probability one over log N. And that way our set at least globally will match the known distribution of primes. So what Kramer then proved was that the limb soup of the gaps in this random set have this property that they are of log squared size. And that's the biggest that they get. And he wrote that for the ordinary sequence of prime members PN, some similar relation may hold. Okay, so we interpret that as a conjecture. The proof of his relation here is very simple because these numbers are chosen independently. Then the probability that you have K consecutive numbers not in the set, in other words, they are random composites. So this is about one minus one over log N to the K which is approximately E to the minus K over log N. Now you plug in K being a constant times log squared N, this will be N to the minus that constant. So if the constant is one plus epsilon, this is at most N to the minus one minus epsilon. And summing over N you get a convergent sum. If on the other hand you have one minus epsilon log squared N, this is at most, this is greater than greater than N to the minus one plus epsilon, the sum of these diverges. And now you finish up with the Borel-Kenteli lemma which is, let's say the big machine that makes this work. So that's essentially how the Cremère model works. Okay, let's go to page nine. So the Cremère model has good points, bad points and ugly points. The good point, as I mentioned before, it was designed to match the global distribution primes. And in fact, it's a very easy first and second moment argument to show that the number of elements of the random set up to X satisfies the analog of the Riemann hypothesis. Okay, so that's something we expect to be true for primes. And that's good. However, if you look at some local statistics, say the count of twins that differ by two in our random set, it gives you an asymptotic of the shape. However, for primes, Hardy and Littlewood predicted similar asymptotic, but with a different constant. Which is a bit larger. Okay, so Cremère is giving the right order, but the wrong constant. On the other hand, if you look at gaps of size one, well, the Cremère model doesn't really care what the gap is. It's gonna give you the same asymptotic. So this is quite different than the behavior for primes.  Right, page 10. So in fact, Cremère's model gives that for any finite set of numbers, the number of tuples that appear in the Cremère random set is asymptotic to X divided by log X to the size of the set H, okay? It doesn't see how to say the arithmetic information that is present in the Hardy-Littlewood conjectures. So by contrast, Hardy-Littlewood conjectured the following formula for tuples of primes, where this German S H is the so-called singular series, which is usually nowadays written as a product, but originally it was written as an inference series. So one thing you can notice is that there is some arithmetic factor. So the size of H mod P, that is number of residue classes occupied by H mod P plays a role in this formula. So let's look at some examples. So if H is a singleton zero, then H mod P is one for every prime and these two factors cancel and you just have identically one for the singular series. And then Hardy-Littlewood's theorem or conjecture reduces to the prime number theorem, which is about the only case where we actually know that this is a theorem, literally only non-trivial case. Okay, so H is zero comma two, so that would be twin primes. You compute the singular series because this fraction here is one over two when P is two and it's two over P otherwise. When you have zero one, of course the singular series is zero. Why? Because if you look at the prime two, the set occupies both residue classes mod two, so there's a zero factor in the product. Yeah, and of course then the Hardy-Littlewood conjecture is in fact trivial in that case. All right, let's move on to page nine. That was just a review of the Hardy-Littlewood conjectures. Now I wanna talk about Granville's modification of Kramer's model and this was designed to rectify the problems that Kramer's model has locally with tuples, with twins and triples and so on. So his idea was pretty simple. You do a pre-siving on your set. So T is a little bit smaller than log X and then we choose our random, our numbers to be in the random set. Well, if N is not in UT, we don't choose them at all. They're never going to be chosen. And if N is in UT, which means all of its prime factors are greater than or equal to T, then we choose it with a factor divided by log N where the factor is designed exactly to capture the density of real primes, right? So we know that the real primes live in U sub T, most of them with very small exception, right? Or at least the primes in X2X will all live in this set. Okay, so by doing this pre-siving, we're getting a little bit closer to actual primes. So we're putting a little bit of real arithmetic information into the random choice. Okay, and then it's a relatively easy calculation to show that in fact, Granville's set satisfies the analog of the Hardy-Littlewood conjectures for any fixed set H. Okay, so Granville's model then rectifies these problems with Kramer's model, which are local problems. And since gaps between primes is a local phenomenon, we would really want any random model to have local statistics which are what we expect. So he proved in his paper in 1995 that the gaps in this model are infinitely often a little bit larger than what was predicted for the Kramer model. And the idea is pretty simple. So if Y is a constant log squared X, which is about T squared, so remember, T is given here in this above, then if I look at the elements of U sub T that lie in the interval zero Y, well, that's essentially doing the sieve of aritostinies. You're sieving up to about the square root of Y. So what's left is essentially just the primes plus a little bit of junk, which is negligible. On the other hand, if I took a random choice for A and I just said, what's the typical size of U T intersected with an interval of length Y? Well, because the density is theta, that should be about theta Y elements in that set. And that by Mertens is a constant times Y over log Y where the constant is bigger than one. So what this means is that there is a special starting point, there's a special interval of length Y, which is somewhat deficient in elements of U T and that's then used as a starting point for Granville's argument. Let's move down to perform a kind of probabilistic reinterpretation of this model, so of this conjecture. And I do that by looking at the two factors in the singular series separately. And the first thing we do is we truncate the singular series at a prime Z. Okay, since the singular series converges rather rapidly, if Z is large enough, this is a very good approximation. And then we look at these two pieces. Well, the second piece in blue, you just apply Mertens to that. The first piece, I'm going to interpret as a certain probability. So it's the probability that the set H lies in S sub Z where S sub Z is gonna be a random set. And it's actually similar to what was used in Granville's model, except we're doing things randomly now. So we take a random residue class for each prime and then with all of these fixed for various values of Z, we'll let S sub Z be the integers that don't lie in any of these residue classes, at least for the primes up to Z. Okay, so you can think of this as a random set. So we're not sieving out the zero residue class as we would in a normal sieve, but we're sieving out by a random residue class module H prime, okay? And then it's actually very easy to see that this red product here is equal to the probability that H lies in S Z because H mod P is exactly, or H mod P divided by P is exactly the probability that H intersects one of the, that H intersects AP mod P, okay? And since we want the non-intersection, that's the one minus for that probability. And then there's a trick that goes back to a suggestion that Paulia made in the 1950s. And that is to take Z, well, if you're sieving by primes, usually people think of taking Z to be squared X, but that gives a wrong prediction as well-known and people work in sieve theory. But if you take it to be X to the one over E to the gamma, then what happens to this product in blue? The product in blue becomes just log X to the power H. And then notice that then that will cancel the denominator here. And then what we'll get is this interesting probabilistic reinterpretation of the Hardy-Littlewood conjectures themselves. So the left-hand side is like the probability that a random N up to X, that all these numbers are prime. On the right side, we have now this purely probabilistic expression, which, and there's no singular series anymore. The singular series is gone, which is going to be really important for us later. Okay, go on to page 13. So here I'm taking a pause to simply compare the evolution of the attitudes toward probability theory, at least using probability theory in serious mathematics. So this is a quote from Hardy and Littlewood from their actual, the actual paper where they introduced their K-tuples conjecture. So they wrote, probability is not a notion of pure mathematics, but of philosophy or physics. Well, decades later, the theory of probability was put on much firmer ground. And in the paper that Polja introduced this X to the one over E to the gamma, he wrote essentially defending the choice that for the main reason that it works. So it works, therefore we're going to use it, and then we'll try to find an explanation for it later. And then he makes some, how to say, a colorful comparison here with physics and buying sugar or salt in the grocery store. All right, so now let's move to page 14. And this is now the new random model that I introduced together with Bill Banks and Terry Tao. So we'll first define the Z of X a little bit more precisely by just taking the largest Z for which this Merton's product is at least one of the log X. So then you do have the asymptotic that we want here. So our random set has a very simple definition. It's a set of integers such that N is not in the random sifted set, S, Z of N. Okay, so for each number that we're choosing, we're sieving up to essentially N to the power one over E to the gamma. So let's just, as a sanity check, look at the probability that a given number is in our set. Well, it's given by this expression, which is given by this expression, but which is very close to one over log N because that's the way we designed it. Okay, so that's good. It matches the primes. It matches Kramer, matches Granville. The only problem is that if I take two different numbers, N1 and N2, the events of these being in our random set are very much dependent, okay? And it's this dependency that leads to all the difficulties when we're doing calculations. On the other hand, we claim that the primes and this random set share the similar local statistics.  All right, page 15. So to state some of our theorems, I want to introduce the notion of a uniform Hardy-Littlewood conjecture for an arbitrary set. Doesn't have to be primes, just an arbitrary set. Okay, so the function Y governs the length or the width of the K tuple and capital K is an upper bound on the number of terms and C is governing the error term. Okay, so there are various conjectures out there that the primes satisfy this with, let's say the width up to root X, K up to log X and with essentially square root cancellation, okay? So there are various conjectures that appeared in the literature. The computational evidence is pretty good for these at least in special cases and what I mean is the square root cancellation on the error term. So for our random set R, we proved that with probability one, we have this Hardy-Littlewood condition not only for individual values of the tuple H, but uniformly. So the width could be a power of X to a power of log X. The K can be as much as a power of log X and then we have some exponent here which is always less than one. In other words, we have power savings all the way up to when C is close to one, we're still getting power savings in our Hardy-Littlewood expressions. Okay, so this is very strong and I haven't written it here but Granville's model does satisfy the Hardy-Littlewood conjectures in some uniformity but it's very weak and the error term only has a logarithm savings as opposed to a power savings. And that's essentially because in Granville's model you're not sieving up very far, you're only sieving by primes up to about log X and ours were going up to a power of X itself. Okay, next slide page 16. So I want to now bring in a connection that Gallagher made between the Hardy-Littlewood statistics and gaps between primes. So he showed that if you assume the Hardy-Littlewood conjecture where the size of H is fixed and in some interval of say zero to log squared X, then he showed that the gaps between consecutive primes essentially have this exponential distribution. So the primes themselves suitably scale behave somewhat like a Poisson process. The main tool in Gallagher's theorem is an average estimate for the singular series over all tuples of a given size in a given interval. This was later improved by Montgomery and Sound who gave some further terms in the asymptotic here. However, the formulas are very poor, they have poor uniformity in K, which is actually really important for our work. And I should mention that Gallagher's argument works for any sequences. In other words, so if you make the same assumption for any sequence, you'll get a similar result for the gaps in that sequence. Now, page 17, we proved a very general result that says if we have an arbitrary set that satisfies the Hardy-Littlewood conjecture in a certain range of uniformity, so we want a width of log squared X. The K is bounded by, okay, kappa log over double log where kappa is some, it could be either fixed or going to zero, so here's the range, there's a big range of kappa that's available. So kappa is determining how good our error term is. Then the large gaps in the sequence are infinitely often at least kappa times log squared divided by log log, okay? So that's for any sequence, it applies to the primes, it applies to randomly chosen sequences, any sequence you want. There's nothing random or probabilistic here. So in particular if K is fixed, then the right side here is almost the size of Kramer's conjecture for primes. So here in this corollary, we suppose that for primes, we have a uniform Hardy-Littlewood conjecture where the K in the K tuple is as large as log over double log, and with a mild power savings, then we'd have this bound here, okay? Again, there's nothing, there's no model here, this is just assuming the Hardy-Littlewood conjecture in a certain range of uniformity, then we get lower bound for gaps between primes, which is almost the size of Kramer's. So we're pretty happy with this because this goes way beyond what you could do with Gallagher's techniques, which I'll talk about on the next slide. Another corollary, because we know from the previous slide's theorem that our random set satisfies a uniform Hardy-Littlewood conjecture, that implies that the gaps in our random set are at least a size log X to the two minus little o one. Okay, unfortunately, this roundabout way of obtaining the gaps in our sequence through the Hardy-Littlewood conjectures is not the most efficient way. So we can actually say more about the gaps in our random set, which I will talk about very shortly. Okay, page 18. So this is briefly a rough sketch of how you obtain gaps in an arbitrary set from the Hardy-Littlewood conjecture. And what's interesting about it is even though it's about an arbitrary set, even a deterministic one, the probability, probability of our random set makes a role, plays a role. And the reason is because it's, well, it's very easy to first reduce by inclusion-exclusion to a sum, a multiple sum over sort of Hardy-Littlewood K-tuples sum here. But now we're going to reinterpret that as a probability instead of writing down the asymptotic in terms of the singular series. And that then greatly simplifies life because we can then reinterpret, again, throwing away error terms, reinterpret this as a certain expectation, which by the binomial theorem is just the probability that our random set has no elements in the interval zero Y. Okay, which is something that we can work on computing. All right, so let's now go to page 19. And this brings up a question which was asked in previous versions of this talk. We know that our random model, or sorry, we know that any set that satisfies sufficiently uniform Hardy-Littlewood conjectures has large gaps of size, almost log squared X. Can we obtain any upper bound on large gaps? I mean something better than, say, square root of X. And the answer is no. And the reason is actually very simple. So if you have any set A that satisfies the Hardy-Littlewood statistics and you simply remove the inner, so the integers from a square root Y length interval for, let's say, a very sparse infinite sequence of Ys, that's not going to affect the Hardy-Littlewood statistics at all. It'll just be absorbed into the error term, but we've created then a very large gap. So we can't do anything about upper bounds on large gaps starting just with Hardy-Littlewood. Okay, page 20. So the last thing I want to talk about is what kind of gaps our new random model produces. Now it's not quite as simple as with Kramer's model because we have this dependence. So I want to connect it with something called the interval stif. So with Z being just a little bit less than square root Y, and that little bit less is important, I'll define WY to be the minimum over all choices of residue classes AP of zero Y intersect SZ. Okay, by the Chinese remainder theorem, if that's a little too complicated, you just look at the next line. This is the minimum over all starting points U of the number of integers in an interval of length Y that have no prime factor at most Z. Okay, so that's an interval sieve. Now, because Z is a little bit less than square root Y, we can use the linear sieve bounds of Yvagnets to obtain a lower bound for WY. We also have a trivial upper bound which comes from taking U equals zero or to take all the APs equals zero. That's the same thing. And that's just, we're just counting primes essentially. Now notice there is a gap between the upper and lower bounds. So we have Y over log Y is an upper bound, essentially Y over log squared Y is lower bound. What's the truth? Well, there's a folklore conjecture that Y over log Y, the upper bound is true. On the other hand, this preprint that Andrew Granville sent me just a few days ago, shows something quite interesting, which is that if Z equals zeros exist, so getting back to the topic I started the talk with, then infinitely often, the WY will be little O of Y over log Y. And in fact, depending on the strength of the Z equals zero, you could make this quantitative. But I don't think that it's possible to reach the lower bound just because we're right at the cusp where the linear sieve bounds just start to become positive. But yeah, so that illustrates the difficulty of pinning down what WY is. I mean, at the very least, we have to solve the Z equals zero problem, which has been around for eight or nine decades now. Okay, next slide. Now we get to the punch line. So I'm gonna let G sub R of X be the largest gap between elements in our random set, which are less than or equal to X. Okay, WY I defined before. G of U is now a new function, which, well, it has a funny definition, but since this WY is going to infinity, you can see at least as it exists, the existing bounds on WY show that G of U grows at least linearly and at most a bit faster than linear. U log U over double log. Okay, so with these definitions, we prove that our random set has maximum gaps that are essentially little g of two e to the minus gamma log squared X, okay, where this epsilon can ignore it if you want. So we use a lot of tools from number theory and large deviation in qualities from probability. And incidentally, as a result of our analysis, the same bounds hold for Granville's random model, okay? So his random model also gave bounds of this type. So now go to the last slide, okay? So here's the theorem and plugging in the bounds that I showed you for the little g function. So we would actually conjecture that we can remove the epsilon here, but we don't know that the little g function behaves nicely locally, so we have to leave it like that. But conjecturally, it would behave like this and the little g function is at least U and possibly slightly larger in magnitude. And that gives us predictions for the largest gap between primes, which are at least of size two e to the minus gamma log squared X, that's the lower bound that Granville gave. And an upper bound, which is a factor double log over two triple log larger. And our, okay, so there's no uncertainty here except the uncertainty is in the answer to the interval sieve problem. So once we know the answer to the interval sieve problem, then our prediction is something specific about G of X. So this opens the possibility that the gaps between primes could be a little bit larger than what, even a larger order than Kramer predicted. And in fact, if you assume Ziegels zeros exist, then our prediction, well, our model then has gaps of larger than log squared X in order, and we predict the same would be true for the primes. So somehow this now, comes back full circle to the topic I started at the beginning of the lecture, that if we know Ziegels zeros exist, you can somehow do better with large gaps, okay? So that concludes my talk and I guess we'll open it up for questions.


Hola, hola, buenas tardes, ya funciona el micrófono. So I'll switch to English now. So welcome. So my name is Israel, like the country. But I don't name after a country. The country is named after me. No, I'm not kidding. I'm going to talk today about machine learning. But this is not going to be the typical machine learning talk. So I'm going to talk about how to have an impact in the world with machine learning. You have to do much more than just learning from data. So when we talk about machine learning, a lot of people think of the tragedy of machine learning and artificial intelligence. We will give birth to machines that will be so intelligent that will enslave humanity in the future and will do our jobs for ourselves. And then we will be out of jobs and so on. But to be honest, for this terminator point of view of machine learning, to be honest, I think we have already actually created autonomous killing machines. So I actually want to ask you something. So do you believe the same? So when do you think that we have created as humanity the first autonomous killing machine? So I have here three different dates. Who thinks it was like option A, 1997. Can you raise your hands? And the B, 1982. Well, a lot of people. So maybe you have already seen this talk, maybe. And option C, anyone? So 1997 is actually Judgement Day in Terminator 2. And then 2017 is Judgement Day in Terminator something. It's another movie that wasn't successful. But it was actually in 1982. It was in 1982 when we as humanity created a machine that killed people in an autonomous way and independent way. This machine wasn't created to kill people, but it actually ended up killing people. So I'm talking about the Stedak 25. This is a health industry device. It's a machine that gives radiation to try to cure some types of cancer. It was manufactured in 1982. It was put into production in 1983. And ended up killing some people. So if you want to know more about this specific machine, I'm not going to talk about the machine itself a lot. So there are those links. You can actually search for it in Google. And you will find a lot of information. So this radiotherapy system killed four people between 1985 and 1987. So it was four people, not one, not two, not three. But four people killed by a machine and actually injured more people before the administrators of this machine decided to stop the machine and look into the situation to try to find out why this machine wasn't working as intended. It wasn't a human error in the sense that it wasn't that the operators of the machine decided to give a lethal dosage of radiation to these patients. It was the machine itself not actually fulfilling the orders that it was given to perform its tax. So then why the machine decided to kill these four people, not obeying these orders that it was given? It was actually because of software engineering. Rather because of lack of software engineering. So this machine has a software that controls the machines, a firmware. So when this case was investigated in detail, the investigation team found lots of issues. So take into account that this is a machine in a hospital. So it's a health department. In this case it was Canada. This is a very serious issue. So the investigation was a very thorough process and it was really a big deal at the time. So they conducted a very detailed investigation and they didn't find any problem with the hardware of the machine. But they found several problems with the software of the machine. They found that when the developers of the firmware of this machine were working together in a team, for instance, they were not doing code reviews. Every developer, when they wrote her own code and committed into the common repository that were using for source code, did so without any approval or opinion by any other member of the team. When they were writing software, they were actually not conducting any kind of unitary testing. So they were not testing the code at all. So when they wrote a new piece of code, a new feature, and they put that into the firmware into the code of the firmware of the machine, they didn't test it. They were, so it was a company that had several devices, several, it was a health industry company with lots of different machines, with common parts for different parts of the firmware because there wasn't a lot of shared hardware between different kinds of machines. And the team was copy pasting a lot of code from other projects that they didn't really understood what the code was doing. They were not really aware of all the implications of that code, but they had like a basic knowledge or a basic intuition of what the code could be doing and it was useful for the post-ports that they needed in their code. And so they copy pasted to try to solve the problem in a quick way. And actually the investigation, when it tried to reproduce the situations that led to the death of all these four people, the development team and the hardware team of this company was not able to reproduce the situations that actually led to the killing of people. They were not able to reproduce any situation when you gave orders to the machine and the machine was actually doing something very different to that. It was very difficult corner cases, so it was with the knowledge that they have of the machine, the creators of the machine with the knowledge that they had, they were not even able to reproduce these situations. So when I think of all these things, it sounds very familiar to me. So I'm gonna give you some hints. So I work in now as a cloud engineer at Google, but I consider myself a data scientist. I've been always involved in data science years ago. And when I see the software project with all these features, it's data science. So in data science you have things that don't really follow rigorous software process that they don't do testing and so on. They just focus on getting new code written. So to me, this is data science. So if we compare data science with software engineering, so data scientists would never call itself herself a software developer or a software engineer. I know lots of data scientists that they actually defend that they are not software developers. I disagree with that. So data scientists write software and therefore they are software developers. Of a very special specific kind, but software developers nevertheless. So the problem is that with data scientists we have not been educated in software engineering. So many of us are learned to program by ourselves in a self-taught manner and we don't know, let's say, the tools of the trade of people who do professional software development. But software engineering is very important for data scientists because the role of a data scientist in a company or the aim of a data scientist in a company is to have an impact on the business. So when you are hired by a company that have lots of data, what they want to have is to try to leverage that data to obtain some kind of advantage in the market to know better the market, to know better the customers so they can target better the offers. So you need to have this impact in the world. And if you are not able to industrialize because you're not following the best practices to do a software project, this is very difficult. And if you don't put something in production, in coordination with the rest of the systems on the company, in coordination with all the policies and common situations in your company, if you are not able to do that, you will not have an impact in the business. And then your aim as data scientist is not fulfilled. But data scientists should not be blamed for this. It's actually a problem of tools. So we are in need for better tools in order to be able to do our work. If we focus in the case of software development, so software engineering is now even an academic discipline in the past life. I was a professor at the university doing research on software engineering. It was very fun. So, but it didn't start as an academic discipline. It actually started in industry when people, the first people who were doing software realized that it was a very complex activity and that it led to many different kind of situations to many kinds of incidents and problems. There is a famous video by Dijkstra talking about the Apollo 11. I think it's the program that they discovered back like a couple of days before the launch of the rocket. And when he asked, oh, so how did you discover the back? So what kind of testing strategy did you use to discover that? The answer is none. They discovered it just by pure luck. So the situation is similar in data science nowadays. But the good news is that software engineering is like the realization of all these experiences in the form of a body of knowledge that we can now try to apply to a new situation that is a data science and to avoid committing the same mistakes that the software developers have committed historically in the field. So notebooks. The title of this talk was notebooks are not enough. What is a notebook? So I'm assuming that most of you are familiar with what is a notebook. We're not talking about the laptop. I'm not talking about the paper notebook. I'm talking about a specific tool that is one of the favorites of data scientists. Here in the picture, we have a Jupyter, very well-known tool. You probably are familiar with it. A notebook allows you to write code in combination with documentation, in combination with images. It's a very nice tool. It has very nice capabilities. And you can really create beautiful code and beautiful notebooks with these tools and share it with other people. And in fact, Jupyter is not the only tool that provides these kinds of interfaces. They are now becoming kind of popular. For instance, here are another example of another tool, which is Apache Zeppelin. You see there are some code with some graphs and the graphs in this case are interactive. They are a very nice tool. Because they are so nice, notebooks are very popular between scientists in general and data scientists in particular. So the authors of Jupyter, when they started the project, the use case that they had in mind, the problem they wanted to solve, is sharing experiments in science. So there had been lots of cases where people tried to reproduce the studies of others, like previous studies, previous papers, and they were unable to do so because of the complexity of all the dependencies between software, the separation of software and data. And when they tried to reproduce, they were unable to do it. So thinking of that particular use case, Fernando Perez started to work on this tool and now it has become very popular between data scientists precisely because of that. Sharing an experiment written in front of code, documentation, graphs inside the notebook, it's straightforward. It's just, you send the file and that's it. And if the notebook contains some data inside, like in forms of variables, a data frame, a table, whatever, it will be included in the notebook too. And if you have some external data, you just bundle everything together and share it. It's really straightforward to share. But this tool that was created with science in mind to be able to share things very easily doesn't work very well with software engineering tools. It wasn't thought to be working with this kind of tools. So like testing code reviews, conversion control, all these kind of tools don't work well with notebooks. We have some workarounds for some situations, but those workarounds are not enough. And this is a big problem because this lack of integration within engineering tools is encouraging bad habits and bad processes. And bad habits and bad processes are a big problem in data science. So when you don't do code reviews, when you don't do testing, when you copy paste code from another notebook in due to your notebook because you cannot import easily code from a notebook because it's not a package, it's not a library, when you do this, you may end up creating an autonomous killing machine that will kill people by itself when it decides so. And it will be very difficult to control. So specifically, what are these bad habits and bad processes that notebooks encourage? In a notebook, you have a lot of hidden state. So when you are writing a Python script, think of a Python script in a text file, when you are writing a program in a text form, which is, let's say, the traditional way of writing code, in order for a line to be executed, the previous line needs to finish. This seems kind of obvious. You write the program and the program is executing line after line after line after line. In my experience as a professor, one of the courses I was teaching was introduction to programming for engineers that were not computer science engineers, civil engineers and other kind of engineers. This simple thing, it was the first problem that beginners, people who were starting to learn to program, this was the first problem that they had trying to understand how a program works, understanding that the program is not fulfilled all at once with all the lines defined in different conditions and they are trying to be fulfilled at the same time all at once. It's just that you need to execute one line and wait until the previous line has finished to execute the next one and so on and so on and so on. In notebooks, when you have cells that can be executed in many different orders so there is research about this topic, this is actually highly confusing for beginners. Considering that data scientists are sometimes, even learning to program at the same time that they are learning to do data science, I think this is kind of a very dangerous situation. Other but having some processes that the notebooks encourage, when you cannot integrate a notebook with a version control system, you will end up with this very well known problem of having files in many different versions so these tools will actually generate files with these kind of names by default. So it's very tempting to use this really silly way and a poor way of managing versions as your version control system instead of using proper version control. You cannot do any kind of a libraries management so I give you a notebook, how do you import it in another piece of code? You cannot do that. You can actually import notebooks. There is some evil plugins for Jupyter that allows you to do those kind of things and again, that's also quite dangerous. And testing for instance, can you test code in a notebook? You cannot write unitary testings for each one of the cells that you have in a notebook. That's simply not possible. And for code reviews, because notebooks are not integrated with a version control system, you cannot do a code review of a modification, a small chunk of code to review what it's doing and approving that before it's merged into the code base. So notebooks are actually quite dangerous because of all these. And in data science, because we are using notebooks, we are facing many challenges that the software engineering community in general faced decades ago. We're talking about the 60s and the 17s and that are normally overlooked. So here's a picture of a friend, Ahamed Hassan, research in Canada. This slide is actually copied from one of his presentations. It's actually, if you look at the picture, it's exactly the same thing that is in the picture. You have the link to Twitter where this picture is. So there is no common accepted development process for data science to do software in data science.  So evaluating whether a model is performing well. So we have, of course, metrics to evaluate the performance of a model, comparing with some data. But things like, for instance, values and furnace, these are not so easy to test. So a model will easily reproduce the same unfairness and values that is introduced in the data. And this cannot be easily evaluated just with the typical metrics that you have in a machine learning model. When people do machine learning, they try to focus on getting the best model for a particular situation. They try to get the best classifier or the best regressor. So they focus on performance. And in order to be able to understand what's going on in a difficult situation, like a machine killing people by itself, you need to be able to explain what the machine is trying to do. So its planability, it's better than performance in data science. And the things that are doing data science, they normally overlook everything that is related to operation and maintenance. This is not taken into account from the beginning of the project. And this becomes a problem later on. So data scientists are a very specific kind of people. So I've been very happy working in industry. When I discovered that I could do data science, I have to confess that I didn't even know what it was at the time when I was first hired as a data scientist. But I was very well treated in my first position as a data scientist. And industry is treating data scientists very well because they consider that these kind of people are very valuable for business. So one way to be able to make data scientists happy is to try to give them autonomy on the way they work. So when you are hiring chess players, you don't want them to become chess pieces. You want them to be chess players. So companies, when they are hiring data scientists, they try to organize them in small teams with autonomy on how they work, on ownership on the tools that they use, the processes that they use, and so on. For instance, this is mapped very well to agile methodologies in cases where companies are using agile methodologies. This way of organizing data science teams integrates very well with the rest of the methodology. But this has a problem. When different teams of data scientists are working in isolated teams that don't talk to between them very much because each team has its own goal which is achieving whatever they're trying to do. So that's the main goal and collaborating with other teams becomes like a second-class aim. So each team starts using their own practices which are not similar to the practices that other teams are using. So the projects start to be inconsistent between them and the promotion to production is very painful. So normally in a company, data scientists working in these kinds of things are separated from the engineering part. So people who need to put into production the things that data scientists do are not data scientists. They are different kinds of people and they will talk to the data scientists once data scientists have finished their first prototype. They will inherit this prototype and now data scientists are done and it's the problem of the engineers to be able to put this into production. When those projects are not always using the same practices, when each project is a new thing and the engineering team has to start from scratch, has to start over again, then putting that into production is a very sensitive process and it takes a lot of time and many times projects die at that phase just because it requires a lot of effort to try to understand that complex prototype in order to be able to put it into production. And also the sharing of knowledge and the collaboration and sharing of experiences becomes more difficult because people are not actually talking in the same language, even between teams of data scientists. And this is very important in order to do this because it's a very complex activity and requires a lot of studying and sharing knowledge and learning on the go and life of learning while you are working and without collaboration that becomes just more difficult. So how can we fix this problem? So here I have written seven, not ten commandments of how you should do software in data science. And this seven actually, this is nothing new. This has been standard practices in software development communities for decades now. So you should be using version control. A lot of the data science teams fulfill this and you should be using also continuous integration. Every time there's a new change into the repository, you should be testing automatically that, you should be checking that code automatically. That's not so... Not found so commonly in projects. Code should be reviewed, period. You cannot put code into a repository without having someone else having a look at your code. That's a very dangerous practice. And if you don't do that, then you're just living on the edge and you will have a problem sooner and later. If you got new code, you should write the corresponding test. Otherwise, your job is not done. When you find a problem, or when someone requests a new feature or someone has a new idea, you should record that in order to be able to follow up later on, in order to be able to have an historical look about what were the most common problems, in order to be able to write a roadmap for the future of your system. So you should have an issue trying to fulfill all these needs. And it should be connected to your repository. So when you go to a bug report, when you go to a feature, you should be able to recover exactly the code that is related to that bug or to that new feature. And you should agree on how you are gonna write code. You cannot let every person, these geniuses, data scientists, test players to do things in the way they think are better. You should agree on something, whatever, but you agree and everyone should be using these same practices. And this should be checked and tested automatically with every new change in the code. So in summary, data science needs a software process. With the adjusted gravity is a typical software process. And this is a software process that is focused on achieving software quality, on trying to achieve zero problems, zero defects once the code is released. Of course, zero defects in software is impossible. It's like an ideal goal. It's all software has bugs that we know that. But focusing on trying to reduce this number of problems that are found once the software is released will minimize situations like an autonomous machine killing people and it will ensure or it will foster the quality of your code. So you may think that this is really like a pain. It's like any data scientist, I don't really need this. I do code that is so good, that works, that does it. But when you don't do this, you are not doing good science, period. So this is not something that I say. It's something that was said sometime ago by Francois Chollet in Twitter. So he's the author of Keras. When you have code that contains bugs and you do nothing about that, then you are doing bad science. Because source code is the core of any scientific project, including data science. Not your very sophisticated methodologies and machine learning methods. It's the code, it's the main artifact in your project and you should look after it. So in summary, what I'm trying to say when you are doing science, doing science is very fun. So imagine yourself in these nice mountains trying to go around hiking. So you do exercise, you breathe fresh air. So it's very nice. And then you discover how to reach that city through a way that you find in the mountain and how to go through the next mountain, through another way and so on. But now imagine that you have to do that with 1,000 people. So you have to move 1,000 people from point A to point B in these mountains. And you have to do that through the ways that you have discovered using science. Very soon you will have people falling off a cliff and killing themselves in accidents. You will have twisted ankles. You have all kinds of problems. How can you solve this problem? We think you need it. So not only you will have accidents, you will be wasting a lot of resources because people will take a lot of time to reach from point A to B. And we know how to solve these problems. We have invented beaches for that. So here I'm showing two different pictures. People may understand that science and engineering are different things. But they are actually in data science in particular. And in machine learning, they are the same thing. You cannot understand science without engineering. You cannot do engineering without science in machine learning. They are the same thing. In machine learning, you don't do machine learning with pen and paper. You do machine learning with data, code, and a computer. And therefore, in this specific case, machine learning science and machine learning engineering are the same thing. So we are almost finishing. So some of you may be reacting to this talk, like reacting to this idea. It's like, no, no, no, no. So data science, machine learning is a very special thing. That is not just a process that you can follow to do machine learning, to do creative work. Because this is just art. It is not. It is a very complex activity that requires intelligence, dedication, resources, and so on. And probably the process that you need in order to do this in the best way, it's different to the process that has been used for genetic software in the past. But there is a process. OK, so thanks all for your attention. I'm almost done. You can wake up now for the next talk. So I started this talk talking about killing machines, about how when I talk this with my family, my political family, about artificial intelligence, all that we are doing, machine will really enslave us, and so on. It is not true. It is not true.   So software has eaten the world, and we have to do it better. So the academic community is already looking into this topic. If you want to know more, I recommend you to read this paper. It's a very short position paper. There is not a lot of research on the topic yet, but they are starting to look into it. So this paper appeared one month ago in the IEEE software. It's an academic half-industry journal of the IEEE organization. And the people who wrote it are very well-known researchers in the software engineering community. So they also know about machine learning, some of them. So thanks all for your attention. One more minute, and we finish. So remember, so tonight, when you get home, and you have to talk to your wife, sister, family, and they ask you about this talk, so these are the points you have to tell them. So remember these points, the rest you can forget. So data scientists are software writers, period developers, very specific developers, very special, but software developers, because they write software. Without software, they cannot do their job. So they are developers. Full stop. Notebooks are a very nice tool. Really, really, you can do very beautiful things. I like a lot using notebooks for things like sharing documents with my students after class and during class and so on. They are a very good tool for some purposes. But then we can, an evil tool when you start writing products in the form of notebooks and then you give your prototype to the engineering team in the form of notebooks. If you are doing that, you are being evil, period. You need a process to do data science. It doesn't matter that it's a creative activity. It is. You need some creativity and intelligence in order to do machine learning and data science. But that doesn't mean that you cannot follow a rigorous process in order to ensure quality. So in definitively, what we are talking about is how about this is the right tool for the right purpose. Machine learning is science and engineering. And for this purpose, we need to use the right tools, which is, in this case, software engineering.  Not only terminators. Terminators were highly advanced machine learning systems.  And of course, this is not an easy endeavor. So we don't know how to do this yet. But the academic community is starting to look into it. And you have a lot of well-known and learned past in the form of software engineering.  So thanks for your attention. So if you have questions, now it's time for that. Thanks. One question there. Thank you. Do you think it's possible to have two roles? So a data scientist who does the research and provides a prototype written not in NodeWood, but in Python, for example, with clear inputs and reproducible outputs, and machine learning engineer who implements the production code? Yes, for sure. So that's totally possible. So it's very difficult to find all these skills in just one person. So it's very well known in the data science community that the best data scientist is a team. So you don't have to have all these skills in the same person. So you don't need to have the engineering and the science skills in just one person. But you need to have that within the same team. So working together.  But for sure, so you can have different specializations inside the team. Yes. Thank you. Yeah, hi. Hello. So when you develop a machine learning software, you're not only interested in knowing that it will work in the way that normal software works, but you're also interested in knowing that the performance is going to be good. So how do you test that? Because if you do that in a naive way, you can have tests that last for hours because you have to train the model. So it's indeed very difficult to test. I don't have short answers to give you. But it's something that you can test for sure. So if you have a control data set that you always use for the same kind of test, you can do things like that. So you cannot do probably in a continuous way for every change that you do in a program for sure, that it's just unfeasible just because of the time it requires to train a model and so on. But it's something that can be done with some periodicity. So I haven't talked at all about that situation. So that kind of tests in software engineering are called integration tests. When you test the system as a whole to see if it keeps working as expected. You are not just testing small chunks of code just to make sure that that piece of code works.  So that can be done. It cannot be done in a continuous way, or probably not with the resources that most of the people have available. But it can be done. So you could do nightly tests or weekly tests and things like those and making sure that with a closed data set, you are producing metrics that are within your expectations for the model and that the changes have not affected those metrics. OK, no more questions? Well, thanks all for your time.