Thank you very much for the kind introduction and yes good morning good afternoon and good evening to people who might be listening so I'm going to speak today about the extreme universe which is of course you know the main theme of this series of CTA talks it is essentially the focus of these ground based Cherenkov telescope so I'm going to talk about the extreme universe as viewed from the viewpoint of someone who uses radio telescopes to study the same kind of objects which are the targets for the Cherenkov telescope array and other related arrays that is the most extreme astrophysics so exploding stars accretion events around black holes relativistic jets all the same kinds of objects but observed in the radio bands rather than in Cherenkov light from high energy gamma rays and particles. So the talks going to be in six parts so in the first part I'm just going to give an introduction to the transient radio universe. And you can essentially divide as you'll see in a moment you can divide radio transients into two different flavors I'll talk you through the briefly the physics behind those. And I'll say a little bit about what our observations of with radio telescope can tell us about the physics of these extreme events sometimes in unique ways which really can't be probed in other ways. Later on in the talk, I'm going to talk about a number of specific projects which I'm involved in some which are broader some which are a little bit more narrow, which I hope you'll find interesting and which you you know I hope you'll see are relevant to the kind of projects that the Cherenkov telescope array will be interested in. I'll wrap up the talk by talking about the prospects for for future radio telescope arrays as we move through a very exciting era from the current very powerful and still relatively new radio arrays to things like the square kilometer array which will be here on the time scale of five to 10 years. Okay, so the introduction part one of six the transient radio universe. There are two types of essentially of radio transients, and the first type and the one which I study primarily myself are synchrotron transient so these are. This is a synchrotron radio emission from relativistic electron spiraling round magnetic fields as I'm sure you know, so it's an incoherent process. It's associated with particle acceleration and kinetic feedback so essentially in every case where you have something that's very active or very fast moving flow in astrophysics that's transferring a lot of energy to the rent to the local environment. It's moving much much greater than the local sound speed, and so you get shocks and you get particle acceleration, and we've known this for a long time we've seen this for a long time. In most cases and a lot of cases this is really a unique way to actually trying to measure the kinetic energy which was associated with this extreme astrophysical event. There are all sorts of very exciting and interesting physics with observations other wavelengths, for example, x-ray wavelength will tell you about the temperature of the hottest material optical spectroscopy can tell you about the elemental abundances, but in many cases it's really only the radio observations from the shocked ejected material, which can tell you about the kinetic feedback. And I'll show you two very different astrophysical sources. So this is a range of scales from almost the smallest before we had EHT further and this is for, this is for the active galactic nucleus M87 stepping up in scales to the very largest scales, and all of the radio emission you see from the relativistic jet in this sequence of images this is synchrotron emission produced by shock accelerated electrons probably certainly producing synchrotron radiation from relativistic electrons spiraling magnetic fields. What you see in the figure on the top right here is something quite different. This is a supernova remnant produced by a massive star which exploded some thousands of years ago, and yet the radio emission in this image also this is radio synchrotron emission produced in exactly the same way, except that the detailed physics of how the kinetic energy from the event was transferred to the local environment can vary in the case of jets. It's more collimated, perhaps in the case of most supernovae it's more isotropic. So the originator physics if you like of the event is different, but the physics, which leads to the fact we see the radio emission is very similar. So using a unique path to estimating the kinetic energy associated with these events, radio telescopes, radio telescope arrays, I should say interferometric arrays. They offer a unique combination of things which you can't really do with any other single instrument, unless it is a interferometer array, which is that they can offer very wide fields of view. Particularly in the modern era this is because we're using relatively small radio dishes, the smaller the dish the wider the field of view or what we call the primary beam. But when combined together in an array with long baselines between individual elements, they can also also offer very high angular resolution and that's usually that's really unique characteristic of interferometric array. So it's worth making a point that in terms of general transient surveys for most classes of object they still effectively less sensitive than optical or x-ray telescopes okay so what that means in practice is that, while they are fantastic diagnostics of the kinetic feedback and these usually most astrophysical transients will be discovered first as a result of optical or x-ray surveys and of course we are also in an era of incredible optical survey such as this wiki transients the facility and very soon we will have the the LSS to LSS T Rubin observatory producing lots of optical transients. The other type of radio transient which I won't dwell on much in this talk are coherent transient so this is essentially transients which seem to operate by something like the pulsar emission mechanism or some highly tuned an extreme version of that. Fast radio bursts are the the obvious examples of this, these are very short durations and have very very high brightness temperatures, we can see fast radio bursts at significant red shifts, which is extraordinary really because we think that these fast radio bursts originate from things you know, essentially neutron star magnetosphere so things perhaps you know very very small compared to for example a super massive black hole. The details of the underlying physics is very unclear, it's very exciting and very interesting to study, but the intrinsic pulse narrowness means that we can actually use these fast radio bursts to study the intervening intergalactic and interstellar medium. So here's an example here this was the first ever discovered fast radio burst the so called Lorimer burst. This was a very narrow burst of just a few milliseconds, but what you could see is that is you went to lower frequencies. So that's going downwards on the y axis, we see the pulse which are the the dark bits here arrived at later and later times in this kind of quadratic delay tells us that we're measuring dispersion in the cold interstellar medium. Radio telescopes are currently the only way to discover the sources of this is different to the synchrotron sounds that I just talked about these are really essentially being discovered in large numbers now by facilities such as chime just at radio wavelengths. Now if I tried to put this all into into a sort of a plane or context for all of radio transients one of the best ways of doing that is is using a version of this figure. So essentially what you're looking at here is on an x axis you're looking at timescale timescale of variability and on the y axis you're looking at radio luminosity. Okay, now there's an approximate dividing line here at 10 to the zero gigahertz second so which you can think of as a timescale of about one second on timescales longer than this we typically find and study our radio sources in sequences of images and that's the same with most of my studies and a much shorter timescales going down in fact sometimes as short as nanoseconds, a different operating mode is used, which is essentially called a beam formed model for those of you are not an expert you can think of this as the mode which is used to study pulsars. In the high time resolution regime there are all the standard pulsars there are some very short pulses observed from some, but importantly on the same timescales but a much higher luminosities are the mysterious fast radio bursts and of course there are, there are many there are hundreds more fast radio bursts which could be added to the plot now since China. So here in synchrotron sources live in this space over here this blue shaded region, and the reason for that is that synchrotron emission is limited to a certain brightness temperature of 10 to 12 Kelvin. And we find a whole range of objects from flares stars and brown dwarfs to supernovae to to know v containing white dwarfs x-ray binaries containing black holes and neutron stars, all the way up to to blaze ours associated with supermassive black holes. These are the most luminous objects and they also appear, often to have slightly higher brightness temperatures, that is their surface brightness appears to be a little bit higher than it should be able to be excuse me for synchrotron emission, and that's because they're relativistically beam towards us in many cases, as are some of the GRBs. So I'm going to mainly be focusing on the image plane synchrotron transient so that was an overview of the kind of the field of background of radio transients. So the first of the sub projects that I want to tell you about is Thundercat. Thundercat is a large survey program which myself and Patrick Vaught at the University of Cape Town we co-lead, and this is an image plane transient survey using the Meerkat radio telescope. So Meerkat is a radio telescope which was inaugurated in the northern Cape of South Africa, just over three years ago, and is extraordinarily sensitive and will become over a timescale of about five years will segue will morph into the first stage of the mid frequency component of the square kilometer array. So one of the main one of the main and most successful components of the Thundercat survey is to study X-ray binary systems. So X-ray binary systems are binary systems in which there is a relativistic accreting object, which is a creating matter from a companion star. So if I zoom in on this artist's impression here of an X-ray binary sort of as viewed from the viewpoint of the companion star then you have matter which is overflowing through the inner Lagrangian point. It flows down and it produces an accretion disk around the central object neutron star or black hole. Because the size scales of the inner accretion disk are very small and there's a lot of energy released there the temperatures are X-rays, hence these objects are called X-ray binaries. But the black holes and neutron stars in these objects we now know they produce very powerful relativistic jets, just like the supermassive black holes. And these are sites of some of the most extreme astrophysical processes within our own galaxy, and these we can study in the radio band. It's estimated there are around 10 to the 8 so it's 100 million stellar mass black holes in our own galaxy and maybe 0.01% of these are in binary systems they're really the tip of the iceberg. So what you're seeing here this is a this is a projection of our galaxy as observed with neocat as part of our Thundercat monitoring program. And first you'll note that there is the date which is changing in the lower left hand panel here so this is about two and a half years of radio monitoring of the sky. And actually what we do is whenever we get an indication from other surveys that something is perhaps X-ray bright, and we start monitoring it in the radio band and we do regular monitoring for a long time. So those of you who've worked in the past on x-rays would have seen movies like this for monitoring the x-ray sky, but this is the first time that we've been able to produce such a movie for the radio sky. All of these sources are within our own galaxy and every time you see one of these sources appear or brighten, this means that a stellar mass black hole or perhaps a neutron star within our own galaxy has undergone an accretion event has produced a very powerful relativistic jet, which is causing shocks and synchrotron radiation and is happening within our own galaxy. And there's all kinds of knock on effects for that there's a lot of injection into the surrounding interstellar medium. So this is really a first this program of monitoring all of these sources regularly all the time and really giving us an insight into particle acceleration in our galaxy as a function of time. Let's get forward to the end of the movie then let me highlight some sources here. So everything which has a box around it. We have discovered it with our myriad observations a large scale structure around it. And some cases these are large scale structures which look rather static so this is the long term action of a relativistic jet on the nearby interstellar medium. But in some of these cases we've actually observed jets launched for the first time, and we've seen them propagate through the interstellar medium and decelerate on time scales about a year. Now to put this in context and I'll come back to this in a moment in the entire history of radio studies of these kinds of objects. We had one example where we've managed to track a large scale jet for more than one year. We now have already five new examples from me a cat and it's really revolutionizing this field. And I think this is important for the chair and cough telescopes because we're really seeing in situ particle acceleration over a period of a year from relativistic jets within our own galaxy and we can monitor them. We can see when they re bright and when they might be most active. So let me just focus in on one of these particular objects this is from bright et al 2020 which was published in nature astronomy. This is a sequence of images we made with me a cat of this source and at later times with the VLA. The thing was we did not expect given the relatively poor angular resolution of me a cat so I should say me a cat is an extraordinary sensitive radio telescope but at the moment, until it becomes the SKA it's relatively poor angular resolution. So we did not expect to resolve jets from this source but we did expect to be able to monitor the core black hole for a long time. And in fact we monitored in the end for over a year. So you are surprised by around day 90 we were actually seeing a separated component, and we managed to carry on tracking this component. So this is the approaching side of a relativistic ejection. The central line is location of the black hole, and the receiving line is the appearance and propagation of the the ejected material going away from us on the other side. So to detect and track the motion of these jets for over a year, as they propagated away from the central black hole, and this really was a unique and really exciting data set. I'll just show you a few aspects of those data but I won't be able to summarize everything about those data right now. First off let me show you the light curve of the binary so what we see here is over a period of about half a year detailed radio flux monitoring of the emission from this black hole when it's associated jets. And I can identify a few phases I can say that probably what happened around here was an initial very rapid phase of particle acceleration, followed by a phase of adiabatic expansion from a relatively small size initially, which meant the electrons cooled very rapidly and we lost a electron emission. But then there was a long phase for over 100 days or nearly 100 days where we see the jets propagating through space, but they are not fading away very rapidly and this means that there must have been in situ particle acceleration. And ultimately we see that this deceleration is happening we actually see the components decelerate. And then later on, well I should say sorry, there is a particularly important epoch which I'll just focus on for a moment, which is here, where we observed the one of the ejected components from this black hole, at the same time with the one in radio telescope which is an interferometer in the UK, and with the meerkat telescope in South Africa. We observed them at the same frequency but at very different angular resolutions. Okay, so let me explain why that is important. What it does ultimately is it gives you the energy contained within that component or a lower limit on it. So, here are the observations which we made on that date. So, what you see first of all in this image here the central black hole is over here where you have the pointer. So if you look at these contours you can see that they are extended, and this is the meerkat image and we know therefore that there must have been a new component, which appeared over here. That's the kind of angular resolution that we get with meerkat. It's enough to say there's a new component, but not really enough to see it in detail. But with our Merlin observations, we managed to find we managed to see in a great detail the black hole over here but also exactly where this component was here. But what we also found, I should say, is that we measured more synchrotron emission, more flux with the lower angular resolution telescope, and therefore we were able to associate this missing flux with the range of angular scales between Merlin, Emerlin and meerkat. So meerkat has an angular resolution of a few arc seconds and measured two milli Janskis, Emerlin has an angular resolution of about 0.1 arc second and measured 0.4 milli Janskis. We can put these numbers together measuring the size is really the most important measurement when you want to do the energetics of the ejector. So we can put them together. And essentially what that tells us is that at 90 days after launch from the black hole, there was still over 10 to the 42 ergs. That's a huge amount of energy contained in the ejector. Furthermore, this ejected component was still moving superluminally, which is an effect you get from when things are moving intrinsically, not quite at the speed of light but highly relativistically, which meant that there was still more kinetic energy to lose. So this is a very strong lower limit and this is a huge amount of energy much, much lower than than has been often estimated for these kinds of sources previously. The jets, remarkably the jets from this, this source were also observed in the x-ray band with the orbiting Chandra observatory. So let me just focus on one of the epochs here this is from Espinacetel, but this is clearly this is where the black hole is and this is where the resolved jet component is. We can look at the spectrum from the radio emission all the way through to the x-ray band and we see that both regimes are consistent with a single spectrum of syncron emission from this component. And our best estimates of the magnetic fields mean that this means the electrons were observing in the x-ray band are TEV energy electrons that is that they have Lorentz factors of greater than a million. So we're seeing leptons, electrons accelerated to extraordinarily high energies in near real time in these ejector. Well we saw these ejected decelerate with time I'm showing you three panels here first of all these are actually now for three different ejections. And what you see is the distance from the black hole as a function of time. And if something continues to move at the same apparent speed you get a straight line. And what we're now seeing in every case is that we see these things move in a parent straight line or apparently ballistically for about half a year, and then we see abrupt decelerations. So in a nutshell what the ThunderCat program is showing us is that we're actually able to track relativistic ejections from black holes in our own galaxy. We were able to measure the internal energy of them were able to track them throughout an entire period of typically about one year until they deposit all of their initial launch energy into the surrounding interstellar medium. And by following up on these studies we're probably going to be able to place some of the strut the tightest constraints on the real intrinsic energy of black hole jets that's ever been possible. Of course those those objects are very good candidates to be high energy sources of the kind of thing that Cherenkov telescopes are interested in. But let me change tack now slightly and talk about exploitation of these meerkat data in another direction. So this is a project which is led by Alex Anderson who's a PhD student here in Oxford jointly supervised by myself and Professor Chris Lindtog. And what Alex has been doing is he's been using citizen scientists to look at the wide field images which we get from meerkat, which of course contain hundreds of radio sources. And to see if we can, if we can use sequences of those images to actually find transients and variables for the first time in the radio band. So you'll recall that I said earlier in the talk that you know it's easier to find new astrophysical transients or new extreme phenomena for the first time in optical or x-ray bands but there may be some phenomena, which are actually brightest and easiest to find in wide field radio data. And the way that we did this is we did it by employing citizen scientists, that is general members of the public who accessed our data by the Zooniverse platform. So just to explain what I mean this is, this is a single meerkat image, which is centered on a very interesting binary system called Sursonus X1. So right down here in the center of this, this little nebula in the center is a binary system containing a probably a massive star and a neutron star and we study this source intensively, and we look at how it varies. We're interested in this nebula, this little thing around it, but this whole field here within the enclosed circle is about one, it's about one degree diameter. All of this is image to high quality and there's lots of radio sources in here. And we make multiple sets of images of the same field, and we can compare those images to see if we find variables. So Alex together with our colleagues in the Zooniverse project launched on December the 7th last year a project called burst from space meerkat. And what he did is he presented to enthusiastic volunteer members of the public sequences of images from our meerkat data where our software had identified sources which might be variable. The members of public eyeballed looked at themselves with their own eyes and brains at these data and classified them as to whether or not they really thought they were transients or not. And the human brain and the human eyes as I'm sure you know appreciate are often much much better than any simple automated software you might be able to write finding these kinds of sources. So this is a project that's within the the the overall Zooniverse project, which is most famous for the galaxy zoo project but has done an enormous amount of stuff beyond astronomy beyond physics. And it's the largest citizen science project in the world. So what did we get. Well, here's an example of what the, the citizen scientists saw when they logged on to the program. So they were shown the light curve of one of many hundreds of objects within the field and they were, you don't really see it very well on this particular plot but these points have our bars this is a high significant source. And they were asked to say, based upon some training they'd be given whether or not that really looked like a genuine variable, and they were also shown images snapshot images of the source to say you know, so they could identify artifacts. They were asked a number of relatively straightforward questions which culminated in them saying you know, no I don't think this is variable it's an extended blob which makes it unreliable. It looks like some kind of radio imaging artifact or maybe it's a real bona fide transient or variable. So we set this going on December the seventh we closed it a couple of weeks ago because essentially all our classifications from our initial small data launch were complete. There was something like 10% of all the thundercap data. So a relatively small subset of our data. So in that data release which we released to the public there were nearly 9,000 unique sources across 11 different thundercap fields so about 11 different one square degree fields. So 89,000 individual classifications by 1038 different volunteers so essentially most sources were classified about 10 times which was our goal. And therefore, you know at that point we, we, we announced that the project was closed. This corresponds given the time of which this took to about one new classification every one to two minutes for a period of about 90 days. So there's some interesting sociology associated with these projects so for example the top 20, you know volunteer classifiers each classified over 1000 light curves and objects each. So we needed some way of identifying you know which were really likely to be the real bona fide transits now when we do this ourselves as astrophysicists with with you know much less manpower than 1000 volunteers. There's a couple of diagnostics so we produce a plot like this you see on the right, and we try to use this to identify where interesting sources might be which of course might be new exciting extreme astrophysical transients. So the the y axis here is a is a measure of the fractional variability so this is the amplitude of variability divided by the mean, and the x axis is simply the statistics of a chi squared the chi squared statistic of a fit to a flat line. I you know once you once you consider error bars is something significant variable. All of the sources which are circled our sources which we as astrophysicists, because of course we looked at this data beforehand to it to a relatively shallow degree had identified as certain or probable variables or transients. All of the objects which are not gray all of the objects which are colored. So the purple and blue things down here are objects which the citizen scientists identified as likely to be real transients and variables. Okay, so there's obviously for the things which we classified as variables they got, they classified as variables as well. But there's a whole bunch of objects, particularly in this part of parameter space, where things the variabilities perhaps a little bit less dramatic where the citizen scientists said yes this looks like a variable but we wouldn't have necessarily looked into that cloud of points because we'd have needed to look at well, you know, at 9000 different sources which is a lot of work. And we wanted to figure out some way of identifying one of the most likely ones so Alex has been looking at this, and at the moment he has eyeballed all of the sources for which more than 40% of the volunteers thought it was a real variable, and he's found that about half of those look like they're bona fide variables. So this means that we have about 160 new radio variables and transients just in this small subset of our data. And we don't know what they are yet this is this is a very small fraction of our initial sample we don't know what these objects are yet I'll show you two examples in the following slide but honestly there's a lot of potential discovery space here and this is only going to get better with time. And as I've already mentioned it's important to focus on or note the fact that he sits and scientists identified variables and transients that we as scientists based upon our preconceptions and limited resource would have missed. So let me just give you two examples of objects. Let me focus first on this this one here in the mid right. So this is an object which varies over time with a fractional amplitude of variability of about 50%. The y-axis here is the flux density or the flux that we measure of the synchrotron emission from this source varying between about 0.9 and 1.6 milli janskis over a period of about a year. These are our best estimates of error bars to the sources clearly vary dropped dramatically and then has risen again towards the end. We find a lot of sources which look like this, more than 50% perhaps more than 80%. So what we're seeing on that looks something like this. And these are likely to be low level activity from AGN it may be intrinsic variability or may actually be scintillation that is the propagation of the radio waves through the intergalactic medium makes them appear to twinkle. And it's very important this kind of work is very important because it tells us about the low levels of supermassive black hole activity throughout the universe in a way which perhaps isn't accessible via perhaps studies optical or X-ray wavelengths.  So here's an example of an object which had a single bright detection early on and was never detected again in another image. And what we show here is a sequence of three snapshots of the radio image around where this source is. So this is obviously the detection image and this looks like a radio source. This doesn't look like some kind of weird interferometric artifact this looks like a real radio source. And in other images that you know you see all the structured noise that we're used to, but there's just no evidence of a source there at all. And we really don't know what this is there's no obvious counterpart in online catalogs and we're still, we're still moving forward. The next steps will be to relaunch this program with a lot more data, and also to use our classifications remember we now have 10,000 independent classifications of objects of radio transients or not, and we'll be using those in collaboration with colleagues in South Africa to train machine learning algorithms for the first time to to identify and possibly classify radio transients and who knows some of those might turn out to be objects which are very interesting for studies of the extreme universe in the context of things like the Cherenkov telescope array. I want to change tack again now moving back to a more specific class of object which has very very direct relevance for the CTA and the other Cherenkov telescope arrays. What I'm representing here is work which is led by my, my PhD student Lauren Rhodes and this is studies of the radio mission from VHE GRBs. Gamma Ray bursts are phenomena which I'm sure most if not all of you are aware, which has been known a long time since the 1960s they're short duration flashes of gamma rays from space, which probably have two different origins, one associated with the deaths of massive stars, and another one associated with the mergers of neutron stars which are probably also associated with gravitational wave events. In the last two or three years it's been become apparent that some of these gamma ray bursts and, you know, in fact, all from the longer class of gamma ray burst are detectable with ground based Cherenkov arrays such as Hess and magic. And this allows us to probe the very high energy emission from this phenomenon in ways which was just not possible previously. So, of course, one very interesting question is, you know, do the raid does the radio mission from these very high energy gamma ray burst does it look like the radio mission from other gamma ray bursts for which there's a large body of theoretical literature, and the somewhat smaller body of observational results but it you know there's a well developed theory of relativistic jets behind the radio mission from these GRBs. So here's the position in the first half of a PhD to jump straight onto this topic using a couple of the telescopes to which we have access and studied all of these VHG GRBs at the rate in the radio banter in quite some detail. So here's a result from one of the earliest VHG GRBs GRB 1908 29 a and this is from Rhodes et al 2020. This is one of the best radio data sets ever on any GRB not just a VHG GRB. And it combines observations which we made with the Amy radio telescope here in the UK, and to which I'll return shortly, and also with with the me at radio telescope which I've mentioned down in South Africa. So here's an extraordinarily nice data set and what it shows the combination of the data at the two very different frequencies 1.3 gigahertz, relatively low frequency and 15 gigahertz relatively high frequency, showed that there was both a forward and a reverse shock in this camera burst. So this means that as the relativistic ejector as the jet left the camera burst it propagated into the installer medium. So with the installer medium the jets hit it and a shock is propagating through the installer medium in the frame of the jet, the installer medium is hit it, and a shock is propagating backwards. Through the camera through the jet material, both of these accelerate electrons and produce shocks with similar physics, but different micro physical properties. The mission suggested that there was an early time reverse shock which later on became a forward shock, and then at later times we saw a plateauing or a flattening of the radio mission, which suggested that at late times we were detecting the host galaxy. And there's some interesting detailed physics which came out of this, such as the suggestion that this one exploded into a low density medium, and had a relatively low kinetic energy. This is a recent paper which is just about to be accepted and in press. So those of you are interested in this area may see this on the archive in the next week or so. It's on this GRB 2012 16 C. Now in this GRB. And then a very high energy GRB but the ratio of gamma rays and x rays to optical emission means that it can be classified as what we call a dark GRB so this means it has much lower or significantly lower optical emission than we would expect given typical particles and the obvious interpretation of the typical interpretation of that is that some of this optical emission is being absorbed by dust, perhaps in the galaxy or perhaps even in the immediate environment of the GRB. The study of this GRB led to some very interesting results or as I say this from Lauren's paper which you'll which will shortly appear so she combined some early time optical data with some very detailed radio observations, both with Berlin with me a cat and with the VLA in the United States, and it's impossible to fit a single forward shock model to these to these ejector to these data. And a single forward shock model is often fitable to radio data from gamma ray burst but in this case it was not and maybe that's just because these data were very good or maybe it's because there's something unusual about this event. The easiest interpretation of what's going on actually is if you look at this schematic over here so here's the observer he's there the cataclysmic event which produced the gamma ray burst. And as with all gamma ray burst a highly relativistic jet came straight towards us, there's a forward shock and that produces bright early time radio emission, but it also seems or it's also easiest to model some of the radio emission with a more mildly relativistic wider angle emission or ejector, perhaps, you know as low Lorentz factor is one or two, which you actually still carrying a significant amount of the kinetic energy, and it's producing late time radio emission. Well this is very interesting because this kind of model is actually similar to the models that people have put forward for the for the gravitational wave gamma ray burst event GW 170870. So when Lauren looked at all the gamma ray but the VHG gamma ray burst together and put them in the context of normal or non VHG detected gamma rays gamma ray burst it's not clear that there is any difference in the population. So what you see here is redshift corrected days since the burst. So what you see here is the the the monochromatic luminosity or the radio luminosity from each of these grbs as a function of time. So the gray points you see in the background or a selection of non VHG grbs and the blue points are the VHG grbs which Lauren has been involved in observing at radio wavelengths. The first thing to notice that if you look at the overall distribution of luminosities of these objects then you know and applied, for example something like a column or golf spurnoff test, you wouldn't be able to see that these populations are different. I mean they're probably given the, the low sample size we can't say these populations are different, but it is very interesting that note to note that two of the VHG grbs are particularly low luminosity. And again that's probably a selection event effect because it's easier to detect the very high energy gamma rays went for relatively nearby grbs. So at the moment we we don't know but of course you know it might be that with a bigger sample we do begin to see some differences and of course this is an extremely exciting and interesting area of research potential for the CTA. So let's talk about one more science area which I think would be interesting to the CTA which I'm certain would be interesting to the CTA, and it's observed in the radio band before I just discussed briefly a little bit about where radio astronomy and radio rays are going to go in the next five to 10 years. So excitingly, there is a possible association between a relativistic jet from a tidal disruption event and the detection of an astrophysical neutrino.  So first of accretion onto this supermassive black hole, this produced amongst other things a powerful relativistic jet. Now the radio data on this object as I show in a moment is so good that we can see that for up to at least half a year after this event began a powerful jet is being produced and about, and at about half a year. The neutrino was detected by ice cube, which has a fairly strong probability of being astrophysical so it's a very exciting possible association. So, let me just put these time disruption events in context so this is a, this is one way of representing the whole range of black holes and accretion across the universe. So what I'm plotting here is the mass of black holes and what I'm plotting here is the relative or fractional rates of accretion onto these black holes in units of Eddington which means that they should be about as high as they can go. So the extra binaries I was talking about earlier in the context of the Thundercat project there at the lower end and that's actually handy because we can we think we understand how those can be created. They're in the same mass range as the the LIGO black holes and neutron stars which we see at the highest mass end are the most massive AGN at higher accretion rates and at the highest mass but low accretion rates around 87, which has been so spectacularly imaged by the event horizon telescope. The tidalist disruption events are super massive black holes but as I say perhaps only a million or 10 million solar masses, and because they experienced very rapid changes in the accretion rate we actually think that we can, we can understand a lot of what they're doing in the context of patterns that we've learned about x-ray binaries so that makes them of particular interest to me, but of course the you know the possible detection of very high energy emission I think makes them very interesting to all of you. So this is these results are from a paper published by Stein et al last year they made the cover of Nature Astronomy a new source of neutrinos and in a nutshell then for those of you who haven't seen these kind of schematics before. The idea is that a star passes too close to the gravitational influence of a supermassive black hole, it becomes distorted, something like half of the star ends up spiralling towards the black hole producing an accretion disk and ultimately a jet, the other half is unbound and basically splayed out across the environment of that black hole. The ice cube neutrino detector detected detection of a probable astrophysical neutrino prompted a search with Zwicky, which is an optical transient facility. Zwicky found in looking at their data that there'd been an optical variable in the direction of this neutrino for actually half a year already, and the combination of the optical monitoring with the neutrino detection and subsequently radio monitoring put together a really nice story. This is time since discovery of the TDE but what you should start off by looking at is that this vertical dotted line here this is the time when the astrophysical neutrino was probably detected. Now ice cube independently estimates this has something like a 60% probability to be astrophysical so that's tempting but it's not super convincing. However, a jet bright tidal disruption event in the same patch of sky combined with this brings the probability down to significantly less than 1% so actually makes us moderately confident they're associated. So what we've seen then looking at the the optical data was that they've been an optical variable identified now as a type of disruption event for half a year before the neutrino was discovered. Excuse me. In radio monitoring observations we now can see or we could see then that there was something unusual going on in the radio. So here is a sequence of radio spectra from one gigahertz to about 15 or 16 gigahertz taken at four different epochs so between 40 days and 180 days after this initial event so I recall the neutrino was detected around the time of this last epoch of radio observations. So the key point to take away here is that there is a peak at all times in the radio spectrum, and that very strongly implies not only that there is ongoing energy injection into whatever that's producing this radio mission a relativistic jet, but also that we can calculate the size of the ejector from from measuring that peak. We're able to do that and we're able to see that this is since a function of time we can make measurements associated with these four peaks, we get the size of the ejector as a function of time, which tells us that it's expanding mildly relativistically, and we get the internal energy, which tells us that energy was still being pumped into this jet, at least up to the point where the neutrino was detected. So there's a very nice circumstantial background picture here, which suggests that a very powerful jet from this black hole was being produced at the time the neutrino was detected and I think this is, you know, this is a very exciting probability. The reason that this is exciting or one of the reasons this is exciting is that if you see the neutrinos it tells you that hadronic processes I processes associated with a with a proton are involved. So if you just see gamma rays alone and this could also always just come from leptonic processes, such as direct synchron or inverse content emission. So this is a summary of what we saw, and I recommend those of you are interested and haven't read it already to read Stein et al 2021. So those were there the astrophysical things or radio sources that I wanted to highlight a subset of all the things that's going on in the radio band. So here's your little feel for what's going on then with the development of radio telescopes and over the next five or 10 years we're really looking at a development from the current new generation, which are fantastic towards the square kilometer array, and the next generation VLA, but I think it does raise a question of where will we get our high cadence monitoring data from, which has been really essential for these transient studies. Here's a picture of me account, as it was as it is today and as it was at its inauguration two years ago. On a timescale of about five years, me cat will become SKA one mid. So that means the mid frequency component of the square kilometer array. For all of the science that I have talked about today, all of the syncron science, this is going to be the most exciting and important component of the SKA and it's going to be the most important radio telescope in southern hemisphere. It's going to increase the sensitivity of me cat which is already extremely sensitive by an order of magnitude and increase the angular resolution by more than the order of magnitude. There is another component, a low frequency component, which is under construction in Australia, but my sense is that for most of our physics, it's going to be a little bit less important. The next generation VLA, it's hoped to start construction on around the same timescale, and it's hoped that it will be at a stage by the end of this decade that'll be more sensitive than the existing VLA and maybe has full operations if it gets fully funded by about 2035 and that will in turn be the most powerful radio telescope in the northern hemisphere. So this is already a shiny new radio telescope producing fantastic results in the future is looking very, very bright that this has become more powerful and more exciting and it's a very exciting time for radio studies of transient. However, let me just raise one point for you to think about. So the SKA or the next generation VLA if you're using something in the northern hemisphere will be the main facility for radio science in the 2030s. And it may well be that in the era of the SKA the other radio facilities become off board because it can. It's the nature of the beast with radio telescopes and it can do essentially everything. However, as you will have seen a lot of the science that we've done over recent years, some of which I've highlighted has been with sources which are much brighter than you need the SKA sensitivity for. And if we only use the SKA to its fullest capabilities observing very faint objects will miss a lot of these nearby objects we don't need this sub micro jancy sensitivity to monitor a lot of these bright transients. Now optical astronomy also face this issue and up to astronomy responded by building large numbers of small telescopes to do optical transient surveys. However, the radio astronomy world might not do this and the reason the radio astronomy world might not do this is because the SKA can in principle do this as well. So remember, even though it's a large array, it also has a wide field of view. It's actually sub array which means you could you could branch off small numbers of antennas to do monitoring of bright sources. But my concern is that you know will this really happen this will make scheduling extremely complex. And I can imagine and I could imagine being in their shoes people proposing to observe extraordinarily deep with the SKA will not want some transient scientists using 10 or 20 of their antennas to monitor some title disruption and gamma ray burst gravitational word gravitational wave merger event. So another possibilities which I'd like to see into your mind is that maybe it you know now is the time to consider actually building a small dedicated array in the southern hemisphere for rapid radio follow up and monitoring of transients and radio mission associated with with extreme astrophysical sources and you know and I just I'll highlight the case of Amy which is a radio telescope we use all the time. This is two antennas of the Amy larger a this is me with some of my team when we visited a few years ago. There's actually six antennas used in Amy. It has short baselines the dishes are more than 50 years old it's extraordinary this is very old. It's a very good condition you can use them up to high frequencies, and we use them all the time. And they are monitoring transients more than 30% of their program it's an extraordinary program high cadence. We are associated with many, many publications lots of nature and science papers. These figures here are just a number of figures where Amy has made a vital contribution to extreme transients astrophysics just in the last. So this is for example this is the light curve I was showing you earlier of a stellar mass black hole this is the neutrino TDE, Amy contributed to that study. This is a long term study the jet from a different type of TDE. This is radio flaring from a cataclysmic variable that is a white dwarf. This is radio monitoring of the extraordinary radio history of a transformative supernovae. This is radio monitoring of the very high energy gamma ray burst, all of which, you know, Amy made a very important contribution to. So you don't need to escape to do all any of this and my concern is that perhaps this stuff won't get done if we only have SKA. So perhaps you know I just see the idea in your minds perhaps we do need our own dedicated radio transient array or arrays to do this kind of work in the future and really support the work of things like the chair and cost telescope array. So I will wrap up there radio transients studies I hope I've convinced you is an extremely exciting field it's a growing field. And we're getting to play with a lot of really exciting new radio telescopes and discovering lots of things. Nearly everything we're studying has a counterpart at higher energies and there's real synergies between studies and x rays gamma rays and the radio band. Here cats are favorite toy at the moment my group and others around the world are getting extraordinary science out of this covering black holes of all masses gamma ray bursts. And also you know really opening up this field of commensal transients just doing transient surveys where you find the transients for the first time in the radio band and I love the fact that we're doing this with citizen scientists it's really it's really been a fantastic story. So I think the future is very bright for synergy between radio observations and chair and cove and and other high energy telescopes but I do as I say you know just think let's think about whether or not the SK a low will deliver everything we need to monitor the sources. And I will stop there thank you very much for your attention, and I'm very happy to take any questions.


Are our people unrecruitable? Have we done such a good job leading our people making sure that they love us making sure that every single day We're training them to get better that they are unrecruitable When I have a meeting and I said hey whoever lost the keys is gonna pay the guy that lost the keys isn't mad He's laughing. He's like it wasn't me. I blame my brother. Yeah, it was Ian. It was Evan Here's the goal if I'm telling you right now that your entitlements get in the way You're entitled if you're getting mad I'm gonna tell you this right now. Everybody's got another gear and another level that they can go to am I right would everybody agree Okay, what's it gonna take? What's it gonna take? Watch Berkheimer. Come here. Just come over for me You have no idea and he's he shared some stuff.      You're not a manager.  I mean, you're not leader. You're a manager Does this world need any more managers? Do we need any more people to manage we need leaders? Guys, I want to ask you a question. I'm gonna be brutally honest. Are our people unrecruable? Are our people unrecruable Have we done such a good job leading our people making sure that they love us making sure that every single day We're training them to get better that they are unrecruable If I called any of your people right now Burkheimer and I said I'm gonna give you $20,000 more than Burkheimer's paying you per month 20 grand more. No, you know what? I'm gonna give you 50 grand more Come work for me.  I'm not going anywhere. I love Burkheimer He loves me. I love him. I love his company.  It's a privilege to be a leader It's not a title. It's a privilege You know the reason why it's so easy to beat all these other stores right now because they have managers They suck. They don't take care of their people and they don't say what they're When they say do this they do the opposite or they don't stay consistent How many times have you guys had a goal how to meeting pay from now on we're gonna start doing this But the next week it goes away All the time Look, I'm gonna tell you this. I want you to win in every area of life in this room And I want to tell you this I'm blessed to be here with you guys Okay, and I understand what it's operate to operate as a company business owner I have a team of 30 or 40 people that will give their lives Somebody walks in with the gun they will take a bullet for me every one of them You know why because I'll take a bullet for them And I've watched this guy lay out money to build the empire so you guys can have this great life They sacrificed and leveraged in ways that no one knows I guarantee and they'll never even speak of Because I don't want to stress you out Who's willing to share the pressure of all the years and everything that they put on the line together as a family Who's willing to share the pressure for these new next two years why the market's hot And pay them back for the risks they've taken who's ready Now i'm gonna ask this question How many of you are telling the truth? Why is I need you to support your husband tussments? I needed you to tell your wives the truth That right now we're never going to get this market back. You're making history You're in history right now I'm going to tell you whatever you made last year. You could have tripled that Just to let you know you could have tripled it But we can't go back and get what we lost last year and as as much as you said you did great last year you did do great You know what pisses me off is we could have done three times more and you guys know


Hello. Good afternoon. Thank you everyone for coming. My name is Akshay. I am an engineer at LinkedIn. I work for the Hadoop development team. In this presentation, I will be talking about how to optimize your Hadoop cluster by tuning the jobs that you run on them. So, I have divided this session into kind of three phases. To begin with, I will talk about performance in Hadoop. Some of the challenges users face by tuning job. And then in the second phase, I will share some of the learnings, some of the things we did to address these challenges. And then in the third phase, I will talk about a tool called Doctor Elephant. It is a self-serve tool to help users easily optimize and tune their Hadoop and Spark jobs. So, before we begin, I would just like to talk a little bit about Hadoop at LinkedIn. This will help you get a better context of what I will be speaking in this session and also scaling in Hadoop. How does that work? So, in the year 2008, when we started with Hadoop, we started out with a single cluster with about 20 nodes. They were hardly about 10 users and about 10 workflows running in production. They were predominantly written in Java, MapReduce, and PIG. So, fast-forwarding to 2016, we now have more than 10 clusters. There are more than 10,000 nodes, more than 10,000 users, thousands of workflows running in development, hundreds in production. And now people use a variety of frameworks, including MapReduce, PIG, Hive, Spark, Scalding and others. So, how do you scale Hadoop? Simple. You add extra machines to the cluster to improve its performance. So, you already saw how we scaled from a single cluster with 20 nodes to 10-plus clusters with 10,000-plus nodes. But there is one issue here. Though Hadoop is scalable, that is, you can add extra machines to improve the performance, Hadoop does not guarantee an optimal working cluster solution for you. So, even if you add extra machines, the given cluster may not have the right configurations. Even if you run a job, Hadoop doesn't guarantee that the job will run optimally on the cluster. In addition, you cannot keep throwing machines to improve your performance forever. Adding machines is a huge investment. So, once we decide that we need to tune the given resources, once we know that we cannot keep adding extra machines, we need to try and get the same performance by tuning the given resources. And once you decide on tuning, you need to start measuring the performance of the cluster. So, measuring performance will highlight some of the hardware failures in the cluster, some of the poor performing components in the cluster.  Maybe you want to upgrade the version of Hadoop that you are using. You might want to update the disk, update the CPUs. So, once we start measuring performance, then we can think about tuning specific things on the cluster. So, I have divided Hadoop tuning into two broad categories. So, one is the cluster level performance tuning and the other is the job level performance tuning. So, by cluster level performance tuning, I mean tuning your network bandwidth, your input output to disk, your CPU cores and other cluster metrics. Now, we already have a lot of cluster monitoring tools like Ganglia, which help you tune your clusters. You can even take help from an Hadoop expert to do all of these. But what I am more interested in is, or rather another important part of tuning your Hadoop cluster is tuning the jobs that you run on them. Why is it important to tune jobs? Because we usually schedule our jobs to run periodically. We run it daily, we run it hourly and they keep running infinitely until they actually break. So, if your job is not tuned, then over a period of time, it's going to consume and waste a lot of resources. So, how do you come up with a cluster where all your jobs are running optimally? So, here's what we tried at LinkedIn. What we did is we created a production cluster and we made sure that we let workflows go into this production cluster only after verifying it is tuned. So, the Hadoop dev team would act as kind of a production gatekeeper and we would make sure that users can run their workflows in production only after verifying it is tuned. But there's one problem. This is this kind of restriction to the users. Users are running their workflows successfully now, but we want them to tune the cluster and then only we say you can run it in production. So, we started receiving hundreds of emails from users asking us how to tune a Hadoop job and we started helping them out. But how difficult is it for the user to tune a job and to what extent one can help them? Hadoop itself is designed to let users tune their jobs, right? But there are a couple of challenges with it. One cannot optimize your Hadoop, one cannot optimize Hadoop jobs they write without them understanding the internals of the framework, right? And even if you decide on tuning your job, the information required to tune your job is scattered. So, in most cases, people usually use some high-level tools like Pig or Hive to run their Hadoop jobs and where they are just exposed to a console or a terminal where they can just see some client logs. If you want to tune a job, it's not just enough if you look into this client logs, but you also need to go look into your resource manager logs, your application master logs, there are job configurations, there are job counters, there are task counters and each task will have a different set of counters. So, there's a huge library of information and this is just for one MapReduce job. If you run a Pig job or if you run a Hive job, that might trigger say 100 other MapReduce jobs and then you can imagine how complex tuning becomes, right? And thirdly, Hadoop has a huge set of parameters and it's really unfair for you to expect someone new to know the significance of each of these parameters. These parameters are also interdependent. So, if you tune one parameter, it might affect some other parameters also. So, in short, I would say you cannot tune what you do not know and you cannot improve what you cannot measure, right? So, how do we do this? How can you help the users? All these challenges put an extra burden on the users. It tries to lower their productivity. So, how can you help the users? How can we improve their productivity? So, we started out training users. We would keep regular training sessions and teach users how to tune a job. We would say what parameters you can change, what parameters, what is the effect of changing a parameter? What is split size? How can we change split size? What's min split size? What's max split size? We start teaching them all of these and gradually they pick up how to tune jobs and then they do it on their own, right? But there's one problem here. This doesn't scale. So, as more people started joining LinkedIn, as more people started onboarding on to Hadoop, we had to keep up with these trainings and people coming were from different Hadoop backgrounds. Some did not know anything about Hadoop. Some were experts in Hadoop. And people use different frameworks now to run Hadoop. Some use PIG, some use Hive. So, our trainings had to be specific to our audience, as well as specific to the framework that we started to use. So, at some point we started realizing that we were spending way too much time training users and focusing away from other important issues. So, how do we solve this? So, what we next try to do is rather than educating users rather than training users and asking them to tune, why not have some experts review it on behalf of the users? So, we had the Hadoop dev team, some experts in Hadoop dev team review workflows on behalf of users and give them specific suggestions on what you can change to improve your workflow. So, this again, sadly has a problem with scaling. So, again, if you have more people in Hadoop, there are more workflows in development and the probability of more workflows going to production. So, there are more review requests, requests keep that coming. But the number of experts are limited. But there are a couple of other issues here also. Typically, how this works is a user would set up an appointment with the Hadoop team. He would then go talk to a person from the team, get suggestions. Then he would go back, make changes to the workflow and he would set up another appointment with the Hadoop dev team to review it again. Now, by the time he does that, the Hadoop expert would have reviewed 100 other workflows and it's difficult for him himself to keep track of what suggestions he gave earlier. And what happens here is, sometimes this happens, a lot of two and four. So, user reviews, he goes back, the expert says some other changes and he goes back. So, sometimes it takes months for a workflow to go into production. And the other issue here is, we really do not know by changing specific parameter, has it really improved the performance of the workflow. You cannot actually compare your current execution after you make the change with an execution before the change. And these reviews are subject to human errors. And some expert might overlook certain aspects, different experts might have a different opinion on the matter. It's sometimes difficult to come to consensus. So, both training and expert reviews did not work as we started growing. At this point in time, we did not have a clear picture on how to solve this issue. We couldn't even go back and the help requests kept coming. And we had to spend more of our resources helping them out. So, here's a lesson that we learned from here. That is, scaling Hadoop infrastructure is hard, but scaling user productivity is much harder. So, this is when we decided that we will try out an exploratory intern project to fix this problem. So, we will build a tool to automate all of these. But what's the idea? How will we do this? While doing these expert reviews, we came across several very common repetitive optimization patterns. And we thought we will gather all of these patterns and feed it to this tool as a rule. And this rule will automatically now analyze all the jobs that are running on the cluster for the existence of this pattern. This is what gave birth to the tool called Dr. Elfin. So, what is Dr. Elfin? It's a self-serve performance monitoring and tuning tool for the jobs that you run on the cluster. What does it do? It helps every user get the best performance out of the job. It's a self-service tool for the users of Hadoop.  And Dr. Elfin will then make suggestions like how you can improve your workflow, what parameters you can change to improve the efficiency of the job, how quickly, how can you speed up your job, all those things. And you can even compare and analyze your historical executions. It's also a platform for other performance-related tools to make use of this and come up with some new metrics. Here's a high-level architecture of the tool. At the top, you can see a job generator. So what this does is it fetches the application from the resource manager periodically, say every minute. And once it gets a list of application, then depending on the type of the application, whether it is MapReduce or Spark, there's a separate fetches. There's a MapReduceFetcher and there's a SparkFetcher. The MapReduceFetcher will start fetching all the information that it can get for that application. Say, it will query the job history server, get all the information for that job. And then similarly, there's a SparkFetcher which will fetch all these Spark-related information from these Spark history logs. Now, once you have all the information for a given application, we have now fed these rules into Dr. Elfin. There are MapReduce-specific rules and there are Spark-specific rules. They will run over these data and then they will classify the job as critical or safe. And then you save it in the database and then there's a UI where you can go and look into how your job performed on the cluster. So I'd like to discuss a couple of rules that it has. So one is the MapperData skew. So what is this rule? When you run a MapReduce job, it runs a set of mappers and set of reducers, right? If there is a skewness in the input to the mappers, then we say there's a skewness on the mappers side of the input. So for example, here you can see there are blue boxes and then there are yellow boxes. The yellow boxes are input to the mappers and the blue boxes, which are the mappers, the height of the mapper, represents the run time of the mapper. Now, suppose mappers one to K are reading a larger split of data. Mappers K plus one to map N are reading a smaller split of data. Now, mappers K plus one to map N which are reading a smaller split will relatively complete sooner than the other mappers. And once they complete, they have to wait for all the other mappers to complete. So ideally, we want every mapper to get the same amount of data. So none of the mappers have to wait for other mappers to complete. So how do you fix this? What you can do is you can combine all these smaller splits of data and feed it to a single mapper. If you go to Dr. Elfin, it will tell you what specific parameter you can use. I think it's the combined file input size. If you set that, then all the splits will go to a single mapper. But what's the cause of this? Why does this happen in the cluster? One reason is, suppose you have your file which is a little larger than your HTFS block size, right? In that case, your file gets split into a block size and then a smaller chunk. Now, if you have a large number of files, then there'll be a large number of smaller chunks that will be formed. And the framework, what does the framework do? It allocates as many mappers as there are these number of splits. So how can you fix this? You can set a combined file input size and all these smaller chunks will get combined and go to a single mapper. Another rule, mapper memory. So when you run a mapper-reduced job, you request certain container size from the resource manager, and then your job runs within that resource manager. Now, suppose that the user requests a 4GB of container, but the job when it runs within that container just uses, say, 500 MB of it, right? So the problem here is you are actually requesting more resources which means you're relatively going to wait for more time for this 4GB of container to get allocated. And even once it gets allocated, you're not going to utilize the entire 4GB of resource area. You just need 500 MB of it. And you're actually blocking 4GB of resource on your cluster. So what you can do here is you can request a lower memory by setting mapper-reduced.map.memory.MB. That's for the mapper container. For the reducer container, you would set mapper-reduced.reduced.MB. So all these suggestions will be given by Dr. Liffen to the users. Now, these are two of them.   The spark-specific rules are not as vast as the mapper-reduced. They're still in the early stages. If you go to Dr. Liffen and search for your job, you will see a result. So the highlighted rectangle is one specific mapper-reduced job. And then you can see a username that's the user who ran that job.  And then there's a job ID. And below that, you will see these colored rectangles. Now, these are the rules. So you can see at the leftmost, there's a mapper-data skew. Then there's mapper-time, mapper-speed, mapper-spell, all the others. The color of the box represents how severe it is. So if you click on that specific job, you will get a report on that job. So here's a small snapshot of some of the heuristics. So here at the top, you can see mapper-data skew. It's saying the job is severe. And why? Because there's a group A which has 197 tasks. Each of these 197 tasks are reading an average of 9 MB. Then there's a group B which has just three tasks and each of these three tasks are reading 512 MB. So you can see the skewness here. All those 197 tasks, which are reading 9 MB, will complete soon. And then they have to wait for the other three map tasks which are reading a larger chunk of data. And if you click, there's an explain link. If you click on an explain link, and then it will tell you how you can fix that problem. So similarly below, there's a mapper memory which I explained. It is also severe here. You can see the average physical memory that's the memory it took in the container. That is 658 MB. And at the bottom of that same box, you can see a requested container which is 4GB. So he has actually requested 4GB, but using just 658 MB. So it would be ideal to request a lower value for your requested container memory. You can look at your history. You can look at your jobs history. You can look at your flows history. So at the top, you can see a performance graph of how your job or your workflow performed. At the bottom, there's this table which shows all the executions. At the top most, you can see the latest and then the previous and so on. Each of these colored bubbles represent a rule. The color represents the severity of it. Now say for example, your workflow is running on your cluster. You scheduled it. And today, suddenly your workflow failed or your workflow took a longer time to finish. Then you can actually come to Dr. Liffen, look at your execution. And probably what will happen is your color in one execution might be green and in the next current execution that took longer might be red. So you can identify which specific rule is that, which specific dot is red. And then you can just hover over that dot and it'll give you information like what was the requested memory? What was the delay? What was the runtime? You can then actually debug. Was it actually your job that took that long or was it because it was waiting for resources to get allocated? So you can do all such kind of analysis. Maybe suppose you want to change a specific line in the code. You want to change a specific parameter in the code. You can then, by changing that parameter and running, you can then actually see how it has reflected in the performance of your job. Has the performance improved? Has the wait time improved? Or has the wait time reduced? Or has the runtime increased? You can do all such kind of analysis by looking at your job's history. So how do you define a rule? A rule has an input, a logic and an output. The input are the task counters, the task data, all the inputs from Format Produce. And then this, you do some logic over these task counters. You compute a value and then say, if this is greater than a threshold, then you classify that job as critical if it's less than this threshold, then you say it's green. In Doctor Elephant, we have five different output levels. Red indicates it's critical, and then there's orange, which is severe, then there is yellow, which is moderate, and then there are low and none which are represented by green. So ideally, you want everything to look green for your job. What can you customize in Doctor Elephant? You can write your own rules. So if you have an idea, if you have experienced tuning MapReduce job, if you have experienced tuning Spark jobs, and if you know how to tune a specific job, then you just convert it to a rule. You can look at the way how other rules are written, and just by looking at them, it's very easy to write a rule. Once you write a rule, you test it out, and then you write a help page explaining what that rule is, right? And then you also say what parameters you can set to fix that rule. And once you have the rules, and once you have the help page, you can just go to Doctor Elephant and there's a heuristic con. Heuristic is a terminology in Doctor Elephant for a rule.  And you just restart Doctor Elephant, and Doctor Elephant will start analyzing all the jobs with the new rule that you have written. What else can you customize? You can change rules. You can disable rules. You can change their threshold levels. You can expand to other schedulers. You can expand to other fetches. You can expand to other application types, other job types. Almost basically everything is pluggable in Doctor Elephant. So earlier I talked about something called production gatekeeper. So I said that we maintained a separate cluster for production, and the Hadoop dev team would act as a production gatekeeper to make sure that workflows go into production only after it is verified, it is tuned. So now that we have Doctor Elephant, the Hadoop dev team just can refer the Doctor Elephant's report and then see if everything is green, then you can directly go and run your workflow in production. But if there is something, then we can just ask the user to go back and fix this and come again. But this is again a manual process. So what we did is we created a tool to do this. So we created a Jira bot. So a bot that is integrated into Jira, and the user just has to create a Jira ticket and give an execution to its flow, right? And this bot will automatically fetch that execution, get all the reports from Doctor Elephant. And if everything looks green, then it goes ahead, comments back to the user saying everything is good. If there are some issues, it comments back, asks the user to go back and fix them. And if everything looks good for the tool and it goes to production, creates a project on production cluster. If there is some setup to be done, give permissions to the user, it does all of that. And the user can go and run his workflow in production. So this entire process is now replaced, like the tool itself does it. So about 80% of the workflow is going to production using this tool, without the intervention from the Hadoop Devs. What else can you do? You can monitor, you can use these reports to do other kind of performance analysis and reporting. For example, we have a tool that generates daily reports saying what were the top 25 workflows to tune today? What were the workflows that were failing continuously in the cluster today? And there are other tools which make use of these reports to do cost-to-serve analysis, to compare different revisions of your workflow and others. Good news, we open sourced this year in the month of April. Since we open sourced, we received an overwhelming response. There were more than 60 pull requests. Since we open sourced, there are more than 10 contributors, more than 50 topics under discussion. I welcome you all to contribute and make suggestions and feedback. One thing that would be good is we can all collectively gather all the good practices, all the good optimization techniques that we have and feed it into Dr. Elephant and make it as a single hub where to analyze any job completely in Hadoop, any job that you run in Hadoop, you can do it with Dr. Elephant. Since we open sourced, a lot of companies have shown interest and they have started using Dr. Elephant. A couple of them have running it in production. There were folks from Airbnb who added support to Airflow Scheduler. There's a pull request on Uzi. There are a couple of pull requests trying to scale the fetches and a bunch of other pull requests, adding new features and bug fixes and all of the others. A roadmap of Dr. Elephant. So we're soon going to add real-time support for the Hadoop jobs, show analytics for failed jobs, visualize your workflows and jobs through dependency graphs and also as we expand to other rules, we expand to other schedulers and other frameworks. Couple of references, you can go check out LinkedIn's engineering blog. There's an article on Dr. Elephant. If you want the source code, it's on GitHub. There's also a mailing list and a Gitter chat, so feel free to discuss and share your ideas. And it was also presented in the Hadoop Summit last year. Thank you. Can we monitor Spark SQL? Here, I'm here. Here, here. OK. Spark SQL runs on Yarn. Can we monitor via Dr. Elephant too? Yeah, so as long as you run your framework on Yarn, it can analyze any job that you run there. And by default, it uses MySQL, right, in case if I need to change it to PostgreSQL as a meta store. Yes, it's a player configuration, so you can change your backend as you wish. Which specific configuration file? Because when I tried changing from MySQL to PostgreSQL, I was not successful actually. OK. So OK. There is one app.con folder. There is one elephant.con folder. That is a place where we need to change the PostgreSQL. OK. Maybe you can bring this up in the mailing list and we can discuss it from there. Hey, hello. Yeah, so it will not work with Spark, with Cassandra or something other than Hadoop. Right now, we have support for MapReduce and Spark. Spark may be running on Mesos or? Right. For Mesos, we don't have support right now, but you're welcome to contribute and add that solution. There's already someone who raised that in the forum, so to add support to Mesos. So that's something we would look forward to. OK. OK. Thank you. Here. Here. So like you said, right, that one map, one pick job, can launch multiple MapReduce jobs. So is there a place where you can see RAM, CPU, average, max, min of mapper and reducer, all in one place for all the MapReduce jobs that the pick job launched, that pick script launched? So that's what I mentioned in the end, that we will visualize these things through a graph. So then you can actually click on your pick job and see what were the other pick jobs, the other MapReduce jobs that were triggered by that. You can actually look at your entire workflow and drill down to these. And you can see the RAM, CPU, taken by each one, each job, each application. You can look into all the Hadoop specific parameters. By RAM, I don't understand. That may be a cluster metric. This is more of a metric for the users to tune their jobs. OK. You can't see that RAM, because counters are not user-friendly. OK. And that's fine. OK. Thank you. You mean these profiling information for a job? Yeah, that is something that we are looking forward to do that in Dr. Liffin. I'm afraid we're out of time for more questions. So if you could take the questions offline, that'd be great. Yeah, I'll be here throughout the day. So you can catch me up and ask questions if you have. OK. While the next speaker.


Flu Forces Visitation Restrictions at St. Nicholas Hospital South 14th Street Lane Restrictions Announced Mike McCarthy headed for the Big D These and other local stories are coming up on this edition of Community News Review This is Community News Review, a service WSDS-TV news content provided by WHBL News Review for Tuesday, January 7th, 2020 The National Weather Service is out with its annual summary of weather conditions for 2019 Based on climate reports that are issued for Madison and Milwaukee only 2019 was within a degree or so of historic means and slightly cooler overall than 2018 But while the temperatures were average, precipitation was anything but with rainfall nearly a foot above normal, that made 2019 the third wettest on record for Milwaukee and the fifth wettest in Madison Meanwhile, Lake Michigan water levels were near or at record highs most of the year and remain there at this point The RE Corps of Engineers Weekly forecast indicates that in a month from now Lake Michigan should be an inch below current levels and forecast mostly agrees that another year of near or new record high levels is in store City Forester Tim Bull announced that the removal of trees will require lane restrictions in the city of Sheboygan on South 14th Street between Virginia and Jefferson Avenues from 8.30 in the morning until 3 o'clock, daily beginning today, January 7th No completion day was given during tree removal There will be a speed reduction throughout the work area and northbound traffic will be reduced to one lane Hospital Sisters Health System has placed restrictions on visitors to the eastern Wisconsin Division hospitals including St. Nicholas in Sheboygan, Angela Deja, Public Relations Coordinator made the announcement on Friday in response to the uptick in flu cases now affecting Green Bay, Accanto Falls and Sheboygan Effective immediately, no visitors under the age of 12 will be allowed in inpatient areas and no visitors of any age exhibiting flu-like systems may visit those and those symptoms include fever, sore throat, cough, runny nose, fever, vomiting or diarrhea and only one or two healthy individuals at a time should visit The hospitals note that no restrictions apply to pediatric patients attending an outpatient appointment The restrictions were placed to protect patients' safety and prevent the spread of germs and will be lifted once the situation improves Meanwhile, if you have not received a flu shot, you are encouraged to act now as it is never too late Wisconsin lost nearly 775 dairy farms in 2019 but 2020 is reportedly looking better The National Farmers Union says Wisconsin lost an average of two and a half dairy farms a day last year but Wisconsin Farmers Union District 3 Director Ed Gorel tells WQOWTV that he thinks the younger people will start supporting local farmers because they want to know where their products come from Gorel also says milk prices are higher now than they were last year Wood County Fire Departments responded to a home fully engulfed in flames The fire call came in at 10.30 p.m. on Sunday reporting a structure fire on Highway 73 in the town of Hansen Known as home at the time and fire crews from Pittsville, Vesper, Arpin, Richfield and Rudolph responded along with the Wood County Sheriff's Department Highway 73 was closed between Highway E and Route 186 while crews extinguished the flames The cause remains under investigation And finally former Green Bay Packers head coach Mike McCarthy is headed to the Dallas Cowboys The reports come on the heels of Dallas firing coach Jason Garrett A formal announcement and press conference is expected later this week McCarthy had a 135-86-2 in 13 seasons as Packers coach including a victory in Super Bowl 45 He was fired with four games to go in the 2018 season It was the second year in a row the Packers missed the playoffs McCarthy also interviewed with the New York Giants Cleveland Browns and Carolina Panthers And that is our report for today. Join me again on Thursday for more local news and information on Community News Review News content for this program provided by WHBL in cooperation with WSCSTB