So how much does UX designer actually get paid and what's the top range? What do the top UX designers in the world get paid today? Let's have a look. Hey, just a quick thing so I don't forget. If you're interested in topics like this like salary and you know how much money you can make as UX designer and upping your game as UX designer in general, make sure you join our newsletter. It's an exclusive once per week newsletter written by myself and my team. It's free. It's just like going super deep on UX design and yeah, there's like 35,000 people on that newsletter. It's really great. You'll really enjoy it. If you don't like it, you can unsubscribe, but check it out down below. I think you'll enjoy it. Back to the video. Alright, so I have a pretty good idea what the top UX designers in the world get paid because I'm definitely friends with quite a few of the people working at these companies which have very, very, very high salaries and I know exactly what they get paid. But I realized looking at the comments on our YouTube channel that a lot of people don't really know what this range is. And the first thing I'll say is look, obviously every country is totally different. So if you see me showing you the Bay Area, London, Berlin, whatever, don't send me a message saying, well, it's not like this in Tokyo and well, it's not like this in Delhi. Yeah, sure, it's not. But you know, you can kind of work everywhere because now everyone's working remotely. So I'm showing you what the salaries are at the top range because I think that's what's most exciting, especially because, you know, anyone from any country can get good enough, especially if you've seen our channel long enough, you'll see a lot of the ways to get really good at UX design. Anybody from any country can get good enough to get these salaries, okay? So don't get angry at me if I'm showing mostly US and Western Europe. Okay. So first of all, I'm looking a lot at LinkedIn salary sort of thingy. So this is linkedin.com slash salary. And here you can find, you can just type in user experience designer, you can type in the area. And I think from what I've seen, these are not very accurate. So I'm looking here at senior UX designer, six to 14 years of experience and saying that 54,000 euro is the top range. Absolutely not.  Maybe we're overpaying our UX designers. But this is absolutely not the top range for UX designer. This is actually very low. I would say this is closer to more the mid-level UX designer in Europe. So what I know, what I can tell you is that a senior UX designer in Europe and in Germany and also kind of in London, London has higher salaries and Ireland places like that is definitely closer to the 80 to 90 mark, especially if you've got like eight, nine years of experience, you're definitely going to be making more like 80,000 euro, 90,000 euro. That's the range. If you're a good UX designer, this whole video is about the top range. This whole video is about if you're very good, if you've got some experience, I'm not talking about junior mid-level. We talk about that enough in other videos. I thought you might be excited to hear what sort of the top absolute top level are making. So this is definitely not true. What you would be looking at in Europe if you were an excellent UX designer, in most good companies, you'd probably be more around at least 70,000 euro. So these prices are very low. They're not realistic. I can tell you that for sure. I run a UX agency in Germany. This is not true. And I also know the salaries of UX designers at my clients' offices in the big companies here in Germany. They're a lot higher than this. In fact, I even know people who are closer to the 200,000 euro mark salary as a senior UX designer. Now, let's look at a place where everyone knows there's very, very high salaries and that's San Francisco and Bay Area. So companies like, I'm just going to show you down here, San Francisco Bay Area, it says that like with all compensation combined, the top level user experience designers working at software companies are getting $130,000. This is not, maybe this is the, maybe this is the medium that like, this is kind of like the range is from 77 to $235,000. But I know for a fact that almost every like UX designer who has more than eight years experience is at least all of the ones that I know are all making more than $200,000 or at least very close to $200,000, $180,000, $192,000. I definitely know people in the 250 range as well with the total compensation. And what that means is that you have your base salary and you get things like annual bonuses, stock options, stuff like that, right? So that's something that's built into these tech companies. So yeah, I would say that like, you could say that the within range, yeah, you could work at a startup in Silicon Valley, which is newly funded and get 70. But if you're working at places like Google, Facebook, you're likely to be in the higher range of the salaries. You can see here the different kind of median based salaries. And this takes like the lowest reported and the highest reported salaries. Here you can see Google, their median is kind of 137,000 per year. You can see LinkedIn, 123, Apple, 122. But generally, what's clear is that the salary is always over 100, right? And it also depends on things like company size. So if the company has more than 10,000 employees, you're going to make more money. And I think this is also something interesting. It says your education level, if you've got a master's, that's interesting. Oh no, sorry, this is an MBA, not a master's. If you've got a master's, you get $5,000 more than a bachelor's. I don't know if I really believe that. I definitely think this is not accurate. It's saying like graphic design is number one, absolutely not in terms of salaries. The salary, the highest salaries in the design space are definitely UX rather than graphic design. So human computer reaction, reaction, human computer interaction are definitely the highest salaries. So yeah, I think the interesting thing to see here, and I can tell you as an insider or as a real insider, these companies generally pay their senior employees. If you're working in Silicon Valley, you can expect to be making in the $200 range, $200,000. Keeping in mind, the expenses over there are very high. So if you're living in Silicon Valley, your apartment's going to be very expensive. Your lifestyle is going to be very expensive. So it might seem crazy to be getting paid, you know, $200,000. By the way, if you're curious about who makes more than the UX designers, engineers make more than UX designers. They make a lot more than UX designers in most cases in most of these companies. But yeah, UX designers, if you're living in Silicon Valley, you can definitely get up into the $200 range. However, even though it says here that in Germany and also checked London, it's like saying that this range is up to $80. That's not true at all. You can work in Facebook in London. You can work in Google in Berlin here, and your salary is certainly not going to be 80 or lower when you're a senior UX designer with really good experience and you're also excellent. So if you're the top of your game, you can expect really top of the line salaries. It's one of the best paid jobs on the entire planet. And I think one of the things that is easy to forget is that, you know, you do also have to be very good. You have to be very passionate.  You take an online course and then maybe you start at 20K, 25K.  Most people that started AJ & Smart, my company here probably start, you know, around the 30,000 euro mark if they're junior, even if they're only here for three years, they might end up getting to this 80K if they're really, really excellent, but usually takes quite a long time. So yeah, I just wanted to show you what the sort of top range of salaries are for UX designers. I definitely know people getting the $250,000 mark. I definitely know people getting $200, $180, $190. And I also know people, I mean, in Germany, you know, the highest, probably the highest UX salary at my company here, because we're a small company, I think it's like maybe 85 or something, 85,000 euro. So that's just to be super transparent. That's as high as it goes at AJ & Smart for the UX design roles. There are roles that are above that. But yeah, that's something just to give you a bit of a view. And you know, I think it's going to be interesting over the next few years now that COVID has happened. Now the companies like Facebook and Twitter are saying everyone can work remotely. I'm interested to see can someone living in Berlin work for Facebook, but get a salary that's a lot higher than what you're getting in Berlin. I know they said they're going to adjust for the places where you live. But I'm still curious, I think that they're going to be able to get amazing talent at an amazing price that's still higher than the local region. And I think that's going to be very, very interesting. It's going to be super interesting to see what happens. But let me know if you've heard any higher salaries than the 250. Let me know what sort of range you're in, what you want to be in, how much do you want to be making? And I really hope you enjoyed the video. If you did, hit the like button. A lot of people who watch these videos are not subscribed. It would be great if you could subscribe. Thanks so much for watching.


OK, so firstly I should say thank you very much to the organizers for giving me the opportunity to give some lectures on ADS CFT, which is always fun. I also am an Imperial College R group was founded in some sense by Abdus Salam. So it's actually nice to come to ICTP and see many of the same wonderful things you did here. So I was planning my talk, which I've rewritten, or talk lectures, I've rewritten them a few times and decided it's actually an impossible task. So this will be an attempt to give a sort of coherent summary of ADS CFT to a very mixed audience. So maybe I can have a show of hands just to frighten me. How many of you would class yourselves as physicists and mathematicians? And how many of you have taken a course in quantum field theory? How many of you haven't? Oh, that's actually quite a lot. OK, that's better than I thought. OK, and that's probably judging by the rough ages. That's probably in recent times as well. OK, so yeah, that's more encouraging than I was potentially worried about. OK, so ADS CFT can be thought about from a very geometrical perspective, but really that's missing the point, in my opinion. There's a lot of geometry in ADS CFT, but where it's really interesting is from the perspective of quantum field theory. And I think if you don't understand anything about the quantum field theory side of it, the geometry is somewhat arbitrary and meaningless. So I want to talk about ADS CFT and get somewhat concrete about how it works with a focus to the gravitational physics and aspects of it. But at the same time, I want to give you some flavor for why it's interesting from the quantum field theory point of view. And so let me just state roughly what it says. I should say, as with the other lecturers, I would definitely, if you have a question, do ask it. And if you can't read what I'm writing, do yell. I'll try not to use these end boards, because from my position, I've not been able to always read them. But if my writing gets small, I may be trying to hide something, but don't let me get away with that. So the conjecture states certain, and this is very important. It's not a generic statement. This is very special theories have this property, and only very special theories have this property. Certain conformal field theories, CFTs, which is a type of relativistic quantum field theory, with more symmetry, and we'll discuss that. Indeed, dimensions are dual. And by dual, I mean this is physically equivalent in a way that we'll make precise later on in the lectures. And this is a full quantum duality. This is not a classical statement. This is a full quantum duality, full quantum level. These things are thought to be the same. And it's dual to specific gravitational theories in at least d plus 1 dimensions. So it may be considerably more than the d dimensions here. And which have, let me say, at least locally, and maybe I'll explain that later, ADS asymptotics. So the word ADS CFT comes from these two acronyms. And if I do nothing else in these lectures, I want you to at least have some vague notion what both ADS and CFT mean, or at least one, one or other. I don't mind. That would be good enough. And what do I mean by a gravitational theory? Just to be clear, I mean by a gravitational theory, I mean, I said I wasn't going to write here. I lied. I'll write big. Spacetime is dynamical. And there's more to gravitational theories than Einstein gravity. Just having dynamical spacetime, in fact, is a rather general phenomenon. And there are many instances of it that look nothing whatsoever like Einstein gravity. So this is the conjecture. And it is always important, I think, although often not stated this way, but it is important to understand it is a conjecture. In all cases that I know of, there are no proofs. So for the mathematicians amongst you, it's important that these are conjectures. And I may be able to explain why they're likely to remain conjectures, at least in the interesting cases for some time. Yes. And if you're a physicist, it doesn't really matter in some sense.  But if many of you have taken a course recently in QFT, maybe what I'm going to say is familiar, but at least hopefully we'll get on the same page and notation. And then I'll tell you quickly about some basics of conformal field theory that I'm going to need. And it's really going to be the very basics only enough to then allow me to sort of state more about this conjecture. And then I'm going to talk about what ADS spaces are. And then we'll talk about the duality, this ADS CFT duality. And I actually don't really have a good feel for how much time we're going to have. So there may be quite a few subtopics in here or not depending on how we're doing. But I would rather people understood what I was saying than I said lots and you didn't. So let me just say a few more general comments. The best understood cases of ADS CFT are where we have the gravitational theory is a string theory, closed string theory. So it's some sort of theory of quantum gravity, but other stuff as well. And these typically have some low energy semi-classical limits. And in those limits, they reduce to some often quite complicated but basically gravitational theory that looks a lot like Einstein with some other matter. Doesn't really matter what that matter is. But these are what are called supergravity theories. There are various ones. And the idea is that in these cases, there are specific identifications for specific string theories. And it's then known what this conformal field theory is, as in you can write down some Lagrangian for that conformal field theory. However, there is no free lunch in the following sense that the CFT in this case or in this sort of set of cases always has many degrees of freedom and is strongly coupled. And we'll talk a little bit more about that. But if you remember from your quantum field theory, basically we don't know how to deal with strongly coupled theories. Physics is the theory of the harmonic oscillator and everything you can do with the harmonic oscillator in all its wonderful guises. And if anything isn't the harmonic oscillator, that's pretty much the end of the story in terms of being able to solve things nicely. Is there a question? Yeah. No, no, sorry. String theory, there's two expansions in string theory that you have the string length scale, but you also have the string coupling. Or if you like the low energy, from a low energy point of view, the Planck scale. And you need to be semi-classical with respect to the Planck scale and low energy with respect to the String scale. So there are basically two things you need to tune in order to get to this gravity limit. In order to get to a limit where you have some control that this really does look like, some sort of classical or semi-classical gravity theory. OK, there's also some more exotic cases, which I'm not going to talk about. Higher spin, these are more recent higher spin theories or very, well, really quite recently, the SYK models. And these also seem to have, these are theories of gravity, but they're very exotic. So they don't look like Einstein theories of gravity. I mean, if you think Einstein gravity in 10 dimensions with fermions and form fields and so on is exotic, then you definitely won't like these, let me say. So we're going to stick with this sort of exoticness. This is just too exotic for us. So I'm not going to say anything more about that. In some sense, in these cases, you can say more about the CFT because it's not necessarily strongly coupled, but then the gravitational theory is very exotic. So it's whether you want to call it even, well, it's so exotic it may not be useful if we really want to understand the sort of gravities that we're interested in. Where did this conjecture come from? It derives from physical arguments. As with many interesting statements that are true perhaps in physics, but very difficult to prove or maybe take a long time to understand, there's usually some relatively simple physical intuition behind them, which allowed you to make the statement. And this was Maldesena almost 20 years ago, sorry, over 20 years ago, understood by thinking about objects called D-brains and string theory that there were essentially two ways to look at the same system. And one of those ways looks like a CFT and one of them looked like gravity with some asymptotics. And that's where the conjecture came from. And just to be very concrete, I'm just going to state now an example of this conjecture. Just so you know that there really are, it's not an empty set of theories or equivalences. So the classic example that Maldesena wrote down was something called N equals 4 super Yang-Mills, which lives in 3 plus 1 dimensions. And that sounds very grand. It's a maximally symmetric. So it's maximally supersymmetric. Let me say Susie. It's got some gauge group, which will take to be UN. But actually, it's a very simple theory. It turns out that if you want to sort of explicitly understand it, you can think about just supersymmetric Yang-Mills in 10 dimensions, which just looks like there's the Yang-Mills term, trace F squared. And then there's some fermions. What maximally symmetric Yang-Mills looks like in 10 dimensions? There's just one theory you can write down. And if you just classically reduce to 3 plus 1, you get this theory. So there's a bit more field content when you reduce the gauge field. You get scalars in the directions you reduce on. And so this is a theory of a gauge field, the Yang-Mills field, but also some adjoint scalars and fermions. But I just want to emphasize it's not a tremendously complicated theory. And it's got a sort of humble origin. And this is apparently dual. The claim is this is dual to some 2B string theory in, by in ADS, what I mean is in spacetimes that are asymptotic to ADS. And in this example, let me say there are two couplings in the super Yang-Mills theory. Sorry, two couplings, two parameters, I should say. There's N, the rank of the gauge group. And there's also the marginal coupling. The marginal coupling is normally what you would call G Yang-Mills. And so what is G Yang-Mills? So if I wrote down this theory, there would be a coupling for the gauge field. I would have some scalars, adjoint scalars. There's a trace at the front. There's some interaction terms. There's a few scalars here. And then there's some fermions. But there's some coupling. And normally you work with the coupling G Yang-Mills. But when it turns out you're interested in having a large number of degrees of freedom, which is, as I said here, what you need in order to go to a sort of gravitational regime from your string theory, it turns out that when you look at perturbation theory, this is an observation of a tuft, that the natural coupling to expand your quantities in is not G Yang-Mills, but is what's called the tuft coupling, this combination of N and G Yang-Mills. And if you take N to infinity, this rank of the gauge group, holding this is dimensionless, by the way. It's just some number holding that coupling large. But finite, as N goes to infinity, then this 2B string theory goes to some 2B supergravity. And what these correspond to, as I was sort of saying before, is you're arranging that the plank length and the string length on the string theory side are much smaller than the ADS curvature scale. So we already know in gravity, if I have Einstein gravity, it had better be, from a quantum perspective, that all curvatures are small compared to the plank length, because otherwise I know that all sorts of quantum effects are going to be important. My gravity isn't going to look classical. And so one of those conditions is basically taking N large. And the other condition is that I don't want all the stringy physics associated to having my fundamental degrees of freedom as strings. I really want to just reduce to a sort of more conventional theory of matter. And so I just have to arrange my curvatures also to be small compared to this string length. I should actually say at this point, this is an example of this correspondence. I, the correspondence, there are many examples of this correspondence by now, many, many, probably infinite classes of this. It's also true that the theory doesn't have to be a CFT and the gravity doesn't have to be in asymptotically ADS space. So there's a whole much larger generalization. And in some sense, really the point is that there are certain cases where some quantum field theory on a fixed space, which isn't dynamical, so we just call it a quantum field theory, not a theory of gravity, is due to some quantum theory of spacetime, some quantum gravity theory, maybe strings, maybe something else. So it doesn't even have to be a CFT and asymptotic to ADS. It is in many of the first cases worked out. But in fact, there's a beautiful version which I like where the quantum field theory side is actually just a quantum mechanics. So it doesn't even really need quantum field theory. This is actually an old model called the BFSS model, which dates back a long time and it took a while to understand. It was only after ADS CFT that it was sort of well understood what it was doing. But BFSS model is just a quantum mechanics that looks very much like this theory reduced down to just time, so structurally rather similar, but no space directions. And the claim is that's also due to some string theory. So it's all there. All the physics is there already in quantum mechanics potentially. OK, but it's particularly neat in this ADS CFT set of cases. So what are the consequences of this? Supposing it's true, it's consequences, coincidences. So theories of dynamical spacetime at a quantum level we don't understand very well. Some people may claim to. I would claim they don't. I would claim a lot of people are optimistic about that sort of thing. If this conjecture is true, what it's saying is that actually certain theories of quantum spacetime can just be rewritten as quantum field theories. And quantum field theories, we understand a lot better. For the mathematicians amongst the audience, we don't really understand quantum field theories at a deep level, at a deep mathematical level. There's no proofs. There's no rigor, at least for interesting quantum field theories. But we know that the framework exists and works, and it's experimentally verified to incredible precision. So we've got some confidence that we're doing something right, even if we haven't got some rigorous mathematical underpinnings in all the cases we would like. So in that sense, it gives a rigorous, assuming it's correct, definition of these quantum gravities by quantum gravities, quantum theories of spacetime by the trivial statement that they're just due to something which we can understand rigorously and rigorously in this physics sense rather than the mathematical sense. So these are quantum gravities in ADS. As I said, they don't have to be asymptotic to ADS. But these cases where they're not asymptotic to ADS, they're asymptotic to things even more complicated than ADS, not simpler. They're not asymptotic to flat space or cosmology. So I don't think you'll be happier with that aspect. So again, I'm not going to talk about that more. But there are certain lessons that you immediately learn about these theories of quantum gravity if they are indeed true and therefore defined by this dual quantum field theory. One is that spacetime is not fundamental, it's emergent. The quantum field theory lives in d dimensions. And we know how to write it down. There's no dynamical spacetime at all. Somehow, the physics it encodes, at least in some limits where, for example, in this n equals 4 case, we take a large number of degrees of freedom which we crank up the coupling to make it strongly coupled. Then there's some dual gravitational description that looks roughly like some sort of gravity theory, like Einstein gravity. But if we didn't crank it up to strong coupling and we didn't take it to large n, that quantum theory still exists. It's perfectly fine. It just is that what was the dynamical gravity side presumably is still some 2B string theory. It's just very strongly coupled and very quantum. And it doesn't look anything like gravity. In fact, it obviously doesn't look, it looks just, for example, if you look at this in the perturbative regime where you make the coupling very small, whatever that 10-dimensional 2B string theory is, it looks like a four-dimensional, weakly interacting quantum field theory. So space time, at least in these settings, looks emergent. I mean spacetime. So I mean 10-dimensional spacetime, for example, in this setting. Of course, there's still time and space here, but this is four-dimensional time and space. This is a different time and space to the time and space of the 10 dimensions that this quantum gravity lives in. So they somehow go away. It's almost like, I mean, the reason you need a large number of degrees of freedom in order to recover a gravity theory is sort of obvious. If I've got a four-dimensional gravity theory here, how can it look 10-dimensional? Well, sorry, a four-dimensional quantum field theory, how could it ever look 10-dimensional? It obviously can't unless you give it a load of local degrees of freedom. And then you may be able to encode some higher dimensional-like behavior. That's why it's in that sense only when you crank up the number of degrees of freedom sufficiently that spacetime can emerge. The other obvious lesson is that black hole information well, information, let's say, is not lost. There's this quantum field theory, due to this quantum theory of gravity, within that setting you could form black holes. Maybe we'll talk about that later. You can let them evaporate it's a quantum theory. But the quantum field theory is just a standard unitary quantum field theory. This n equals 4 theory, as I said. You can write down its Lagrangian, apply the usual rules for quantum mechanics. It's a unitary theory. And therefore, information can't be lost. Whatever happens, information is lost. Now, the obvious question is, how is it not lost? And no one can answer that. And no one can answer that yet. Well, in principle, we can answer it. But no one can practically answer it because in the case where you have the gravitational side look like a gravity rather than some highly quantum strain theory, the field theory is strongly coupled. No one, it's OK because no one can answer what happens when you collide to irons at CERN and use QCD to follow the physics. Dynamics in Yang-Mills theory, when it's strongly coupled, is basically an unsolved problem. So it's not that we don't believe that you can collide irons at LHC and see stuff come out. It happens. We think we understand the basic laws of governing it. We just are entirely powerless to calculate. And it's the same sort of thing you would need to do here. Now, there's another consequence which I'm probably not going to talk so much about. Well, in fact, to be honest, I'm not going to talk too much about either of these. They're more motivation. And it's just to say certain strongly-coupled CFTs, a semi-classical gravity description in certain physical regimes. So it's just turning the thing around. It's just saying that when this is strongly-coupled and you fail to be able to calculate with it, maybe you want to put it at finite temperature but at strong coupling and ask, what's its thermodynamics? We can't do that. It's a simple question, but no one can do it because it's strongly-coupled. However, you can re-express the problem as a problem provided you're in this large number of degrees of freedom strong coupling limit. You can re-express it as a gravitational problem. And it turns out finding the thermodynamics of that theory is just the same as calculating the thermodynamic properties of some rather exotic black holes. I mean, not even terribly exotic. Slightly more exotic than the normal ones. Yeah. Well, that's a good question. I mean, certainly we haven't found any examples. And there are some arguments that it should be strongly-coupled. But then I mean, some of these other more exotic theories were precisely attempts to see if one could find cases that are weakly-coupled. So there's no killer argument that says it has to be strongly-coupled. But the caveat here is that the strongly-coupled CFD, this is used a lot. There's been an awful lot of papers written on this sort of stuff trying to understand heavy iron physics or what's called ADS-CMT, ADS-Condense Matter Theory, which is a sort of cute. Just the idea that often in condensed matter settings, you're interested in strongly-coupled systems that, in fact, are conformal field theories or deformations of conformal field theories. So maybe these techniques are useful. And in the heavy iron physics, you're trying to really solve QCD dynamically, which this theory doesn't look a million miles away from QCD. It's a gauge theory with some gauge groups and fermions. It's got some scalars. The fermions and the scalars are in the adjoint, not the fundamental. Sorry. Well, the scalars are there. That's the problem. Shouldn't have scalars. And the fermions are in the adjoint, not the fundamental. And n should be very large. But apart from that, it's basically QCD. However, it turns out that I'm always slightly I mean, I think that all of this stuff is very interesting and allows you to sort of geometries these quantum field theory problems. But one has to bear in mind, just by virtue of the fact that these theories have a gravitational dual description, these are not normal theories. These are rather strange theories. Why should we believe it? That would be your next question. If this is really giving theories of quantum gravity, haven't we always wanted theories of quantum gravity? Well, firstly, these are not theories in asymptotically flat space. And they almost certainly don't look anything like our world. But nonetheless, still, it's a theory of quantum space time that one can write down. This quantum field theory, you can write it down. It's perfectly rigorous in the physics sense. So why should we believe it? That seems like a big deal. Initially, I think it's fair to say there was this physical argument, but many of the bits of evidence were, you might have argued, that there were rather lucky matchings of symmetries and so on. But over the years, I didn't look up how many citations Maldessian's original paper has, but I suspect it's in excess of 20,000. I mean, an awful lot of papers have been written on this. And there's now very, very non-trivial evidence in various directions. So for example, you can match the spectrum of the N equals 4 theory and the dual string theory, which I rubbed out here, the 2B dual string theory in ADS, even at strong coupling using special techniques called integrability. So this N equals 4 super Yang-Mills, because it's got maximal supersymmetry. It's a very beautiful, in some sense, very simple theory. And there's certain statements you can make about it at strong coupling, unlike QCD, which is very tricky to handle. And so there have been some very non-trivial, that's one approach of very non-trivial evidence suggesting that it's doing the right thing.  So when I said we can only deal with a simple harmonic oscillator, it's not quite true. In very nice situations, you can do better. And people have done better, and you get an exact match on the two sides, what you calculate in string theory or maybe just its gravity limit, you can exactly match to what you calculate in the quantum field theory side. Something I've been involved in is there's even numerical lattice work looking at putting the quantum field theories, for example, at finite temperature, looking at the thermodynamics and seeing whether they match the thermodynamics of the gravity. We'll discuss that a bit more later. And again, this direction, it's numerical, so it's not nearly as nice as these, but it's in some sense more non-trivial things that you're trying to match. And again, the evidence seems to be pointing to agreement. There's no evidence that things don't agree, and particularly these are highly non-trivial analytic checks. So if you believe ADS-CFT is incorrect at this point, I would say you have to have a very good reason, a specific reason to say I don't believe it. I don't think you're going to start up your hands and say, it's obviously wrong. If it's wrong, it's wrong in a very peculiar way to pass lots of other checks. I think that's important to say, because there's a lot of, I don't believe in this. I don't think it's a very reasonable stance to now not believe it just outright based on the fact you don't like it. That said, we still don't have a proof. How would we prove it? For example, the n equals 4 case is due to some 2B string theory in ADS. We understand aspects of 2B string theory and perturbation theory, but we don't have a non-perturbative understanding of this theory. So you can't prove it by proving, as you do in math, the left-hand side is equal to the right-hand side, because we don't know what the right-hand side is. If you like this, defines the right-hand side, and then we can check in the limits where we do understand this, that it does agree. But you see the point, the proof would be extremely, extremely complicated. It would either be tantamount to solving this at strong coupling analytically, which we can't even do for QCD. Maybe one day we'll be able to do it, or understanding this non-perturbatively, or both. It's difficult to understand how one could prove this. But that doesn't make it wrong. So in a sense, we're very lucky. It's entirely luck that this descended on us. Yeah. The what, sorry? Oh, I see. These are in various ADS-CFT settings, but not just this case, other ones as well. But always in the context of string, string theories, yeah. But I should also say that most people would, different people have different views, but I think it's entirely reasonable to say whilst ADS-CFT sort of came out of string theory because you can understand if you look at this theory, it's apparently dual-sum string theory, there are almost certainly other examples of ADS-CFT where the dual isn't a string theory, it's something else. So ADS-CFT and its generalizations are probably far more general than string theory. You shouldn't think of it as string theory. It's something more general. It's a more general phenomenon about gravity. I would claim, okay, so that brings me to the end of the sort of ranty discussion. Maybe I should pause briefly for some questions if there are any burning questions.  It's very important. If you buy that this is a quantization of gravity, okay, not just one, probably many, many quantizations of gravity, and you believe that there's some other quantization of gravity, your favorite one, whatever it is you work on, that's just something completely different. And if you want it to be inequivalent to these, then we're done. It's a disaster because then there are, we already, then you will have convinced yourself, well, if you don't dismiss this out of hand, you will have then come to the conclusion that you believe in at least two different inequivalent quantum theories of gravity. What are the chances that we've written down the only two quantum theories of gravity? There's probably many, many different ways to quantize gravity in that case, okay? So even for people who are working on other approaches to quantum gravity, I think it's particularly important. Anyone who works on any approach to quantum gravity should look very carefully at what's happening in the other approaches because as soon as we've got two inequivalent theories of quantum gravity that people basically agree are both consistent, we're doomed. Or rather, we have to rethink how quantum gravity is gonna work. I think, yeah, I was too ranted. No, no, no, I'm not saying that at all. If that was the case, there may be all theories of quantum gravity in some way or equivalent and there's one framework for quantum gravity and there may be just different aspects to it. But it may well be and I think it's probably, I'm a pessimist, I think it may be more likely there are just different ways to quantize gravity, which are not equivalent. They don't, I don't know if you're allowed, I mean there's what you would like and there's what happens and I have no, I'm entirely agnostic. Although many people aren't, it seems. So let me now go to the next section, which is a sort of crash course in QFT or rather maybe I should phrase it, I'm just gonna set some notation for what we're gonna say later. So as in Harvey's nice, Harvey actually has done me a great favor by discussing some of the things I'd like to talk about. We're interested in relativistic quantum fields there is, of course there are non-relativistic ones, but we're interested in relativistic ones. It's natural to then set time and length units so that H bar and C are one and think about things having certain mass dimension. That's all that's left. And so we'll, you know, so we have some fields. Let me think of a field five X, maybe it's a tensor field generally, but let me suppress the indices. And let's think about putting the field on Minkowski, but you could have some other metric if you were thinking about field theory on curved space, but the space is fixed, it's not dynamical, so it's not what I would call a gravitational theory. And then the usual way of thinking about quantum field theory or for those who haven't studied it, fields correspond to particles or rather the excitations of fields, the quantum excitations of fields correspond to physical particles, at least in the simplest theories. So the canonical example of this, of course when you start learning a course on quantum field theory, you always look at scalar fields first, but the canonical example that we're most familiar with is Maxwell theory, which of course you encounter as a field theory to start with and then there's the mysterious photon which we somehow know exists, but the photon is the excitations of, the quantum excitations of the Maxwell field when you think about its quantum dynamics rather than its classical dynamics. So we have an action which is a functional of the fields. It's written, if I'm in D dimensions, it's written in terms of a Lagrangian density and as usual we'll think of the Lagrangian density as being a local functional of the fields. Of course you can write down non-local functionals, but then one has very little control over what theories do and in the usual framework of relativistic quantum field theory we want to sort of understand what's going on. This should be some local functional of the fields just like Harvey was writing down previously. And so EG as we all know for Maxwell you have some field strength squared action and for another example we'll use is the scalar, maybe a massive scalar and I have to, given that I rewrote this several times before realizing it was hopeless, I've changed notation a few times and so any minuses, halves, anything, who knows? So rough structure only, so let me not even try and put them in. So this is the massive scalar and of course they debase some wave equation which you've all seen and the mass dimension of the field meaning just the dimension it would have, how many powers of kilograms it would have if you wrote it out in kilograms in these natural units would be this. Just purely classical grounds. And then when we quantize a field what we really often when you first see a course in quantizing or quantum field theory what you do is you canonically quantize a field meaning you promote the field to an operator that acts on states, vacuum or other states and then you find its conjugate momentum by just in the usual way you would find the conjugate momentum of any configuration variable from an action and then you write down commutation relations between the field and its conjugate momentum which are local because we don't want this nice local quantum field theory and then you just proceed with the consequences and you see this Hilbert space or Fox space of particles come out and so on and so forth but there's a much nicer way which even if you've taken a course in quantum field theory you may not have come across which is the path integral so maybe I should ask how many of you have seen the path integral in quantum field theory? How many of you haven't? Yeah, okay so I will say a few things about the path integral then. So the path integral is an object that looks generally like this. I'm putting an H bar there but really I've set it to one but just to show you that's where it would live. The path integral is a horrific object, okay? It's easy to write down it's very difficult to understand what it means or really make sense of it, okay? And the devil is very much in the details as they say. What it is is basically an averaging procedure where you have some observable so this is some observable by which we mean something we want to measure, some property of the field. Maybe it's just the field itself or the field at one point multiplied by the field at some other point or maybe it's the energy of the field or who knows, something. And in order to measure it what you have to do in quantum field theory and I should say this is entirely equivalent to the usual canonical quantization if that's what you've seen. But one way nice way of thinking about it is the way due to Feynman where you think of integrating your observable over all field configurations. So this is an integral over all field configurations, whatever that means, weighted by the phase given by the action of that field configuration. So it's rather similar in statistical mechanics to the partition function if you've seen that. In fact it's highly analogous. And what this object does is it allows us to calculate the quantum, what's called the expectation value of this observable. So this is the, as I've written it, this is called the VEV, the vacuum expectation value of O, meaning the value O takes in the vacuum of the theory and it's given in terms of objects like this, it's given in terms of this thing I wrote down suitably normalized so that if I don't put anything in I just get one. In one in some sense. So for example, a class of objects are correlation functions. So an object like this where I measure the field at different points and ask how are they correlated? This is sorry, different points in space and time I should say would be calculated from this putting in this functional of the fields into there. And just in principle calculating. Correlation functions are very important objects. They tell us a lot about the dynamics of the quantum field theory. For example, we have what's called the one point function. So that's the sort of state of the field if you like. We've got the two point function which tells us in some sense that you can make precise it tells us how the particle propagates from the space time point X to Y. So the natural particles associated to the field. And then you've got end point functions which like this, which again, it can be made precise tell you about scattering of particles and can in fact be related to these sort of Feynman diagrams that Harvey wrote down earlier and of course you've all seen. Okay, so maybe I should draw some complicated process in between and this would be some six point correlation function, its behavior would enable you to calculate this. One other thing that we'll use later when we define the ADS-CFT correspondence this will be important to us is that we can generate all of these correlation functions from a generating functional taking this object which is some path integral. So it's our normal path integral but we're gonna add in the phase that function J for some operator, a local operator by local operator. I just mean that some local functional of the fields for example, it could be just five X or it could be, there may be tensor indices here as well. So maybe this is the stress tensor which is built from five X and so on. It could be some complicated object but it's some local object. This is a generating functional it's some sort of path integral it's a generating functional for correlation functions in the following sense. If I functionally differentiate it with respect to the source J at all of these different locations I want my correlation function to have arguments at and then set the source to zero. This gives me the correlation function of these operators that whatever operator O it is. And really I should think more generally every single local operator I could write down should have a source I could add a source for it and then I could think of some very complicated object here which was a functional of all the possible sources I could write down and then I could extract any correlation function I liked by appropriately differentiating it. That will be important to us later. You can view this as a trick this object is just a trick to give you correlation functions or more generally you can think of these possibly as external fields for sources. So for example in the case where this local operator is the stress tensor actually the source for it is a two index object I can think of that as a deformation of the metric though deforming by the source in that case would be deforming the metric the field theory lives on because the stress tensor is the variation of the action with respect to a metric perturbation. Or if you know or this could be a source like an external electric field or something. So it can be physical. It's also true that I can take I haven't been at all clear about the boundary conditions that you should consider here. You have to specify some boundary conditions what I've implicitly been assuming is that you just integrate over all possible fields over all of space but there are other boundary conditions you can take and in that case these are not vacuum expectation values this sort of object will calculate different things. A nice example of this is that by a neat trick of analytic continuation if I take physical time to imaginary time and then make imaginary time periodic with period we get this right beta which is h bar over Boltzmann constant times temperature. In fact this this object here Z even without any source Z just Z of one is the partition function the thermal partition function. So there's a very it's just a very neat trick if you want to think what does the field theory do how does it behave at finite temperature the path integral basically after this analytic continuation gives you the partition function of the field theory. Okay and if you can calculate it you've got the partition function you can calculate free energies, entropies, whatever you like. Okay. A few words about space time symmetry because then we're going to talk about conformal field theory and that's all about space time symmetry so again just to say some very obvious things that you all know will have seen. If we put a a relativistic theory on Minkowski space we expect Poincaré invariance which is an invariance under shifting R well we can think of it as an invariance from an active point of view as invariance of shifting R coordinates by some constant R and also Lorentz invariance together translations and Lorentz give you the Poincaré this is some SO1D-1 matrix probably you want to put further restrictions on it like proper orthocrinus so on and then the field will have a scalar field any field should have some should be a representation of this symmetry if we have a relativistic quantum field theory so it has some definite transformation and a scalar field for example just transforms in a simple way as you know and love and we can think of that in terms of we can write we can write that transformation in terms of some generators and some parameters here it would just be the amount you're translating by here it would be this anti-symmetric tensor giving the shifts or the boosts and rotations and then these the generators so these are the parameters if you like and these are the generators and the generators for these you will know we usually call this one P and it's just a derivative and this one is this object here that generates Lorentz transformations at least for a scalar field then you there's an analogous generator for fields that have tensor indices and then there's some intrinsic transformation part as well and then there's the usual algebra of the Poincare group the P's commute let me not write all the indices because you know them I'm sure there's two index contractions here M, M and P have some commutation relation and the M's themselves have some commutation relation amongst themselves and as you all remember there's the Casimir which commutes with everything which is given by the translation generator squared and that's means that representations of the group have definite mass and you know when we talk about particles they can be massless or have definite masses and just one further comment this was you know our discussion on Minkowski but you can promote QFT to a curved space time by just taking whatever action it has in fields making them fields on the new curve space time and then using covariance to constrain the to give you the action on the curved space the only caveat being there may be new curvature couplings that you can add to your action which may or may not be removable in the sense of the last lecture but generically there'll be additional terms that would have vanished in flat space because they go as curvatures but nonetheless on the curved space they can be there and from the point of view of at least effective field theory if they can be there you should include them at least be aware that they could be there few more comments the stress tensor every field theory should have some stress tensor it's defined when you put your theory on a curved space and then functionally differentiate the Lagrangian with respect to the metric it gives you the stress tensor the one point function of the stress tensor as a consequence of the above you can quickly figure out is actually just given in terms of the the one point function of the stress tensor on Minkowski is actually just given by a derivative of this partition function that we had before the two point function of the stress tensor you could go away and calculate it using the rules I told you before and it would tell you about the propagation of energy in your theory from one point to another if you inject some energy you can see how it how it comes out in some other place the sort of quantum field theories that work very nicely are free theories but they're also rather uninteresting free theories have the property at least for bisonic theories they have the well sort of theories you usually come across free theories like the free scaler or Maxwell they have an action that's quadratic or linear field equations the action is quadratic and you can quantize them you can see their particle content they're very they're very nice but they're not very interesting because the particles won't do anything they'll just propagate nicely through space time just to be explicit for the mathematicians amongst the audience just to get a handle of how awful this path integral is let me show you how you would calculate Z this path integral for a quadratic theory so let's let's imagine we've got some field five uh... and let's imagine we can write the action in terms of some differential operator some uh... hyperbolic differential operator d and let's assume there's a nice inner product this is self-adjoint with respect to so the action the action can just be written as the inner product of phi with d and phi and then because it's self-adjoint we expect there's some complete basis so that we can write phi as some sum with some coefficients a n of basis functions u n which are orthonormal with respect to this and then our path integral here which is now looks like this we've now got to make sense of this beast okay so how does how do we make sense of it well typically we do this so we we write the the path integral should be a sum of all field configurations so we think of it as integrating in a product over all coefficients in this decomposition of the field that's certainly one way to define it typical way one defines it this here expanded in these eigen modes then becomes e to the i and i should have said i think i'm assuming my field is real i think i've assumed my field is real the point being that i now get some infinite product of integrals where i have e to the i something quadratic in the variables i'm integrating over just with some coefficients here which are just the eigenvalues and these are integrals that after some suitable analytic continuation i just Gaussian so i can do them and so roughly speaking when i do them i should get something like something like this up to some constants however uh... i mean there are some nasty steps involved like an analytic continuation and the fact that this is an infinite product so you have to think about what that infinite product means physically and what we normally the way we make sense of it is to say we're actually only going to integrate over uh... well there are there are different ways to make sense of this and it's very important to understand in quantum field theory this this is a very severe and deep issue the fact that you've got an infinite number of things you're integrating over and so that the most brutal way to deal with it is to say i'm only going to integrate up to uh... sort of eigenvalues or energies roughly speaking uh... which are less than uh... some amount a cutoff uh... and then i'll i'll calculate all my physical observables and then i'll try and remove my cutoff and see if the theory makes sense roughly speaking and sometimes that works i mean for free theories that's fine the problem comes for interacting theories where you can't just do these integrals anymore and this whole structure becomes much more subtle and this in particular hides sort of terrible infinities that you have to deal with very carefully so it's obviously a rather divergent object and and how you extract information from it is very subtle and interactions take the following form in a relativistic theory so maybe we've got our scalar field and then we add interactions of this form uh... local probably i should put a two there actually no i don't need to put a two there uh... local uh... the action the Lagrangian density should be a local function of the fields and so there are higher powers of fields or maybe derivatives of fields to some n or you know whatever whatever you like they should be local at some point everything's local at a point and in order to this is some dimensionless uh... number uh... but because the fields generally have dimension as we said the dimension of this will be d minus two over two uh... i will need some compensating mass to some power here to make this have dimension d which my Lagrangian density has to have okay and therefore you can immediately see that p plus q here must equal uh... p plus q times d minus two over two must equal okay so if i write down an interaction with some particular power here i have to have a particular number of power of mass dimension in the coupling this is you can deal with this in the sense of treating it as a free theory when this term is small using perturbation theory and you get Feynman diagrams as as Harvey was describing uh... but more generally it's rather complicated and the thing that really makes life complicated is that the coupling will not be small at all energy scales even if it's small at some if p we separate the cases of p irrelevant interactions marginal and relevant relevant and here the interaction is week at low energy and here it's weak a sorry at low energy whoops at high energy and here it's subtle in all these cases there may be some regime where you can use perturbation theory but uh... not overall energy so here for an irrelevant coupling this is of the form uh... where the theory becomes weakly coupled at low energies but if you try and consider high energies even classically you could see here you would run into trouble at high energies the theory will become strongly coupled and almost certainly will become sick in that it will violate it'll start to violate unitarity or you know basic things you want from a nice theory and this is the setting of effective field theory that Harvey was talking about so there is with the relevant couplings effective field there is they're not fundamental they may work at low energies they may describe how physics is but they're only supposed to work at low energies and if you crank up the energy they won't tell you anything sensible these on the other hand are sort of we could call them fundamental in the sense they work up to arbitrarily high energies quantum mechanics is an example of a relevant theory you probably never thought about it like that or maybe you haven't but uh... we don't have to worry you know quantum mechanics is like a quantum field theory just with no space dimensions uh... you have to worry about zero point energy and so on but once you've worried about that you're pretty much done and the reason is because it's of this type and it becomes essentially weakly interacting at high energies so these are nice theories and these are really complicated and these are actually where CFTs will live as we'll see at the beginning of the next lecture but these are very subtle and just because you add an interaction which is dimensionless so p is zero classically doesn't mean necessarily at the quantum level you're not going to develop a dependence on energy scale where does it come from it comes from this definition of the measure here there's always some subtle energy being put in a UV energy being put in to define this theory and it can pop out at you so even if your theory like uh... quantum well like uh... uh... yang meld theory in four dimensions is it is a marginal theory the coupling is dimensionless when you quantize it it can it can uh... turn out that there is some running of the coupling with energy scale so that's sorry that's a rather tedious uh... not tedious uh... it's important uh... I should have said this is like gr this by the way is why quantum theories of gravity is so complicated to write down because whilst you can quantize gravity and looks perfectly fine at low energies we see immediately it's of this effective field theory form and uh... the interactions in it are of this irrelevant kind which means it it's sort of uncontrolled at high energies and so when we quantize it naively we just get sort of rubbish out so next time with all of this we can start to talk about what a CFT is and then hopefully next time I'll tell you what a CFT is and ADS is so that's tomorrow so I'll end it there for today


Kia ora, kouta, katoa. Welcome everyone. I'm David Robinson, gone with it as the name carrot. I'm a pentester at ZX Security in Aotearoa, New Zealand. For those who don't know where Aotearoa is, we're up the top of the map there. I enjoy, a little bit about me, I enjoy aviation and hacking. Queensland and Australia has a couple of great conferences with great hacker content but they're also on the approach path for their local airport so you can sit on the riverbank or on the beach and watch planes come and go. So that's a great combination there. Today we're going to be looking at electronic flight bags in the general aviation cockpit. We're going to go through some example issues and vulnerabilities and security type things and also going to discuss how we can mitigate the issues in each case. I'm going to go through these in more detail as we go through the talk. With the issues I'm going to present and with that quite a high level of the vulnerabilities, discuss more sort of classes and types of issues opposed to diving down into individual vulnerabilities. We're not going to be naming vendors or product names here. We're going to try and keep it general so it's helpful for everyone. We're not just sort of giving specific examples. Hopefully there's something that everyone can take away. Sort of for a bit of a framing of the talks of the issues that I've been going to present. While I've been thinking about discussing pointers to presentation together, I've sort of gone back to the CIA triad and been focusing on their integrity and availability sort of of the systems and how these issues sort of affect those. I will note that a lot of these EFBs and the receivers that are used by EFBs will say, you know, do not use for navigation or safety, et cetera, purposes. We know people are actually going to use them. People are going to put too much trust in them. So where possible, we should try and still make them safe if there are people using them in these situations. So what I'd like people to take out of this talk today is some ideas about how to make more robust and secure systems for their customers, for the pilots using their systems. There's going to be a particular focus on general aviation, the EFBs, but hopefully some of these ideas can apply for other EFB manufacturers and other parts of the aerospace industry. To an IT security professional, most of these issues and problems we're going to discuss will not be new issues. I will be trying to put an aerospace spin and examples to them to help tie them to the aerospace industry. But a lot of these issues, as we've seen industries move into a more digital, more connected world, we've seen a lot of these types of issues, categories, classes of issues come up as they've made this transition into this new digital world. So aerospace is not going to be the first industry to go through all these growing pains, nor is it the last industry that is going to go through this growth of product scope and have to deal with these issues. So what is an EFB in the general aviation context? It's often going to be on a tablet. It's going to have a, it's going to provide the pilot with some combination of flight charts for navigation, airport charts that might provide some attitude and heading reference systems. So your artificial horizon, altitude, speed, direction. It may also provide some situational awareness through ADSB or FLAMN and it will be able to show the pilot what other planes and aircraft are around the plane that they're flying. So this is how a tablet and a receiver and the sensors sort of fit together. So in the middle, we have some sort of receiver unit. These are commonly, you know, and then we'll have the tablet with the EFB and this communication between the receiver and the tablet slash EFB is quite often going to be Wi-Fi. In some cases, it's Bluetooth. And between the receiver and the tablet, the most common protocol used is GDL90 and then some things will also use HTTP or WebSockets. The receiver will has, can take imports from a variety of different sensors. So an import, so we could have GPS satellites as one with ADSB in to help the situational awareness. We may have a gyro and a silisco, sorry, accelerometer that will help with pitch your role, acceleration, deceleration. In addition to the receiver sending information to the EFB or tablet, there may also be a web interface to the receiver where sort of configuration of the receiver can be done. And this normally happens over HTTP. So for my testing, I sort of only use my own tablets, my own receiver, my own hardware, where radio was involved, sort of, I did, you know, I made sure that my signals did stay contained to my test environment. So I used a Faraday cage where possible, I turned the transmit power down. I tried to use non-aviation frequencies. So if I could change the frequency used by the transmitter and the receiver unit, I would use non-aviation frequencies, where possible, I'd directly connect the transmitter to the receiver with a cable and some attenuators to drop. So the receiver want to get burnt out by the additional power. So I was really trying to make sure my system stayed contained to just my test setup. So there's a variety of examples and issues I'm going to go through. And I'm going to go through each one of these in more detail. So first up, we're going to look at heartbreak messages.  The EFB will receive the heartbreak message and use that as an indication that the receiver is up and working as expected. And it can be used to inform the pilot that the receiver is not functioning correctly or the connection between the EFB and the receiver is not working. So as an example, if we start an EFB and the receiver is off, we'll often get some type of red X, some type of no, no connection, no heartbeat, need to know no, no connection established, some sort of error message, it's quite prominent and displayed. Most of them involved a red X, so it's quite clear that the system was not ready for use. So when we did start the receiver and the EFB connected to the receiver, the red X went away and started to display data, so heading, speeds, altitude, in this case. So what happens when the heartbeat stops? This can be because we turned the receiver off. It could also be because a malicious individual is tampering with the data in some way. So what do people expect is going to happen? Do they expect the EFB to inform the pilot, bring back the red X, or do they think everything's just going to continue on hunky-dory, everything's going to be fine, like nothing has happened, it's all okay. So who picked just continue? Well, that is what happened in some cases. You know, when the system is an integrated state, it should tell the pilot that the heartbeat message has just gone away, but I can't trust the receiver anymore. So when we are, you know, for these heartbeat messages, we need to make sure the EFB is using it as an ongoing check. It's not just a check to be used at startup, have all the components started up, it needs to be an ongoing check to ensure that the receiver is still up and running. Because in some of these cases, if you stop the receiver, it will just, the plane will just keep starting out, sorry, the EFB will keep displaying the same altitude, the same speed, or same location, rather than actually showing that there's an error. So it needs to inform the pilot that, you know, the heartbeat has stopped being received. And you could extend this out and start looking at other data. If you had been continuously receiving the altitude and every message to include the altitude, there's suddenly the altitude field was no longer there. That's an indication that again, if something's wrong, and a piece of data that the EFB was expecting or had become used to expecting has disappeared, I make sure to inform the pilot that just because the altitude has disappeared, other data may not be as of the high quality as it was before. So the validity of data. The data that EFBs receive from the receivers may not always be valid. Likewise, the data the receivers receive from the sensors or other impots may also not be correct. Receivers sensors have faults which could result in bad data being sent. There may be corruption happening on the wire. A malicious individual could be injecting or intentionally manipulating the data to make it malicious. So one example is headings. This is a test I did was normally a heading in aviation between 0 and 359 degrees or 1 and 360 degrees, depending on the particular limitation. But what ever happens, the heading should not be more than 360 degrees. So in one case, with a dummy receiver, I sent an EFB and a message where the heading was 450 degrees. It was actually remapped and displayed as 90 degrees, which if you take a step back and you think about the underlying math libraries, which the FB may be using. When you talk about degrees and degree maths, you know, 90 degrees make sense because you've got 450 degrees. So you go 360 for a whole circle, then another 90 degrees 450, you know, in the math domain, when you're dealing with angles, that makes sense. But in the aviation domain, yes, you know, headings are measured as degrees, but we're dealing with headings, not degrees. So in this case, we... Oh, sorry. In this case, you know, in aviation, it's 360 degrees. So if it's more than that, we know it's actually an error. So we take a quick step away from EFBs. And we think about another system where there can be pilots presented with information in the user interface, which informs the pilot that there is a disagreement. There is an issue with a sensor. A recent case has been with 737 MAX and the MCAS. One of the things have come out that the angle of attack indicated the AOA, there's a disagree warning. This wasn't optional extra, the airlines could select. And I understand in the MCAS changes, this warning is now... It's going to be implemented on all 737 MAX aircraft. And what this warning does was displaying that when the sensors no longer send these multiple AOA sensors on an aircraft, when the values differed, it was going to warn the pilot there was a disagreement. So the pilots knew that this piece of data could not be no longer valid. And maybe they need to use other sources to cross reference and understand what the issue was. So it would be good if EFBs, when they start to detect the data may no longer be valid. It's going out of bounds. It's not what they expect. There's a disagreement. It would be good to see if there was a type of warning, if an EFB could provide some sort of warning as the 737 MAX does with the AOA disagreement as one sort of recent example. So this can be when they start to disagree or they go out of bounds, like the headings of the 450 degrees that's out of bounds. It's an indication there should be an error. So what EFBs and receivers should do is they should know what the data should look like under normal operation. They should be checking that their expected data is in the correct range. And when it's out of range, inform the pilots somehow or stop the information being sent so people cannot make the wrong decisions about it. Or you know that you shouldn't be relying on a sensor that is having a fault. Additionally, you look at the trends in the data. You know, if you've been, if the vertical speed results in an altitude change sort of completely out of bounds, maybe you should stop displaying altitude or vertical speed. So for instance, if your attitude suddenly changes 10,000 feet in a few seconds, either there's a fault with the plane and the pilot already knows about it because the plane is dropping out of the sky, or there is a fault with the receiver or sort of the altitude system somehow. So in that case, you should inform the pilot, don't trust their altitude, go trust one of the other instruments that has given you altitude and rely on that. And it may indicate that other sensors involved, which the FB users may also be invalid and the pilot should not rely on them. So with the situational awareness that pilots can have of the aircraft around them, there are some sort of denial of service scenarios. So what some of the FBs do is they have a movie map, which is normally for navigation, showing you where the navigational aids are, showing you your route. But additionally, if the system is capable of ADS-B, it can start showing the other aircraft around and where they are. And with software to find radio, it's easy to transmit ADS-B out. Renderman has and others have discussed this previously at many other conferences. I've learned some other talks at the Aerospace Village on ADS-B Spoofing as well. So this is sort of roughly what it looks like. We've got a movie map, we've got the aircraft, we've got its route and we've got some other planes around it. So what I did was I did a situation where we had this plane and I injected a large number of planes around it. So it's going to be a really congested, super congested airspace. Lots of conflicts, lots of potential path crossings, a lot of potential hazards. So when I broadcasted this, I was expecting the AFB to show all the planes that I broadcasted. But in some cases, you know, on the right there in the diagram, I've just popped up. As we'll see, some of the, it doesn't actually necessarily display all the planes. Some of the planes were being dropped off and then missing. It wasn't always the furthest ones away that disappeared. So it wasn't some sort of range thing. Off there's too many planes. I only showed the closest X number of planes. I couldn't really determine a pattern that resulted that of what planes were missing. And also if I replayed the same input data into the system, I wouldn't necessarily get the same output. So without sort of further research and diving into the code, my current hypothesis is that something to do with the way that the timing of the message received and how it packages up X number of planes to send down every half a second or every 10 milliseconds, depending what the, how often it's sent the current list of planes and there's some cut off and buffers. That's my hypothesis. I'm entirely sure it's just saw this pattern of indiscriminate planes being dropped off. And, you know, this ADSB spoofing that can be combined with TKS issues. I think Pentes' partners Ken talked about this back in May. I believe there's also a talk going on during the Sarah Space Village about this and more depth. But what I'll say on this is with ADSB and TKS, and TKS is the traffic collision avoidance system that you can actually help. You can start putting in malicious ADSB targets to help reinforce the pilot. There is actually a real TKS issue. It's not a malicious one because you can make the two sources correlate with each other. So you could have a malicious plane come in and then you'd start to get TKS messages maliciously, which will come in and reinforce that as well. So whilst discussing some TKS, TKS and some instances from what I said has directional antennas. So it can tell where the TKS message is coming from, whether it's to the left or whether it's to the right or up or down and so on. We could do the same with ADSB antennas. So we know where we know where the aircraft should be based on its location in the ADSB message. But we could also look at the direction the signal is coming from to ascertainers. I believe the plane's over there and the signal's coming from over there so that they most probably correspond. But if the plane's over there and the signal's coming from another direction, we know that possibly this is a malicious ADSB message. So if we've got that, we can roll them out. And the ADSB receivers, when they are not going to pass down a full set of aircraft, there's somehow dropping aircraft, it needs to inform the downstream systems, in this case the AFB, that there is an issue and it is operating in a degraded state. Additionally, if the AFB is dropping planes because of some limitation it has, it should also display some indication that planes are missing from the display and do not, you know, do not use this as a situational awareness tool at the moment. So GPS spoofing. From what my test end and from what I've seen, GPS jamming, GPS signal loss sort of handled quite well in the AFBs. There's either, there's normally indicators that there is an ADSB, sorry, a GPS fix, and there is also an example when there is no GPS fix, there's an indication. But when we see malicious GPS signals, this is not the case. So we see, we're a malicious GPS signal, what I'm thinking of is sending a valid GPS signals replicating a set of satellites, which result in the aircraft determining the wrong position. So last year's aviation village, there was a talk on ILS spoofing and which involved, which involved in the result of aircraft potentially having the wrong horizontal offsets or the wrong flight slope and resulting the planes landing in the wrong location. One of the cases here was, oh, GPS and G in this RNAV can be used to cross reference, you know, the glide slope and stuff. So they should be lining up. And if they don't line up, alert the pilot and they can make further decisions about going round and ascertaining what the navigation systems are. But we could also use GPS spoofing to match up with the ILS spoofing. So they're actually going to correspond to this cross check no longer becomes a cross check because they're both incorrect. But they're both they're both incorrect, but they're both showing the same values. So there is no cross check and there's no indication that one of them is wrong. So back in 2017 at DEF CON, I did a talk where I was doing some GPS spoofing research that was mainly based around time and manipulation of time service. But what I did during that thing during that research was looked at how we could go about detecting GPS spoofing. And in that case, I came up with a bit of a proof of the concept tool called GPS snitch. It's not production really. But what it was was looking at some of the GPS parameters, signal parameters. And so it was possible to actually detect when GPS spoofing was happening. So for a start, GPS is the maths is all based around the GPS satellites broadcasting time, very accurate time. And then based on the time offsets between the different satellites, math happens and you get a location. So if we've got a clock on the receiver that has been synced up with the GPS when we first turned it on, you know, the time should stay and sync together. But if we suddenly start GPS spoofing and there is the GPS receiver locks onto this spoof set of signals from a range of spoof satellites, we'll notice the time will jump whether and it might be, you know, 10 10 milliseconds ahead of time, five milliseconds slow. But if we get sudden change in time, that's an indication that a spoofing that's happened, trying to get trying to start up and have a spoofing attack, start with a time with some millisecond accuracy is quite a difficult task. And that's one parameter. There's other parameters. And I found that actually when you looked at more parameters, it was actually you get a much more stable system, particularly if you're leaving it running, you know, dramatically left it running for a day just to see what the natural variations were. And I found that sort of one look at one attribute wasn't really enough. You need to look at a range of attributes. So if you're in a case in an aircraft, if the location changes more than what the current speed allows, that's an indication that, you know, it could be GPS spoofing, because you know your velocity and if it changes, if it's you're outside of that, you've got a sort of a circle of uncertainty, which may be valid. And if you've outside that, that's a potential issue. Also, look at the GPS signal strength. GPS satellites are 20,000 kilometers away, give or take a bit. So the overall strength of GPS signal when it gets down to ground level is very low. And also the so suddenly if we get a GPS signal, suddenly raise in strength a lot, it's possibly an indication that a GPS spoofing attacks happening because the transmitter of the fake GPS satellites is going to be much closer. So matching the signal strength is hard. Additionally, we should have a range of signal strengths because the satellites directly overhead seem they are closer than the ones way out at the horizon. There's they're going to have different signal strengths because the distance traveled and the amount of atmosphere they have to travel through. But suddenly if the range of different civil strengths decreases, that's another indication that malicious GPS transmission is happening because when you're doing the malicious transmission, chances are it's all coming from one transmitter and it's going to have a very similar signal strength. So additionally, like we said back with the TKS and the GPS was again looking at the signal direction. We know where the GPS satellites should be at any given time. We know the plane's current location. So we know in what direction the GPS signal should be coming from for each of the satellites. So if they don't come from expected direction, it's an indication that they might be GPS spoofing or a GPS attack happening. So what we need to do to help mitigate some of this risk in the airfields is they need and the receivers, they need to start monitoring GPS for abnormalities. So like with some of the topics I talked about with the GPS snutch. Sean indicated that when you have no GPS fix, when you detect the spoofing display GPS, no GPS fix because you can no longer trust that as an import, market is no GPS fix or degraded GPS fix or something whatever meets the relevant terminology and accuracy standards. So integrity of the data, the data sort of in the overall system of receivers and AFBs and sensors, it's nearly always happening in clear text. There is an encrypted version of GDL 90, but from what I saw that was not in very common use. My guess is it's a bit of a chicken egg problem. The receiver manufacturers don't see the point in implementing it because none of the receivers, sorry, none of the AFBs are implementing it. The AFB manufacturers don't implement it because none of the receiver manufacturers have implemented it. Additionally, the Wi-Fi is often open and in clear text so anyone with an range can connect to it with no authorised, with no authentication required. And also, there is when you first sort of pair an AFB and a receiver, there's no key material changed. So you can switch out the receiver and the AFB will still keep talking to if it's the same sort of SSID for the Wi-Fi and the same Wi-Fi password, it will just connect fine. No, it sure won't inform the product.  When you get used to their Bluetooth headphones, you need to pair your headphones and your phone and they are connected and they're only going to talk to each other. You know, if you didn't pair them and you didn't have this sort of checking that they're talking to each other, as you're walking down the street list to the music, you'd start hearing the music of everyone walking past you. You know, and we don't want that to happen with our AFB. We want to make sure we're getting the data from our own plane, not our friend's plane. So, you know, if you have a friend who you're doing some formational flying with and because you're friends, you've discussed receivers and you've settled on the same sort of receivers and AFBs because you both like them. You don't want the situation that everyone's left with the default Wi-FIs and the default SSIDs because it just starts working. You don't want the AFB in one aircraft connected to the Wi-Fi in another aircraft around it because that's going to be bad if you're receiving data for a different aircraft. So, what we should look at here is where possible, start making the migration to GDL 90. Have receivers start to say, you know, on the state, we're going to implement it on the state further down the track. We're only going to have GDL 90. Likewise, on the AFB front, make sure, you know, your system is just set up to actually accept it. When you first pair an AFB and a receiver, make sure there is some exchange of key material. Make sure there is a pairing process to couple them together so you can't introduce malicious receiver and have it connect and have the AFB connect with no error messages. Make sure you sign every message so the AFB can trust that it is coming from a receiver that it trusts. If the message is not signed, disregard it and additionally, ensure there's some protection against replay attacks. So, make sure that malicious thing that you can't capture a packet or a series of packets and have and just replay those over and over and over again. Make sure there is some protection that they need to actually keep changing it. It is rolling. So, to find something, is part of my job. I look at devices, systems, hardware and to make sure they are robust, you know, these testing ventures that have got vulnerabilities but also checking that they're robust so even if they're a vulnerable, I don't know about their attack footprint as small as possible. And when I was looking at some of those of the receiver devices I was put this hat on and I was thinking, you know, have they made the attack footprint as small as possible? In some cases, you know, they had made the attack footprint bigger than it could have been. For instance, you know, there was a receiver that had an internal fan and based on the receiver's temperature, it would put the fan speed up and down to keep the system at the optimal temperature. But instead of setting it up to the fan controller, only listen to commands coming from inside the device, they could figure it so it would accept commands from anywhere in the Wi-Fi network. You know, and that's wrong because in case the only thing the fan controller should be receiving commands from is the temperature sensor inside the receiver. So when the temperature sensor notice the temperature raising, it needed to tell the fan to go faster and vice versa. Additionally, there are weak Wi-Fi configs sort of as mentioned before. A lot of them are open. They don't have pre-shared keys. They don't use WPA2, WPA3. You know, using, you know, pre-shared keys, they will put protection on it. So A, it will encrypt most of the data. B, it will mean that only authorized people can connect to that Wi-Fi network. So you won't have malicious and rogue things connected into the Wi-Fi network. Though even with the encryption, even with the WPA, WPA2, there's still some attacks you can do against it which can have downsides. For instance, there's there's management frames which sort of below the data and control the sort of the Wi-Fi connection. And there's a particular message called a deauthentication packet. And what that does, it's set from the access point to the receiver. So the access point in this case would be the receiver device and the wireless client would be the tablet slash EFB. And what the access point says is say, normally you'll say, hey, please disconnect, re-establish because I want you to re-authenticate. So provide me with the password and whatever again. My problem is these messages aren't signed normally so anyone could come along and set the authentication packets. And this results in sort of two possible cases that you get a denial of service because the EFB or tablet will no longer be able to connect it just keeps being told to re-authenticate. So I'll connect, get a packet to say re-authenticate so disconnect, deauthenticate and re-authenticate again. Additionally, it can also be used to steer the device and steer it to a legitimate receiver. There might be a malicious receiver. So what you do is when it's connected to the legitimate receiver, you send the authentication packets when it's connected to the malicious receiver. You don't send the authentication packets so stays connected to that. So the EFB is receiving data from the malicious receiver rather than the receiver that the pilot put there. There's also additional services running. SSH is a common tool used for remote access to a device for configuration. So in essence, production devices don't need SSH enabled at all because they should be in a secure, hardened format. When it's not an active development or active debug, you don't need the access to the device. And even digging into further, its configuration was not in line with best practices for the use case and following hard-dened guides that had basic things like port forwarding turned on, root long-ons turned on. A lot of things that you wouldn't normally see in something you'd see in the default sort of installs just to make it work but hasn't shown the additional hard-dened steps that have gone through. Also, the web config user interface had no password on first use, no password change on first use, no password at all. So even if the Wi-Fi network had a pre-shared care and password, as I mentioned on the previous slide, you should also have a password on the web UI to control the configuration of the receiver. This is so we have security index. So we've got multiple layers of security control. So if one fails, there's still a safety net to stop it stopping the malicious individual getting further into the system. Make sure you have the safety net just to make sure things don't happen. So with this is make sure that, seek advice on how to harden the configuration. Make sure they're attacked for practices as small as possible. If components have hardening guides, make sure those are read and followed through and used to make the devices sort of secure as possible. So password management. So as I just said, web UI's often didn't have passwords on them or didn't prompt for a password change on first use. Going back to the Wi-Fi case that we used a few times now, there's sometimes options to enable a Wi-Fi encryption with WPA2, WPA3 or with pre-shared keys. One when it did this said, came off a message saying, we've securely set up your Wi-Fi, you're gonna need this password now, we're only gonna show you this password the once. If you forget it, you'll need to do a factory reset. So I went through, I factually reset it again, tried setting up the Wi-Fi, came up with the same password. Go across to a different device, also different instance of the same device, gives me the same password. So it seems to have been randomly generated, this password was hard coded, it always gave you the same password to be used for the Wi-Fi.  Additionally, in this case, I found it a little bit worse because the text and the messaging around the password setup gave the user their experience that this was actually gonna be a unique password, it was sort of randomly generated, it was secure, when in actual fact it wasn't. And that's sort of a double, makes it sort of doubly bad because the pilot or the user is not actually being provided with the correct information. So they can't make the correct decisions if they've been provided with the incorrect import information or documentation. So if there is a hard coded password that you're gonna use, say it's a hard coded password and recommend that the users actually set their own password or better yet, just randomly set a random password, give them a QR code that you can scan because most tablets and phones these days can read a QR code with a Wi-Fi password in and set the Wi-Fi up without actually having to type in a long password. You can just scan a QR code. Some of the FB applications had companion websites. You had to use these, you had to sign up some of the times because the FBs would not work without them because they had some sort of subscription functionality that free users saw so much paid users use this or you had to be a paid user, depending on the thing. And this allowed for flight plans or additional up-to-date weather information, sort of other non-free services that it gave you. I did not test any of these companion websites. I only went as far as registering an account on these websites. So when I did this, I noticed there were some issues. In one case, I got emailed a password in ClearText, which to me as a pen tester, often means that passwords are not being correctly stored in the database, the fact that a ClearText password can be sent for email. So that's a sign that incorrect password storage has been used. Additionally, you also saw some that allowed for weak passwords, so passwords that don't meet the standards of what people think is a strong password these days. Additionally, had some that said, oh, don't use any special characters. So putting my pen tester hat on, if I see a password field or some other user input field that says, oh, don't use quote marks, don't use dashes, and don't use the sort of other symbols. To me, that's a red flag that SQL injection might be available. They're trying to predict SQL injection by just not allowing users to use these characters. So if I was testing this for a job at work, if I saw text like, don't use the quote mark, first thing I'm gonna do is test SQL injection on it because chances are that's why that message is there. So password management. OWASP, the Open Web Application Security Project Foundation has a range of great check sheets. And these uncover password storage, password management, authentication, along with a range of other really helpful documentations. They've got stuff around building more secure web apps. Additionally, they've also got stuff on building secure mobile apps, which most AFPs are. So that's a great resource for information. So hopefully these example issues and solutions have helped you.  So, and I hope that as you do future development, as you add features, as you test functionality, that you think, oh, what could a malicious individual do with this? You know, is there a way that they could affect their integrity or their availability of the system with what they could do?  Also, what can the security industry do to help their space industry in this regard? What help would you like from us? You know, would some example test cases of this malicious thinking be useful? So as an FD or receiver manufacturer, you can take these test cases on board and start working them into your systems. Would a test harness that was say a dummy receiver at which talk to EDL 90 instead of, instead of just sending sort of valid helpful EDL 90, also sent malicious payloads and started pushing the boundaries a bit more, would that be something that was helpful that you could set up this test harness for this receiver, connect your FD to it and have it sent all this malicious information to help you if you're testing, help the automation of the testing, would that be something that's helped? So really good to hear some feedback from the industry. But you need from us as well. So thank you. Hopefully it's been useful for everyone. I really love to hear feedback, chat to some people, because if I'm passionate about it, I'm sure there are other people here out there who are interested about it. All that's why you came along and listen to this talk. Thank you very much. See you all later.