Siemens recently decided to add some security features to their PLC and today we have Tobias and Ali and they will be sort of telling us what they managed to find this PLC. They both come from University Bohum. Tobias is a recent acquisition as a doctoral student and Ali is a postdoc so let's give them a hand. Hmm, where is our slide? I have a presentation mode. We are. Let's say it again. Yes. Okay, welcome to our talk, a deep dive into unconstrained code execution in Siemens S7 PLCs. My name is Ali Abassi and as mentioned before, I'm a postdoc at chair of system security at Royal University Bohum and here is my colleague. I am Tobias or Toby. I'm very glad to be here. It's my fifth time at the Congress and now finally able to give back in a way so I'm very excited about that. So let's get into it. So first about the plan of the talk. We want to give you a little bit of a background of what PLCs, which is program logic controllers, are all about. Why we might want to use them in what kind of setting and then we want to go into the specifics of PLCs in the Siemens case. First we look a bit about at the hardware and then at the software afterwards and the different findings that we had. At the end we will show a demonstration of what we were able to achieve and conclude with some remarks. So first of all process automation. So we all know it or maybe we do it ourselves or we know somebody who does it. We put in some devices in our smart home if we call it smart already and we try to automate different targets or different things to make our lives easier. Things like turning up and down the heat. We might not want to do that our own. We might not want to overheat or under heat and what we do is basically have some sensory systems inside our homes as well as some devices that interact with those sensors. In this case we might have a thermostat and a heater and we want to adjust our temperature based on the thermostat. There are pretty simplistic solutions like this for smart home but what we do if we have very complex control loops for example. Here we can see on the left bottom corner a pretty complex looking picture.  A human machine interface which is basically an aggregation of all the information of things that go on in a factory for example. We need different sensors in this factory and we need to steer different motors and stuff like this. So we need things in the middle to kind of control all of this and we do this using PLCs. Here we can see a setup how it could look like. So basically have a set of inputs as we talked about in a set of outputs and we have some logic going on in the middle and what we typically deploy is a PLC, a programmable logic controller and some logic in the middle. There are different technologies that can be used for example structure text or letter logic which gets downloaded onto the PLC and then which steers outputs based on the inputs that gets. Here we can see some applications of this kind of thing. For example a chemical power plant, chemical plant, an electric grid or some manufacturing. All of those components are pretty critical to the workings even either we see it in our everyday lives and sometimes we don't really see it but they are in the steering everything in the background and we really don't want those systems to get compromised. For example if you went on to Google and looked something about disasters and chemical plants you could see some meditate on plants just because of some mis malfunction in a system or so and we really don't want this to happen neither on an accidental but also not in malicious spaces and this is why we want to secure all the processes going on in factories and the like. We've seen some of the recent attacks so it started kind of in 1999 with the first initial reconnaissance base mainly and then we had some more advanced attacks in 2010 for example where we saw Stuxnet which was very much a really intricate operation if you think about it on a technical level what all went into it what different skill sets were involved it's pretty impressive and then in the more recent time we had some issues in the Ukrainian power grid which in 2015 and 16 just before Christmas some lights went out for quite a while in some cities there so quite a bit of impact. So to give you a bit of impact background on Siemens PLC is here when it comes to market shares we can see that together with Rockwell Automation Siemens actually has more than 50% market share in the market and obviously if we tackle some devices that introduce some security it would be interesting to look at those with the biggest market share this is what we did here in the Siemens case. Here we can see the actual PLC that we will focus on in this talk which is the Siemens S7 1200 PLC it's one of the smaller PLCs not quite the smallest there is the logo as well which is more of a playing around example but this is the one that it's still pretty accessible to researchers in terms of costs so it's like 250 for the PLC then if you need a power supply it can add the same so as long as you don't break too many spoiler we broke quite some or you don't drop them or something like this then you're pretty fine so you can kind of get the resources to play with those devices. We have different applications and we talked about them before so here is what an unboxing of a Siemens S7 1200 PLC would look like we have the top view here in the left picture it's only one of different PCBs which are layered onto each other in this case but the real magic goes on in the top PCB which is the green one that we see here. Looking at it a bit more in more detail we have the top view on the left side which shows the different components that really make the PLC tick for example the ARM CPU that we have or different input outputs that we can connect to PLC as we talked about before which they need in order to steer different parts of the system and then we have the flash chip on the top side as well which is a bigger flash chip holding the firmware of the actual PCs which we will talk about a bit more in detail later. On the flip side we have on the right picture the bottom side of the first layer PCB and as we can see here this is where the bootloader chip resides which is an SPI flash chip of four megabytes holding the code of the Siemens PLC bootloader. Here we wanted to have a detailed view on what the actual processing unit inside this board actually looks like and what you can do if you want to really want to find out you can do some decapping and this is what we see here the result of this we can see that at the core of it is it's a Reneza's ARM Cortex R4 from 2010 and if you afterwards are more juggling with the software side of things you may also want to find out the actual revision number what it supports inside the ARM standard and what you can do there is use a special instruction which resides in the ARM instruction set and you can decode the different bits on it which we did here which you can see here for reference. So if you really want to know what's going on you can take apart those bits and make sure you're actually working with the hardware that you expect to be working with. So here's where we come to the memory part of the hardware and this is where I leave you over to Ali. Thanks. Now that the Tobias like unboxed the PLC for us now I'm going to talk about quirks and features in the PLC. So as mentioned before it's a Cortex R4 revision 3 it's a big endian instruction set and it's also only have MPU so there is no virtual memory basically. There are multiple RAM sizes depending on which year you bought it or which variant of the 71200 you buy and also multiple SPI flash and multiple different types of NAND flashes. The most significant one difference is like in the RAM which sometimes they use Wingbone and sometimes they use micro technologies recently at least micro technologies RAM. It's a LPDDR1 RAM. We do expect the SPI flash for bootloader so it's a again depending on the variance between 1 to 4 megabytes SPI flash. It contains different banks of each size 512 puabytes and basically what the bootloader does is that beside the typical actions of the bootloader which is like configuring the hardware is like verifying the integrity of the framework before it being loaded. So we actually did some x-ray tomography of the PLC. So it's basically 3D so the PCB is basically rotating here because we wanted to also do some hardware reverse engineering part and somebody in university had something so we didn't have to go to our dentist for x-ray. So here is like a quick 15 minutes x-ray which is not that good but once you go in deep eventually what you will have is like this and you can actually just it's like a software animation you can go inside PCB and see all layers it's like amazing. So it's a full PCB layer and so beside VCC and GND there are two layers of PCB connection basically. So let's look at the startup process. Again startup as usual some hardware configuration happens. So vector interrupt controller for example like lots of these handlers for different modes in ARM and then CRC checks some of the bootloader itself which is easily bypassable because you can just overwrite the CRC. Then the bootloader especially in the 2017-2018 variant of the Siemens PLC allows you to overwrite the SPI flash. And also eventually check the CRC check some of the framework before basically loading it. The size of the bootloader itself is like 128 kilowatts isn't really even less than that because half of it is just like 0xFF. Siemens multiple times changed they are different version I think in two years we saw like three variants or four variants of the bootloader so it was evolving it was not something that's everybody forgotten about it. So generally as mentioned so you have the first stage of hardware initialization and then basically bringing the bootloader to the RAM and basically checking the bootloader CRC checks some so make sure that it's not manipulated which again easily can be bypassed. And then a second stage of the hardware initialization happens and then at this moment it waits for a specific command for half a second and if you receive this command it goes to another mode which we will discuss later. Otherwise it basically prepares some CRC check some table for the framework and then it tries to load the framework and then eventually it just removes the memory barrier the SWI instruction those who knows about ARM and basically map the framework to the memory. So the name of the operating system it was not mentioned before it's Adonis. We know it from different ways actually so first in the references in the framework we see lots of references to Adonis but that was not enough for us so we actually looked around to see if like there is any reference to it and well LinkedIn is one good open source like reference and here is like one employee actually talk about a Siemens developer who talk about like working in Adonis. I don't know why he put Windows and Linux beside on this but he says that like he work on this and so it was not enough for us so maybe it's some OS we don't know and we look again further and further and we find this thing which was the best indicator. So a Siemens developer engineer mentioned that he worked on a kernel software development for Adonis real time operating system which is good sign for us it means that we are right. So now that we know about the naming and I'm sure about that let's look at the components so it's actually a start in basically 440 and basically then initializing the kernel and then lots of routines for initializing different components of the operating system. I don't think Siemens actually generalize it in this way we don't have such thing in the framework but we actually did it like that so we generalize it to two groups some of them are core services like Adonis real time operating system services and some of them are related to the like automation part so those people who are like in the automation part like the writing ladder logic and stuff like that those commands on function codes which are available in Siemens they actually know this are like more automated related services. So you have Profinet AWP or automated web programming MC7 Git parser basically for their ladder logic or different kind of SD like basically their own Git compiler inside the PLC and you also have the OMS this configuration system which is very related again to the automation part core part of the automation system and of course alarm central IO and stuff like that related to automation. In the operating system part so lots of this usual thing so file system so PDCFS which Tobias talks later about it the TCP IP stack some seed CC++ libraries which is not from Siemens it's from Dinkonware and mini web web server MWSL parser or mini web scripting language parser and lots of different sub components which is usual in operating system like any operating system we can find them. Also there are some references to core site I don't know how many of you know core site or how much you work on ARM but basically core site is something similar to Intel process tracing or Intel PT for tracing applications and can be used for getting code coverage for example and the hardware part is very well documented by Thomas Weber in this year it's not yet ended this year so this year black at Asia but I have to warn you because I received some emails some people ask about that if you connect to it the PLC have some anti debugging feature which detects that it's being debugged via JTAG and overwrite the Nand flash with random stuff so you break the PLC so just connect it to your own risk next is let's look at the core site just a quickly core site basically have like before I go here I have to mention that Ralph Philip also have a good talk in zero nights about core site tracing so I would recommend you guys go look at that as well but generally core site have like three major parts or components sources links and sinks and sinks is basically the part which you actually get the tracing formation and sources are the part which you tell is a feature in the CPU which you ask what kind of sources you want to get the data from and then links basically convert these sources I have to mention that like lots it's very useful for fuzzing as well too I guess some people very few but some people are working on that things on coverage guided fuzzing via core site arm core site as always possible similar implementation is happened in Intel PT for example KFL Vene FL or home files so sources basically they have like three different components SDM PTM ETM ETM version for is the latest version of it and basically you have also links which connects different sources to different like different or single sources to different or single basically sinks and then you have funnels for core sites sorry sinks sorry you have sinks which is basically different kinds so there are some integrated to the CPU which is four kilobytes ring buffer SRAM or you have like system memory or even TPI you or just for example JTAG DP port high speed JTAG port so now that's a clear thing like the core site we actually create a seven for existence of core site and as you can see like in the software part is already implemented so they actually have some references in their software that they are utilizing or configuring the core site in their PLCs and basically we can see that the ETM version is not the latest version it is ETM version three now that I talked about core site Toby can talk about very much so let's get to something that I'm very much more familiar with and feels easier for me to handle it's firmware dumps or software in general but firmware dumps I think it's close as you can get to what I like when talking to a PLC or trying to understand a PLC so in the Siemens case we have a 13 megabytes binary and at the beginning it's not as many but as if you twiddle around with it a bit and apply some Ida Python functions and stuff like this you can get to like 84,000 functions which it's not something you want to really look at everything manually also like 84,000 function firmware image doesn't really get the sexiest firmware on planet I guess so but this is what I what I looked at and what we will look at a bit more in the next couple of minutes or so as you can see we have different names up there we have one name which is called some gets some max size so this is my internal way of saying I don't really have an idea of what's really going on in this function but we can also see some more meaningful functions so we understood some parts a bit more some parts a bit less but I gave it a cursory look at in most places so now let's get into a lot of address related stuff so we extracted a lot of details which are very interesting if you start looking at firmware code and I will explain along the way why they might be interesting so first of all what you have to know is that Cortex R4 gives you bank registers this is basically a feature that's implemented to lower software overhead and allow more seamless mode switches for the internal CPU and what we get is banked stacks per execution mode so if we want to know what is kind of going on in the state of the firmware at a given point we may want to look at the different stacks of the different modes at any given point and this is the addresses that we extracted for this and you could use that if you as a starting point if you started reverse engineering things like that now we will have some address some tables with addresses and the first one is RAM mappings which show you what's kind of functionality or what you might expect when looking at firmware code which is interfacing with different parts of memory so if you initially go and look at some arm code you may just see a random access to someplace in memory and you may want to know what it's actually doing and it looks very uneventful if it's just an address and it gets gets curate and you don't really know what's going on so for example if you looked at an address within the text section you would expect there to reside code if you wanted to see some global static data you want to you would want to look at the data or the bss section and then finally if you wanted to look at heap memory look how chunks are set up there you would look in the uninitialized section and it goes on like this for different sections another very interesting thing to look at if you try to reverse engineer firmware images is that you kind of want to know what the hardware is that a given piece of code is is interfacing with and then this case we dumped some regions or reverse engineered what some regions are for what is called memory map dial and the way arm is talking to firmware is basically to cure your magic value inside the address base and then it gets something back which is not at all what it's has been written there before so it's basically an address which gets wired through to the peripheral the hardware peripheral on the same system on a chip and here we can see that we have different hardware peripherals residing on it for example we can talk to the Siemens PLC via different serial protocols and those protocols might be SPI or I squared C and we have on the left side kind of in the middle top what top part of it have some region pertaining to that and then if you saw some other code talking to timers for example you would know you have your entailment at the moment or like in the scheduler or whatever it would be finally we have some MPU configurations which are memory protection unit configurations as Ali introduced earlier so what we can see is that Siemens is actually applying some of those configurations to protect parts of memory what we can see for example is where whenever the Xn so the execute never bit is set code is not to be executed within this address region or we have a read-only region we really don't want to have it over it so it's interesting that they started playing this in practice here we can see what actually happens when the firmware itself puts up so it turns out the firmware doesn't really want to depend too much on what the bootloader did probably it's different teams doing different things and to keep this interface as small as possible they kind of redo some of the stuff that the bootloader code also does it sets up the vector table for handling interrupts and similar things like that then if we get past this initial stage we actually want to boot the adonis kernel which Ali talked about before so first of all there is an array of function pointers that gets called one for like every piece of functionality that we saw on this overview of the different components of adonis so if you wanted to look at what kind of components are there or functional components are there this is a very interesting list of functions function endless to to examine and also sets up some instrument structures and stuff like this one and typical operating system would have to set up so now we look at more of the different components of adonis first one is the file system so PLC's part of the specifications sometimes it's how resilient is it against temperatures how low of a temperature can I have this PLC reside in without losing functionality and in this case what they also want to provide is some safety against interrupts in power supply so they developed a proprietary file system which is called powered on consistency file system which they implement in the firmware and we can also see one of the one of the work experience entries of one of the previous Siemens employees who stated that you or she worked on this file system now we have another part very critical part of the functionality of course we want to talk to the PLC it wants to talk to us and and one of the ways is obviously TCP IP and this is to expose the web server for example in different other components and in this case it turns out that Siemens doesn't implement their own which probably is a good idea not to do that they use interneach technologies the TCP IP stack and version 3.1 if you are good at googling you can find some source code and you can actually map this to the firmware and how it works so it could give you some wrapper functions something like creating sockets and stuff like this and I could make it easier for you to find those in the firmware image we also have one of the very critical components of each firmware is updates if it allows an update and the Siemens PLC allows updates there are different modes one of the modes is just you drag and drop a UPD file and update file to the web server and it will start checking firmware integrity and signatures and so on and the second way is doing it via an SD card which has a great total of 24 megabytes and that's for the low price of 250 euros you can can get it thank you cannot really beat that that ratio if you actually decompress this kind of UPD file you you get another representation of it in memory and we did some reverse engineering on that and we have different fields I'm not sure if you can see them now but you can expect what it is it's different offsets into the actual binary file it's an entry point into the firmware magic header to make sure something is not too screwed up and CRC over the whole thing for example we also extracted some of the addresses inside the firmware image which help you find a first foothold into what the logic is dealing with and give you some addresses for you to refer this to later the next component that we want to touch on is mini web which is the web server it kind of exposes to you the different internal parts of the PLC and what the state of different GPIOs general purpose input outputs is the inputs and the outputs and what the health of the PLC is itself and the way that it exposes this is using the MWSL language meaning web scripting language it's as we will see on the next over the next slide I will talk to about that in a little bit more detail we have a it be started as a service as well from one of the function handlers of the adornation initialization functions that I refer to a little bit before so now let's get to some undocumented HTTP handlers which I think are very interesting I think my favorites are lily lily lily and lolololo and if you if you put those together in a clever way maybe somebody is musically gifted and can make a song out of it I would be very interested to to hear that so now let's get to the MWSL the mini web scripting language so it basically exposes the internal functionality by allowing you to inject into an HTML page via templating different configuration parameters and stuff like this for example as we can see here on the top right corner you can see the CPU load of the system at a given time it doesn't really seem to perform any output encoding so it kind of trusting what comes out so there may be clever ways to to do some web related trickery with this and also the parsing of this tokenization is kind of complex I looked into it a bit and this implementation could also be interesting to look at but we will get to this kind those kinds of aspects a bit later with this we want to get to our actual findings and talk about those a bit more and this is where Ali will take over thanks Toby so so now we talk about the capability capabilities which exist in the bootloader which allows us to have unconstrained code execution so basically this feature is available in the UART so you need physical access to the device but once you have this physical access you can basically as Toby has later described we can actually bypass the security ecosystem we developed by Siemens in their product so you need UART access as it's documented here you have TXRX and GND in the PLC and the UART actually in previous research was documented as well every address which I am talking about here or mentioned in this presentation are for bootloader version 421 as I mentioned earlier Siemens actively modified the bootloader so I think in two years we saw like two three modifications or different versions of the bootloader coming up so this exactly is based on that half a second waiting for a specific command after a second hardware configuration happens it applies to Siemens S71200 including C plus and S7200 smart actually somebody from Kaspersky ICS security mentioned it we didn't know even about it we just knew we just investigated S71200 but Siemens later updated the advisory that it also applies to other products as well so let's talk about this is special access feature so as we mentioned one of the thing bootloader does is actually initialize the hardware after this hardware is basically copy some of the contents of the bootloader itself to a memory segment called IRAM basically and then PLC basically waits half a second for a specific command and once it's received it's a specific command it responds with a specific string and it's all happening over the UART so it's if you send a magical string MFGT sorry for my broken German but probably it means mid-fondition großen I hope I did it right and then the PLC responds with dash CPU and says that now you are in this a special access mode I am waiting for your commands and and this address is also available at 0x EDF-8 in the bootloader so here is a decoding of our clients which we release later next year actually which you see that 2d435055 which is dash CPU response from the from the PLC so now we are in it and also we also added some extra thing about your packet format somebody asked before so once you send this command you get lots of functionalities here in this presentation we call them handlers and basically there are something we call primarily handler is like 128 entries and there are some three other separated handlers which are like 0x80 UART configuration and buy so in the primarily handler list there are lots of things so if you go back to the two previous slides I got the the framework version here for 2.3 and basically what is happening is that basically it's this command here get bootloader version we are just requesting the via a special access feature to tell us what is the bootloader version and also you can do lots of low level diagnostic functionalities happening there also some functionalities related to framework update happening there which bypasses the usual cryptographic verification of the framework and doesn't need that so let's look at them because for this work which we are talking about we actually primarily only use two of the handlers so we don't use we don't look at like or we don't discuss now all others 128 handlers which exist in the PLC so it works one of the handlers the interesting one for us was handlers 0x80 which mentioned here update not function so basically what it does is that it it lets you allow you to write to a specific part of the memory IRAM which previously copied some contact content of the bootloader so basically you send this handler after this handshake you have to do this MFGT1 and then CPU and then basically you are going to send this handler and then it basically checks because each handler might have different requirements check number of arguments for example and then you are in this update function and then you have to provide target ID because there are four sub functionality available once you enter this mode and some of them are like for IRAM for SPI or IOC or for flash and then for each of them you have to choose what kind of operation you want to do you want to config read write or check and so you can do all of these things so you can read and write to the IRAM basically this is a function handler x0x80 next is a primary handler like 0x1c this is listed in this handler list here so so basically it allows you to call functions so they basically this function are listed in the IRAM and basically what you do is that you send this handshake you are and you are in this basically this 0x1c handler and then you can call the ID number of the handlers which you want to use so here you have like lots of handler available for 0x1c so the question is what we can do with it and before I ask Tobias want to ask anybody here any idea trace somebody said trace I don't know what that means but okay you mean with jade with the core side no we are not going to use that so let's ask Tobias what he can do yeah so looking at it dynamically and seeing what it does with the memory is I guess a good idea in general if you if like static reverse engineering doesn't give you anything in this case we looked through different or I looked through different of those functions and try to see what can I do with it so the base of where I started looking for this special access feature was basically that I saw there is too much in this code going on I kind of feel like I understood what it should be doing the bootloader what should what it should be doing but it seemed just to be too much and the way we can combine those two functions is basically to in recap use this 0x1c handler which gives us control over what kind of secondary list functions are to be called which as we saw before is copied during the the boot up process to a position in IRM from extra read-only memory and this exposes this function handler table to anything that can write to IRM and as we learned before the 0x80 handler is able to in a limited capacity do just that and here we can see what we can try to do with this so we use in the first stage the 0x80 handler to write to IRM we can actually inject another function pointer together with some configuration values that allows us to pass different checks about argument sizes and stuff like this we can inject this as an entry into this table and we can also write to this table a payload which we can use as a shellcode and then in the second stage we can use this previously injected index that we specified just as a trigger to call our own payload so now we have code execution in the context of the bootloader so which is as privileged as we can get at that point and we can see what we can play around with this and as a little summary is that we chain all this together and we get code execution and with Ali's words with this technology we're going to rocket the PLC and before we go into what this actually allows us to do is a little word about the stage of payload so I wrote this this chain of different invocations and it turns out that this right to IRM is somehow a bit very slow in the first place but then also error prone so the device can just error out I'm not quite sure what this pertains to but maybe interesting to know from the Siemens engineer but it basically led to me having to inject a little encoded payload which just has a little subset of bytes which gives us an interface to do to perform more performant reads and writes with an arbitrary right primitive and then use this to inject second stage payloads and this is what we want to demonstrate here thanks so now we will have our demo four demos actually so first one is actually just seeing the communication basically sending this request and getting a response and basically sending the stager payload so in the op is the raw you are communication don't worry it's getting zoomed later and the down is like our client which actually talking with a PLC and sending these commands so we are just running our UART and here is we are sending our command and if you look at in the op you see that dash CPU signal came from the PLC and now we are sending our stager and a stager just send us just one acknowledgement so we know that's a stager run successfully this is for framework version bootloader version 421 basically so now we are going to do something else we are going to actually dump the framework from a running PLC and compare it with the framework downloaded from Siemens website so first we are going to actually unpack the framework downloaded from Siemens website because it's a compressed with LZP3 so that's what we are going to do I hope oh no we are actually setting up our SSL connection first so SSL port for our port forwarding SSH port forwarding before and we are just checking that the PLC is running properly so like this is not a broken PLC or something like that we wrote something so we just make sure that the web server is opening we open the web server yeah it's opens good and also try to log in to the web server of the PLC just again make sure that the PLC is functional so also enter the password I guess everybody can guess it and and then so you see that we log in eventually and in the left side you see all the like functionalities which loads related to the PLC so it's a working running functional PLC and so now you are going to decompress the framework downloaded from Siemens websites after checking for export license and stuff so they want to make sure that people from Iran and North Korea don't get it I'm from Iran so here we have the unpacked framework but because the framework is very large as Toby has mentioned earlier what we are going to do is that we are just going to export 256 kilowatts of the framework from some part of the web server and in the IDA so you have to set the big NDNS for the CPU and also rebase the framework so as you can see here is no function yet but once you rebase it you have all the functions as well and yeah so then we got just go and export 256 kilowatts from the framework so we specifically slow down the UART because we want to make sure that we don't do it too fast to overflow the buffer which we have internally in the PLC so so here for example in this address 691 e28 we are going to export 256 kilowatts this is from the framework Siemens framework right so you just export it so yeah so it's now called FW 0x 691 e28 in the folder out so now we are done with this part we are going to dump the same address in the PLC so from a running PLC I have to mention again so this is the top part is basically raw UART and this is basically our client part and we are dumping it we are cold boot style attack so we are basically resetting the PLC and before it overwrite the RAM we are basically dumping the contents of the RAM so this is the address 691 e this was the same address basically and we are dumping 256 kilowatts and here we send FGT1 basically and then got the dash CPU and then the rest of the stage and stuff goes so now basically we are sending packet and then eventually get received so basically got all the payload like dumped in mem dump 691 e28 basically so this is from the RAM of the PLC this is not any more from Siemens websites we are just SCP to our own machine and then compare it so now we have the mem dump and original framework 256 kilowatt each and then we are going to compare them with each other and as you can see should look here like you have like 100% match meaning that it's exactly the same framework which is available in Siemens website we dumped it directly from the Siemens PLC memory using this a special access feature so let's do another one so this time we are want to show that unconstrained code execution in just a very basic way so we are actually just writing a custom payload to the PLC and gets a hello or greetings from the PLC so basically we just ask the PLC to send us this message all the time so again so we are sending our custom payload right here it's a hello loop and basically the PLC just sending this loop for us so all of these things again are for bootloader 421 we have to readjust certain things because Siemens I think they updated again the bootloader in the recent 2019 December which we bought new PLC again once again and now here we get a response that's the PLC is sending basically to us these are raw data which PLC is keep sending to us that's showing that we are receiving this but maybe this was too basic these are the raw data which we are getting from the PLC let's actually do it a little more complex show something that is not from us so let's play a game called tic-tac-toe inside the PLC and I guess if you don't know this is how tic-tac-toe is like this is I'm playing and I just draw with the Google so now we are again are going to send our custom payload but this time we are just use partial quotes from somebody else from the Internet and just upload compile it and then upload it to the PLC obviously you have to readjust lots of things there but so we are sending our payload including a stager and these are the raw data again these are our client and eventually you will see a tic-tac-toe interface which you have to enter so player one is actually playing with the X and player two is playing with like zero so you see when you position which you choose like you have X X and hopefully player one wins yes so that was it so that was a demo obviously like there are lots of other ideas which we can work on on injecting other custom code using this special access functionality we are still working on this like lots of other things on Siemens we are sorry Siemens we are just working on this but there are more to come but in the meantime there are some ideas for other people in case they are looking into this and want to investigate security of Siemens PLCs so using this a special access entry you can do some odd certain things so for example you can use this special access functionality to write to the framework as we mentioned this functionality is available and it doesn't require those cryptographic signature which normally during update process of the framework available so you can just bypass that and it's just CRC checks up so what you can do is that for example adding entry to otherness like initialization routine which is available and then or also you can do it before otherness initialization routine which we call internal internal th initial another one which we can do if you remember Tobias talked about some undocumented there and lots of creativity and creating music really low so what one person can do is like basically adding a specific handler or overwriting exists handler and what it makes actually is like something like tritone I don't know if anybody know her about tritone it's a malware which were attacking petrographic in Saudi Arabia so they were trying to do it in a TCP but attacker here can maybe do it in HTTP example and just listen and waiting for commands and also other alternatives like patching one of the jump tables in the AWP handlers which can be also used for process a specific attack so what else is out there so what we looked is we looked at attack surface in the Siemens S7 PLC's there are like for from perspective of local privilege attacks what we can what we looked was bootloader we are still working on hardware attacks and some hardware software attacks on the edge it's a still ongoing work which we don't obviously discuss now also interesting thing I think if somebody who is interested in security of PLC's and especially internals I'm not talking about like this general segregation of network and stuff like that in ICS I'm talking about more advanced low-level stuff we think like MWSL is an interesting target there are there probably are some like box in their implementations also with respect to file system parsing and framework signing there are probably some stuff and also MC7G parser basically which they have from a privilege escalation perspective and also from remote code execution perspective both mini web web server and also any network access of accessible services which they have might be interesting we are actually also looking at this part right now so as a conclusion PLC's are becoming more complex that's true because they are actually providing more and more features and because of this more complexity there will be more box we can see for example in the MWSL which we are looking at now also vendors try to basically more make it more complex they have basically some anti debugging which we discussed in Siemens PLC's but also they have for example frame rate integrity verification so the signed frame rate like like upload to the PLC and stuff like that so they are making it more complex but what we have to know is that if in their like threat model which like lots of people make or this security ecosystem which they built if they have a feature which undermines the same security ecosystem which they designed I mean they I think it's obvious that they have to remove like with the case of bootloader case in the special access feature it is one of the good examples so and of course customers also have to know because if they have such which and they need it as long as customers know it's fine but when they don't they can't risk calculate this risk in their strategy or in this threat model which they have so and also they have to think about we think about security via obscurity maybe they allow us for example as researchers who access their devices better and easier to investigate it more we are still doing it but it's just taking longer and I believe that there are lots of saying more to be done on like PLCs and Siemens will need will not be the last one which we are working on so we have to thank some people Torsten Holtz our supervisor is not here Thomas Alexander Marina Lucian Nikita and Robin for their help in their work and now we are going to answer questions thank you so yeah feel free to line up for the microphones or write your questions in the Elisa room there you go hello yeah so there's one question from the internet did you check the MC7 parser if yes did you find any hidden unknown machine 7 instruction on it or something so just is it recorded or I have to repeat it again so they ask that if we check the MC7 parser okay so it's fine so we didn't like truly investigate at MC7 parser but we are working on it right now hello how were you able to find the MFG security password that's a very long story first of all like we had we had it in front of us for a long long time until Siemens introduced this anti debugging feature and after that like we had to find other ways other means to find it to find like similar function like similar ways that allow us because one thing which we then discuss here is that we didn't tell you about how we for example executed that instruction before in the PLC it was involved some works which we received help from some researchers in Netherlands and in France so this is this was something informed by Siemens in 2013 I think they knew about it but until 2016 they patched it and then it became out like basically they tried to protect their PLCs from this kind of attack it was never published before so we were using it and we don't want to talk about it because the original authors didn't want to talk about it but we replicated what they were that what they were doing and then once we really had to look for other ways like then it opened our eyes that there are some other functionalities as well there so such as for example bootloader but before we before we need it like we never actually looked at this thing so it was like in front of us for like two years and maybe one interesting piece of a background story on this is that we actually in the previous technique that we use we actually overwrote the conditional jump that would lead to this special excess feature being executed with an unconditional jump so we basically cut out 60% of the whole code of the firmware image by accident and then I just because of the hunch that I was talking about before that there is just too much functionality I revisited it and actually realized that it was exactly the spot that we overwrote before and we had to basically replace it and use it for our own sake is there any boot time security other than the CRC check so do are you able to modify the contents of the spy flash and get arbitrary code execution that way as well so depends in which year you are talking about 2017 2016 so we are talking about the same models of the PLC but in 2017 and 2018 no so you could basically just take out SPR flash or write it and that was fine but if you were overwriting and it caused a halt in the CPU core it would again trigger that anti debugging technology which they have with this watchdog basically but from like frame very integrity verification well basically once you write to the framework is written to the NAND flash but it's just a CRC check some but during the update process no there are some cryptographic check but once it's written no there are some problems which there which again it's a still ongoing work and we don't want to discuss about it but nice cat thank you hi thanks for your talk could you elaborate on your communication with the vendor and the timeline yes so first of all we know about this problem for like one year and a half before we reported to the vendor and the primary reason was that we were using it for some other project this actually this result is actually from a site project rather than the main project because the main project is still something else and it's a still ongoing but from the side of that project we had that access and because we were worried that's reporting to the vendor they can't fix it with software update and then do not allow all other CVEs which we find from this other project we didn't want to eventually at 2019 Thomas Weber wanted to talk about his talk on the like basically this JTAG interface with four core site and then we decided to actually talk about it as well but other than that we actually talk in June icing with Siemens and they confirmed that there is this hardware-based special access feature and they are they say that they are going to remove it and that was it we also send them a write-up for them to read so there is one less question from the signal angel over there yeah so there's another question from the internet if tools like flash ROM doesn't have support for unknown SPI flash ROM chip then how do you usually extract firmware if you don't want to decap chip or use SOIC 8 socket can you repeat it again I didn't get the question did you if tools like flash ROM does not have support for unknown SPI flash ROM chip then how do you ex usually extract firmware if you don't want to decap chip and use SOIC 8 socket so first of all we never actually decap the SPI flash so that's we just did it for the CPU and just because we want we know that Siemens re-labeled their PLC so it's not their own CPU is from Renaissance but that's why we did the decapping so story of decapping setting it aside but from other things so basically there are still this functionality this bootloader functionality which actually lets you read the content of the memory so that's one thing you can read obviously you don't even need that thanks to one of my students we now know that actually you don't even need to take out the bootloader chip we basically can just connect directly in the board and dump the the framework Marcelo that's his name for God's sake he's here actually but anyway so you can just directly read it and yeah I don't think the reading part especially some part of it is protected especially in the recent versions which you can't read everything but besides that I don't think there is any hardware now yet I'm sure that they are working on that and we are working also on something to bypass that so okay that was all next talk is gonna be about delivery robots Sasha in 20 minutes so let's give them a hand for attending


Welcome to Toffy TV today. I am joined by one of the most recognisable faces in football. The man who's presenting changed the way we view football. He's still doing it today. He's still absolutely fantastic and I'm delighted to say I'm joined by Richard Keyes. Richard, how are you? I'm privileged to be on Toffy TV, actually. Just a bit. Can we just get something straight right at the very start of this? You're not a Liverpool fan, are you? Let me answer that. Give me a little room to manoeuvre with that. When I worked on Merseyside, my heart was always a goodness. I used to love a night out with the boy, Mick Lyons, George Wood, Andy King. What a great lad. My head told me if I wanted to watch, at the time, successful and enjoyable football, then I had to be at Anfield. There was something quite intoxicating about following Liverpool to one success after another and equally frustrating. I knew how much it meant to the lads over the road to be part of something similar. Eventually, of course, that was the case, not only were they part of something similar, I think they were better during that period of time. I'd moved on from that point. It's a long answer to know if you put me into a corner and said, are you red or blue? Liverpool players of a certain generation, they're a gust when they discover where my heart lies. My head these days as well, to be fair. I'm blue, more than I'm red. Fantastic. But you are a Coventry fan of me and we don't need to know. Everybody knows that. Yes, I am a Coventry fan. There's a burden about being a Coventry fan as well. Although Evertonian should be very grateful to us because I spoke on the morning to Gordon Strachan the year that we came and you needed points, not this time, not us. Strachan was absolutely determined to send you down. He wanted to win that game. Gareth Fowley scores and everybody relaxes. Dion Dublin scores and everybody's very anxious. Coventry came to win that game, I can promise you. The pity is, of course Gordon was not part of the history of our club, but we owed you from the evening you'd beaten Sunderland and Bristol City and ourselves, who needed a point from the game that was taking place at Coventry that night. We got one. By coincidence we kicked off late, 15 minutes later than you did at Everton, so traffic and obviously security reasons. We all knew the score from Goodison that you'd beaten Sunderland 2-0 and they were going down if we finished up in a draw with Bristol City. Managed at the time by Alan Dicks, Jimmy Hill's old number 2, John Sillett was there and as things worked out, that's how it finished. It finished to a piece and that was an extraordinary coincidence, really extraordinary. So we bumped on each other a favour, that was quite good. By the way, you did a big favour that night. Just go on taking you right back to the very start you created. Now you'd started in London, hadn't you been working for an agency in London? Haters, I think. Is that correct? Yes, correct. And then how did the move to Mezyshire come up? To Radio City? When I was working in London, it was in the very early days of local radio. And that's all I ever wanted to do was radio, television. I was the one in the office when if a job came in Ipswich, Radio Wallwell, Birmingham, BRNB, Tony Butler at the time was a big star. If a job came in whereby you were sent to a game to cover it for a particular radio station, I was the man. At this point I was at home when the opportunity came up to go to Liverpool and Mike Calvin, very distinguished fleet street writer now, used to sit opposite to me at 80s, rang me and said, there's a job here. I've just seen in the UK Prescas that you should apply, it's right up your street. Radio City had no idea what that was because it hadn't worked for the ads. The first time I went for interviews, the first time I'd ever been to Liverpool, as I left the city that night, I thought, well, I'll have a look round the place. There must be an open top bus tour in Liverpool. I'll go and ask someone at the station if there is. Well, there wasn't. I don't think this is there. But I got the job. Martin Johnson sports editor, Clive Tildesley was commentator. Martin left within about three months of arriving. He'd replaced Elton who'd gone to Granada TV. I have to tell you, it was four wonderful years. It took me about six months to get the head around it. When I'd settled, working with and for Clive, Clive was a sensational radio commentator. I think the best. I think the best. Others will tell you differently. But for me, the best. He's become a really very, very, very good television commentator. But it was four fantastic years. I did have one major crisis. It was the first one in my life. I was homesick for a while for London. I've been working at the BBC with Bob Wilson on football focus for a while, checking his scripts and facts and things. Bob would ring me every Friday, which was really nice of him. And I remember one weekend saying, Bob, I've got trouble here. I said, I'm supposed to be the expert going on the radio, talking to people about football. And everybody in Liverpool knows more about football than me.  And it was a problem to me because in that 30 second flash, which is the only, we only had that at the time, one minute, half time, get things across and explain. And thank goodness Clive had not been for Clive, I'm not sure I would ever have made it. He was so good, and he was so thoughtful, and he was so caring. I went after four years, I'd worked to a point where I wanted to do all the things he was doing. And there wasn't room for us at Radio City anymore, so I left and went to Manchester. But it was a wonderful four years. Wonderful four years, great people. And I mean, fantastic people around the clubs and both clubs. I mean, characters at both, to this day, are still friends. And I'm extremely proud of having worked on Merseyside. Yeah, I mean, as a football time, obviously you touched on it right at the very beginning. Everton were struggling during your time. We were okay, and then we went into that struggle period. And obviously, Howard come in 81. We had conversely across the park, they were incredible. They were the best team in your era. So we had to cope with that. I guess the answer to this is no, because I don't think anybody saw it coming. But while you were still a city, and Howard brought in the magnificent seven of which only one was magnificent, which is Neville Shalthall. But that wasn't even his signing. Neville was not his signing. Which is bizarre. If you go back to the Gordon Lee's biggest mistake was selling Duncan MacKenzie. Evertonians loved Duncan. He was perfect for Everton. He was everything that they want. He was the new golden vision. He was an entertainer for Gordon Lee. But remember it sold, maybe you don't remember, but for a certain generation they will. Gordon had controversially sold Malcolm MacDonald out of Newcastle when he was there to Arsenal. But the team that Gordon initially had was a good one. Dobson, King, Ross, Bob Lachford was magnificent. Dave Thomas, absolutely sensational. Lyons, Wright, Higgs, Higgins, Darracott, Pedgic initially, John Bailey. That was a good side. And they were finishing fourth. Bob got 30 goals the first year I came up. But then the wheels came off. Then it started to go wrong. And I remember sitting in that big stand there. I mean they used to say the stand at Prince and Park was the coldest in football. It wasn't. That main stand at Everton was. And I can remember being at Goodison with 14,000 people. It was a miserable time. And Howard's arrival didn't change anything. Initially Howard was really... I can see him now dragging his backside up and down right midfield. Desparately having to an effect. He was player manager, remember. And it just really wasn't going very well at all. I was there to see the arrival of the seven. The deal for Neville was done. Neville was coming from Berry anyway. So between sacking Gordon and Howard being appointed. You remember Howard brought in Jim Arnold. His preferred first choice keeper from Blackburn. Jim got the nod as number one. Jim couldn't. He wanted to take the club in the direction everybody was hoping for. Good enough. He was a good goalkeeper at Blackburn. But he was never going to be a goalkeeper at Everton. Ever. Alan Ainsco, lovely fellow. Just bang ordinary. Mickey Thomas, who refused to play in the reserves and was sold as quickly as he arrived. Mick Ferguson. You've got his jersey there, I see. Mick Ferguson. Alan Biley. All these were nice people. Michael Walsh, the centre backer. But they weren't good enough for Everton. Not one of them was an Everton player. In the end, when Howard got boxed into a corner, he rolled a dice twice. Reedy, who George Wood had almost ended the career of in the snow at Bolton one afternoon, did his crucial. But Reedy and Terry Curran, and then ultimately Andy. The rest is history. To be fair, Adrian Heath, he spent big money on India. That was a good piece of business. But it was really hard in the early days. More by luck than judgment. Perhaps he moved easy along Mickey Heaton and wanted to work with Colin. And I think the boys that were around at the time would give Colin great credit. That he came in and started to help change things. But initially they were going nowhere. They were no better than the team that Gordon left. It was a strange time. I was a kid back when he remember that with the first game. We'd be beaming him 3-1 and Rod Stewart, Alan Biley scored a goal. A lovely chip. Everyone's like, he's okay, this fellow. But ultimately you're right. As we went along you could see that. Especially someone like Alan Ains. Decent song games, other games, not good enough. No, bang ordinary. Alan was a lovely guy. I'll tell you the story before we came in. I saw the Ferguson jumper. In those days there was no barrier between us, press and player. Howard threw us off the team coach. That came as a bit of a shock to us. That was a culture change. We travelled everywhere initially with a coach. Me, Charlie Lambert, Brian McEvoy, Eddie Hemmings. We had the front four seats. The Everton coach was brilliant. I was quite good at Monopoly. Linesie always used to insist I went and sat at the back with the players. And played Monopoly. His favourite three, the orange ones, Vine Street. Everybody came out of jail. He always landed on his. I was embraced into the players section of the coach. Howard stopped that. He didn't stop the communication. I'm lying at home one night in Irby. Irby double four, one five. Phone rings. I'm lying on the sofa relaxed. Richard, Howard Kendall. Howard sat up. Royalty. I signed another one. I said, great. Who is it this time? Mick Ferguson. It was everything I could do to stop myself saying. Why? Why? Why me, Ferguson? I'm not being unkind. Mick was a lovely guy. But he wasn't foreverton. And he just kept doing this. Signing people that weren't good enough. I suppose the maybe the pressure of what was going on across the park. He gambled a guess at that stage. And he was young. When he came in he was playing, wasn't he? He was a player manager. We had to play for us. I think he probably was at that time. Absolutely. He was starting. He didn't want to start games. But then the best period came. I'd gone to, as I say, I'd gone to Manchester. In somewhat controversial circumstances. Big money move that was, I'll have you know. And in those days you did not leave Radio City to work in Manchester or vice versa. It's a bit like crossing the park. It was a certain Liverpool to Manchester. So I quit. I didn't tell him. So I quit. I didn't tell anybody where I was going. Then I turned up in Manchester. But I couldn't have been happier that eventually they got it right. And a great bunch of lads. I mean, fantastic. I remember Sharpie before Andy came. And I've said this to him. Moody. Difficult lad. We were at Ipswich and I'm desperate for an interview. And I'm pointing at him on the coach. Will he come off? Just do two minutes. I've got the tape recorder, two minutes. Just didn't. Didn't. They were all under pressure. Nobody wanted to be nice. But I think and I think Sharpie would say Andy turned up. And Andy says and he's right. I think turned him from a score of great goals into a great goalscorer. All of a sudden Sharpie was banging goals in everywhere. Adrian Heath. Great little fella. Great little fella. Reedy. I love it. You can't fail, then. I've always said with Andy, I think a major part of our success at Sky was down to his belief, his optimism, his leadership. You've got one of those around. You've got a chance. You've got two. And him and Reedy could have been brothers. Then you can conquer the work. And all of a sudden, Everton started making inroads. And it was fantastic. Fantastic period of time. That's my standard for Everton. Cos I was obviously a kid at that stage. That paint into your mind, isn't it? But that team was incredible and obviously the films out with Rob Sloman's film, Howard's Way, which is incredible. And we get to live all of those memories again. It's brilliant. But just on a personal level for you, moving on from there, then you were on TV AM. And you remember you poppin up as the morning anchor on TV AM. Was that a bit of a culture change for you? Well, I wanted, I desperately wanted to work in television. And Paul Docherty was the head of sport at Granada. Elton was a young man. So there were no vacancies. It wasn't going to happen at Granada. The lad I moved to Manchester and got close, had I moved in, he went and worked at this new company called TV AM. He took me down there and I wasn't sure at all. I said to Doc one day at Old Trafford, what do you think, Richard? You've been driving me mad asking about opportunities in television. I think you should take it. Well, it was a pretty rough ride. Let me tell you, it was a scary ride as well in the early days. But it was a great ride. And there's another story, another Everton story. It's remarkable this. I'm at TV AM. Howard was kind enough to come in and sat on the sofa one morning. And he said to me, do you want a set of Everton jerseys? I said, what do you mean Everton jerseys? He said, I've got a full set of Everton jerseys and you can have them if you want. Cut final jerseys, the other jersey, but not the one the lads wore. So I said, well, yeah. I'm quite sure what I'm going to do with these. And they were the long sleeve jersey. And many years later I'm working with Andy and there's an auction and his cut final jersey comes up for auction. And he said, that's not my jersey. Mine was a short sleeve jersey, that's mine. And I thought, I think I'm responsible for that because I actually moved them on to Bob Wilson and Bob, I don't know what Bob did with them. He gave them to a charity. So there's a set of jerseys out there all with long sleeves that are not cut final jerseys that Everton jerseys, but they're not the ones the lads wore. Fair play. Fair enough. He was great. Give me them now. Wow, and they're worth fortune. But they're all in circulation somewhere. But no, it was a good period. I loved that. That was as enjoyable a time as it was when me and Andy went on the radio. That was probably the best two years of my career. I mean, we wouldn't have volunteered to go on the radio in the manner that we ended up finding our way to talk sport, but it was a great period of time. Having worked in Manchester, I have an affection for both Manchester clubs, but not like I did Merseyside. That was... I would look at Merseyside and still call it home. Right. Very interesting. It's a great part of the world. I mean, Reds hate me these days. They think I've got it in for their team. I haven't. I just... They hate me more than Newcastle fans. Not quite so much. But I just call it as I see it. And, you know, I was interested to see Jürgen Klok talking a couple of days ago. I feared for my job in the early days. He was no better than Brendan for their identical records for a long period of time. I said of Everton, Ancelotti. I was amazed Everton got Ancelotti, staggered. I thought he was a ready-made replacement for Good Evening at Arsenal. Arsenal's project was further down the line than Everton's. Michael Arthetta would have been fantastic for Everton. I've always said Everton, Moisey made it different, but he arrived and said the right thing. Moisey gave you the People's Club. I've always thought Everton is a club where you have to have touched the ball. You have to understand the club. You cannot go in without fully... And I don't know what it is. I can't explain it. But it's different. It is different. It is special. And Arthetta knew that. Now, listen. Can Ancelotti bring James Rodriguez and others? I hope so. Can Everton rejoin the top six? I hope so. But I still felt and I still do that each club got the wrong one. But I wish Ancelotti nothing but good luck. It would be great to see Everton banging doors down again and being part of the elite. There should be. Don't leave Goodison. I keep saying this to Andy. Don't leave Goodison. I know Bramley Docker and all that. But do what Liverpool did. Buy the houses around and about and develop. You can do it behind the goal that end. Gladys Street is great. You've got the big stand. Come this side. Goodison is special. It just is. It's an old stadium but I think most Evertonians for a long time would have been right there with you. Why aren't we doing it? We should have been doing it from when Liverpool rebuilt their stadium in 20 years. That's something maybe we should have done. We really would have started that process. I still think you could. I still think it would be financially viable to do so. I don't know why not. I'm still not convinced that any club that's left an arena that we were in love with and we favoured has actually gone on to do any better. Arsenal for a decade suffered. Now I don't care how deep Wismanoff's pockets are and the machinery between them. Goodison is home. It is Everton. There will be strains when we leave. But back to you. You've ended up at Skye and then obviously started fronting Super Sunday which was the change. Back then could you envisage when you got asked to head that up what kind of impact you would have and how the Premier League changed? No. I don't think any of us had any idea. In the first contract we got the first Premier League contract and we'd all been doing it for a long time and we had a CDAM with a view to becoming the first to leave the sofa and make a success of being a sports broadcaster Nick Owen tried. Nick's a good mate didn't work out for him at ITV sport but he'd previously of course been a sports reporter and presenter so I had to reinvent myself again 92 there was a quality clause in the contract if we weren't good enough we were going to lose in six months and there were a lot of people at ITV that were obviously chipping away and Fergie was part of that set at that time, didn't want it to work but I do remember I had a sport David Hill at the time saying we'll look back at this period and say they got Premier League rights for the relatively minimal price of and it was 300 million something like that and at the time it was massive but you look back now and you go it was nothing, really he was right we got it right with Andy who reinvented TV analysis and gave us what we've got to I don't think anybody's moved it on in the way that we did I think it's still there are different people there are different toys but it's by and large still the same as ever it was and it was a privilege to be there and do it across that period of time great memories going back to Everton was shocking the contract in those days was you had to see each club home and away once I think and there were only 40 games remember so it was rather limiting but the left of the games were for us to play with, we could go where we like and Bill Kenright called me in one day said you never come to Everton it was good money if you got the extra games I said Bill it's not fun coming to Everton he said what do you mean? I said I can't park the car I can't get in the boys on the security gate want to make life difficult the trucks are an imposition you can't park, you don't want them why would we come to Everton to be fair the next time we went to Everton they were rolling out blue carpets and we were Bill changed everything and to go back to Everton was a treat Ferguson the night when Joe Royal took Joe Royal there's another Joe Royal what a good lad the night that Joe took charge and Ferguson won the Derby these are so many good memories of games there and special I'd go back to special Goodison is special is it as special as your iconic jacket right at the very start well there's a story there as well the reason for the jacket was obviously our job was to sell we were a commercial operation we'd spent an awful lot of money on it we'd given football a shop window for the first time in my opinion our national sport had a shop window in which to present its wares it hadn't previously been the case I think ITV's first live games used to be around about November and BBC came in around about January and obviously there was a cup final once a year but no one was doing it like we did it but they weren't really taking notice of us and we had a we had a stylist by the name of Paul Sherman who came in and said I've got an idea Eve Sandler on jackets we should all be wearing those and ironically Andy had turned up a hybrid in a yellow one before we went that route he was the first to go on air to sky with a coloured jacket but he also had a very good relationship with a gaffer so when he said I don't want to wear these jackets I think Keesie should I was the mug that had to wear them and this day did the job everybody started talking about them what are we on? 30 years? 35 years? I don't know but everybody is still talking about coloured jackets I quite like them for honest but they all got given to charities in various different places found a home once I had stopped using them but it was I had 12 all sorts of different colours whites, greens but I always carried two with me because I couldn't I was worried about wearing one in the colours that the team were playing in and that became useful the day we were at Wimbledon and we were doing the introduction to the show live and the boys out in the dressing room door we were doing it from the tunnel the boys out in the dressing room door that was in a viny squirt liniment and all sorts of things so of course cut my tie off certainly back out I came out a dishevelled mess changed and I was ready again so carrying two jackets that day was a very good idea most definitely you were foreseeing that's what it was especially when you went and met up with Vinnie Jones but I think it was for all of those reasons no one had ever done this before no one had it the way you had it with Andy Gray obviously that relationship was brilliant I loved the footballers football show and the boot room I remember Andy doing training sessions with clubs, with the strikers and sky at it but that just wouldn't happen now because you can't get close to them no you couldn't do it now you can't get near the lads we would Jeff Shreves obviously would he'd managed the tunnel and the lads wanted to come on 300 quid 300 quid for a guest on a Sunday afternoon I remember Jack Charlton coming to Leeds Jack had done a lot of ITV work great but ITV would come on hello here's a game, half time gone, gone, that was it nothing and he sat down we went on air we used to do two hour build up and he said when's the game, when's the game I said four o'clock it's two o'clock man I said yeah what are we going to do when the game started he went over the road well there's a chip shop opposite when the game started he went over the road for a pie and chips didn't see the first half came back and sat down he was both bored and tired wanted food he was great but we used to do two hours then the game and then an hour on the back of it five hours incredible but that changed it didn't it that got everybody hooked I remember Wimbledon against Covent Wimbledon, Aston Villa there'd be three and a half thousand but it was on sky and you'd sit down and the programming started seven and you'd watch it three I think the first game was first Monday night was at Main Road Manchester City and QPR I remember my opening words that night weekends will never be the same again because it did extend the weekend going back to work Monday morning was miserable so if you've got a football match to look forward to and there was one of some quality in those days you could get good games on because there were no Champions League matches so there was no demand for clubs to play Tuesday Wednesday so you'd get some good games on you had to go everywhere like I said you had to go at least once at home and you had to follow them away but it did make the weekend and certainly Monday was a whole lot more palatable those that remember we put on halftime shows and the like poor old Sonia came to Coventry and got battered one night that was a shame but if you did that now I think you'd get a better reception for that we were too far ahead of that time I was going to say you were so far ahead of the curve weren't you at the time with some of the with some of the stuff you were doing but it was great it was good fun and it made Monday an entertainment which is the whole idea we nick the title of course from the Americans their Monday night football was very different to ours but we'd stolen the title and it survived and it morphed and it became when we used to go to the games then we came inside and it became Andy's baby then he could really start to analyse football matches and that was my second crisis I'd had one on Merseyside and when I got to know him I thought I'd educated myself to a level where I knew something about the game met Andy and realised I knew nothing again because he'd just taken it on to another level and I think he introduced a generation to a better understanding of football than any of us I mean I watch stuff, we're in the middle of lockdown I watch stuff now, I watch legends Brian Moore and Coleman and people like this and I'm seeing things on the screen that they haven't won the other night, a Liverpool game Ray Clements was supposed to have touched a ball over the bar and he got nowhere near it and I think it was Barry Davison or one of Clements who is laughing he didn't touch that but it came off the shoulder of the centre half here, have a look at that those are the things that Andy started to get us to look for and I know even now if I watch a game with Andy he'll see something I haven't of course he will, he played, he knows and that's great but I always ultimately bow to his because he'll be right and I won't be that's the bottom line I mean you have got that on screen chemistry, I still watch Kees and Grey every day on YouTube because obviously can't have, no I do but we can have it's hard to get being sporty but we want it there I mean you nearly lost them in 1997 to Everton do you think I mean they spoke to Andy last week and he was still adamant that he would have offered him the job when he went for it, he'd have took it and he didn't which is bizarre but that's another story and another time I guess but do you think that he's known he's worked with him for so long knowing how thorough he is about the game do you think he'd have made a good manager or has he I've always said, by the way the story he told you he was correct that the job was never offered and I think he's got a lot to do with the fact that he went in and particularly for me made the point that the first team was the first priority that he needed to attend and there was a feeling at the club and you need to be organic there isn't time but I've always said I think he would have been a sensational coach Reedy's never had the credit for what he achieved sensational what Reedy went on to do Andy I'm sure would have done the very same thing and I know that deep down there is a regret that he didn't get the chance to go back and manage his club Reedy should have been invited back Peter Eid didn't end up coaching managing at Everton probably people that had the job were scared of him but he should have been given the job at some point but I am certain Andy would have made not sure now I think the games changed to a point where Andy's and probably Reedy's motivational manner shall we say is maybe outdated but I think back then I think Andy would have been and I've always said this to him it would have been fantastic I know they say that good players don't necessarily make good managers I'd argue slightly differently Bobby Moore was given Oxford City nobody trusted Bob Bobby Moore didn't make a good manager Bobby Charlton was given Preston to start with Bob didn't make it as a manager if you're put into a position whereby you've got the wherewithal and you've got good players there's no secret chances are you need a little bit of manipulation and motivation around them but good players will always win trophies so if you've got a coach that understands and can get a performance from people and Andy can Andy gets performances from people he fills the room, he inspires people I think he would have been a fantastic coach but equally as I say I think Reedy would have been and I don't know how is the case that Peter was never given ever to know when I think of some of the appointments that have been made he was the right man for me but there we are I totally agree with you you're in very diplomatically put there as well their management style might have changed slightly with the way it is but that football has changed hasn't it you can't rally people you can't get in people's faces then days are gone now and you know when you play at any level that was part of the game you needed to do something up sometimes you needed it now you'd have a lawsuit on your hands almost wouldn't you well you would do you'd certainly have HR that's an absolute certainty life has changed but that go all the way that team that Howard had they were like the great Reedy teams all of the great teams they look after each other Brian Marwood half dingy inevitable that Brian Marwood was going to get half and for the rest of his career didn't go anywhere near Reedy because but they looked after each other in she goes to Andy at Newcastle that Glen road is kicking me Andy so Andy goes and leaves a bit on Glen unfortunately breaks a metatarsal and in she comes in for 12 games that's it isn't it that's football isn't it but it was it Leeds would hand them on if somebody was having a round with Billy Brenda then Johnny would take over that's how it works but again you don't see that these days I'm now at that stage we got banned by Fergie in the first year we had the temerity to turn cameras on it when he had a tantrum in the dugout at QPR explain to it that's our we're not going down the tunnel with you we don't want to go to the dressing room with you but if you're in the arena we're going to point a camera at you and if you're misbehaving you're going to look daft but anyway he decided to ban it so we turned in that first year to George Bess Dennis Law, Bill Folks I mean all these guys legendary characters at Manchester United but all of course of a certain generation well I am now of that generation I'm now the old boy and so is Andy and there's a wrath coming behind us but it was great incredible incredible times and then obviously you went to talk sport and you had a really good two years there how did two best years of my career and I mean the story how we got there is a lengthy one an entirely different story to how we left sky but the story how we got to talk sport bottom line mate of mine was program controller Mike Parry, Evertonia had resigned once too often having had too much to drink one night and I met with Mars and talked about joining them and I said to Andy we're going we spent five hours on the roundabout at Wickham on the M40 and I had five hours and I said we're doing this, no we're not Andy we're doing this, no we're not it's too soon Andy it's not too soon there's never too soon there's too late but it's not too soon we're doing this, no we're not and eventually I think I grounded to a point whereby he said okay and everything was agreed that night but he'd never done radio he didn't understand what fun it was and it is it's still a fantastic medium and I just think we sat there the first show we did I mean I remember we looked at each other about and of course we went to the news an hour and a half in and the street was full of press and it's the most uncomfortable place to be at that time but I looked at the clock and thought an hour and a half to go yet just done that an hour and a half and oh we're going to come back tomorrow oh and the next day five days away and that three hours felt like three months and then it started to feel like three weeks and then three days and then in the end three minutes it was great and we're doing the same thing again now with what we're doing what better is there to do than sit and talk to people about football I can't think of anything especially when you've got your black book you can get anyone and it's fantastic again you're still doing things that other people aren't doing and I don't think other people who've come after you just going back to Sky they've come after Andy don't get me wrong but it's not quite the same I'll tell you what I think the biggest difference is I'll tell you what I think the biggest difference is and no one's ever really thought about it Andy was a forward Andy was someone that did the hardest thing on the pitch Barnum, score goals so I understood how much it hurt for people to create and he looked at the game from a creative eye he would always say if I'm in the box being marked by a centre bank I'm favourite I'm proactive, he's destructive he looks at the ball, he doesn't know where I am I move behind him, next minute he turns around I'm gone, bang, goal so he always looks at a goal from a creative and a constructive point of view I think there's a massive difference between that and a defender and an ordinary defender or two telling you how the goals should have been stopped there's always a reason why a goal should be stopped but sometimes you say do you know what, that was magnificent I think not everything is bad not everything should be destroyed not everything I had this conversation, there's a name just to drop I had this conversation with Arson when he was out here with us and Arson said I'm in the middle of having a big row with sky at the moment and I said oh that's interesting, why he said because I don't understand why you would spend money they have, 1.1 billion on rights, and then set about destroying the product week after week why do they do that and I said well, I've got my own theories but obviously it's not for me to worry about anymore Arson but you don't have to sugarcoat it, but there are ways of doing it that offer a more constructive conversation than destructive defenders look at the game differently, they just do and I think that's a really good point because it is, it's the two defenders so they're looking at why this was done wrongly and Andy's such a great character as well, that he was able to make you look at it from like you say a positive thing of ok maybe the defender maybe the defender could have been better but how good to strike it to do this because goals, essentially we're there to look at goals aren't we, you'll have sat in that studio and watched probably a million nil nils and you've thought, I wouldn't mind a defender may have made a mistake today in someone's score Well that's what you want, that's why I'm so frustrated with the way that they've implemented VAR, the game's become it's too technical now it's a game that we should enjoy through the eye with our eyes that gives us joy not destruction, how many goals this year that is allowed with VAR, I think we were at 50 when we stopped, 40 of them for that that's not why we pay our money to go and watch football matches goals are the commodity that we go to see so enjoy them I mean you've just brought up Arsynfengir there I mean for me, he's come up with the solution to the offside thing I'm fed up of armpits being offside with it and now they'll say it's the rules and everything but years ago I played the front Andy Gray will say this he played at a much higher level of course all you do is you get yourself in line with the defender it might only be your leg and then when the ball will go you but as long as you add the part of your body in line and this is what Arsynfengir is talking about now let's forget about him well he is, but I I go with that to some degree look it's not going away in my view it should never ever have been introduced it has added nothing it's the same with a piece of equipment when you're analysing football Andy would always say there's no point giving me that what does it add? nothing if it adds something, fine VAR has added nothing it's destroyed and it's destructive and it is there largely for that purpose it is not there Arsynfengir got one at Manchester United I can't think of too many occasions when we've said good job we had VAR today because that might not have been allowed but I think the price we've paid for it has been too great and I always said this my view on goal line technology was consistent I didn't want it not because I didn't think it might occasionally help might occasionally help I didn't want it because I knew it was the thin end of the wedge that the minute we got it people would start saying let's have cameras now for offside and frigate we've only just started with VAR which phase are we going to be arguing about soon how far back do we take it are we going to be arguing about soon and I don't think the game I don't think that's not the game I want I love watching championship football because it's pure, it's clean and a goal scored and people celebrate and crowds are enjoying it immediately not waiting what's happened there if you're in a stadium it's hopeless you've no idea I mean Everton so the last game I was at a goodersun was the man United game Dominic Halvert-Lewin scores in the 93rd minute it's essentially the winner by the way it is exactly so we're celebrating and you're looking and going well we can't really celebrate because the referee's having a chat here and you know what Richard they've either gone to the screen and looked at it and gone okay for me I thought it was a goal because the heihe didn't he could see the ball his body showed it he could see the ball because he moved he's not getting it but if the referee were to look at it and see this decision himself had he gone okay wouldn't they have liked it no but at what stage how is someone telling him and he's not got any influence on that decision it's bizarre is that just them giving responsibility away do you think totally this is whether they will never admit it but if I'm Martin Atkinson and the lad's name from Australia escapes me at the minute but the boy they brought from Australia and he's only done VAR at the minute you're not overturning decisions I've made you're not making me look a mug and so it depends who's on the pitch and it depends who's in that bunker and look it's not going on me but I wish we'd never started with it we didn't need it it's added nothing for me it's all negative and that's not why I watch football matches I want to be entertained and enjoy it but somebody scores a goal great all right onside yeah fine no reason to decide that get on with it but you lad's there on the where were you glad he's straight I don't know you in the posh seats now still wouldn't that's it we've beaten Manchester United no we haven't are you sure? no maybe that's exactly it just finally on that do you think it would help if the VAR people where referees who were still refereeing in the Premier League were almost independent or do you think it doesn't make any decisions I would prefer to see I would prefer to see people have done the game you don't need to be a referee to be sitting in there yeah I would prefer to see people that have retired that have recently left the game there are people there are good examples of people sitting on their backside Dean according to Mike Riley quotes to be not good enough to work in VAR well how good do you need to be to sit and watch your monitor I mean it's jobs for the boys by and large but we have to accept it's here and sadly I think that Blatter was right I would rather have people behind the goal and keep it a game that's occasionally going to let you down but give you something to talk about and reward you for the money you've spent you're going to see goals today not see technology disallow goals fair enough finally then and this is probably listen I kind of know your answer because of what like you say I watch the show but in terms of where we are at the moment obviously we're in a very very strange place Lee Gain yesterday cancelled France said nothing till September Italy have said the review and you know what they'll do Germany was supposed to be starting a week on Saturday and now the messaging's changed slightly but it appears here like everything's still geared towards this June the 8th June the 10th start in your opinion what do you think should happen do you think we should just leave it now till September or I said right in the first week and I had a row with Andy about the consequences of ultimately coronavirus in general he wasn't convinced but in the first week I said we will have to reset the clock my own view remains that it's the best look if you want to give Liverpool the title fine but it will always be a title that comes with an asterisk which says didn't finish the season but if they want to do that fine but I think it's unfair on others I think the best solution is to reset the clock then you're not as far as the television companies are concerned if it is purely about money they get the extra year and we can negotiate down the line no one's threatening anybody now the Premier League remember are made up of 20 shareholders they run and make decisions about the league themselves there's no reason at all why they need to they don't have to invite Leeds in who haven't yet established the fact that they're up my team are top of league one look there are far more and I keep saying this how can we be talking about playing football when people are still dying how can we dismiss the effort put in by front line workers who probably want to go to a match or watch their team play but can't how is it to lift spirits for six weeks in isolation in Germany they'll wear masks really it's not a nice subject but you can't spit because you're not at a social distance so can you tackle I mean I think reality ultimately I think we're edging there we'll say look I think the French league finishing is the first domino first the big five we'll have to we can all start again at the same time we can all kind of finish at the same time now if we say enough we didn't want it we didn't foresee it not everybody's going to be happy but there are far less problems and complications for me if we call it a day now am I excited about going back yeah I want to watch one live football course I do done it most of my life but do I want to go back in an era where players are concerned about their own well-being it only needs one player to contract coronavirus and heaven forbid suffer the ultimate consequence 100 million pound player I mean the risks are too great for me and I don't think when you sit and think it through there is a comfortable way in which would help the Tory government who would be very happy for people to start playing football again and get us talking about things other than the mess they've made of this crisis but that's not the point it isn't about money or shouldn't be we ended talking about morals and the morality of the game as the bubble burst as far as wages are concerned I hope so is there a reality about the game coming back I hope so but is there a morality about the game then we have to show that that is the case and I think we can't finish this season start again next year as we were as we were and then get on with it I know people will say what about cities European ban maybe we have to put that back a year as well none of this was foreseen by anybody it's it's uncomfortable for everybody any scenario going forward is uncomfortable what's the best of a bad choice I think reset the clock I really did players out of contract on June 30 they've said to Andy if I'm a player out of contract on June 30 if I'm William I'm not playing on for another month he's gone mad with me that's a disgrace that's typical of you I've got a family to think about in my career I get injured in that month who's looking after me I'm sorry I'm not playing on beyond that point let's have five substitutes that's not how we finish that's a distortion of the competition we had if we have five substitutes I mean Danny Higginbotham let's have ten well let's have 20 Danny I mean how far do you want to go for me it's just all of the how do you have VAR you can't sit in a stock your part next to each other so then competition changes push it through we may end up playing again I don't know if we are I'll be watching football with the same as everybody else but I am massively uncomfortable about the fact that we are nowhere near in my view getting through this or anywhere near the middle of this and lives matter more than asking footballers to be guinea pigs and get back playing sport so we can have something to do at night the stakes are too high for me and I said that in week one and I was battered by people this is why he's wrong again well I don't think I was and I still don't think I am well more and more people are 100% with you so maybe at the very start people might have thought you were wrong I don't think there'll be a lot of thinking you're wrong now it does seem to be it's becoming uncomfortable a desperation to retain when you're absolutely right we haven't got enough PPE on the front line and why should footballers be tested when staff can't it's becoming almost the issue of is it right to even bother it's a game at the end of the day we love it but it's a game and if it gives all those tests that they have or that they want or that they need to buy 4 million get them to people that are saving lives get them to medical workers sitting at home that can't go to work because they don't know if they've got coronavirus that's where these tests should be not for multi-million pound footballers it's not going to give me a lift it's going to make me as nervous as hell if I start playing football again let's leave it there fantastic and so listen Richard massively appreciate you taking the time out being fantastic and love to do it again in happier times most definitely just have a general footage here anytime you like I'm flattered that you bothered to put the Mick Ferguson jersey in the background ha ha ha ha ha listen Richard thank you so much and stay safe is the biggest the pleasure for me thank you very much what a fantastic guest that has just flown by almost an hour lovely of Richard to take the time out busy man but absolutely brilliant to have a chat with them make sure you subscribe give the video a thumbs up and if you want more videos join us over on page and see you later