Let's get started. This is Switches, Get Stitches, and it's the third episode. You might want to know what the first two episodes were if you missed them. One was a workshop at 44Con. 44Con believed in the research very early and said why don't you bring it to a small number of people and do it as a workshop. Then I took it to Chaos Communication Congress and had a lot of fun there. And I wanted to get these guys to join me to bring in some different elements to the talk. And it's a talk focused on industrial ethernet switches and breaking the management plane of those switches. So we've got the Siemens Galance, the GE Multilink, and over there, the Open Gear. So that's what this is about and that's the previous episodes. So, who are we? I'll let you guys go first. So I'm Colin Cassidy. I've been a senior software engineer for 15 years and then gave that up because it was a horrific job and decided to break stuff for a living. Rob Lee, so I was one of the guys you probably hate. I was in the government recently, but I'm no longer. I got out two weeks ago, so I'm alive and a civilian again. And I'm a researcher in the industrial control system community as well. And I used to be a senior risk researcher at the Cambridge Center for Risk Studies until I started telling jokes at DEF CON. All right, so we expect you to be a DEF CON audience. We expect you to have a lot of fun. So there's two things I want you to make noise for in this presentation. One is private keys when you see them on the screen or we talk about them. And the other thing is when you hear a vendor patch time story that you like, I expect you to make some noise. So switches get stitches. Let's get started. The scalance, X family, so this is X200 switches, X400 switches, basically this is the one I worked on, right? They have session IDs in their web interface. And we all know that session IDs are a good place to get started, see if you can do some session hijacking. As we said before, all of the vulnerabilities that we're talking about today are in the management plane of the device. So when you go to the website of the switch to manage it, you get these session IDs. Now any of you who do a bit of reverse engineering, does anyone in here reverse engineer? Good. I'm glad that there's a few people in here who can teach me something. These vulnerabilities should be pretty obvious to you, right? You see the CO and the AA and you probably think to yourself that looks like a local IP address, right? But you'll also notice that they're strictly increasing. So these session IDs are increasing only. They're not as random as they could be. So if you don't mind flipping to the next, you can see that the ones on the right-hand side, if you pull the device every second, it will actually give you incremental session IDs. And when I compared that with an SNMP request for the uptime of the device, I discovered that it was indeed uptime in hex. And then I got confused because I checked the IP address at the front of the session ID and it wasn't the IP address of the switch, right? So I kind of thought about it for a little while and then went no, surely they didn't. And of course it was my machine that was connecting to the device. So client uptime and sequential session IDs based on time was one of the first vulnerabilities that we found here. Weak session IDs. And you can sort of see in the mind of the developer here some sort of madness, right? Like, I know I'll create all of the IP space and all of time and that will be impossible to ever estimate, right? But clearly it's a very weak session ID. I actually go back one and I'll say one other thing. The bit at the top is essentially how the passwords were hashed on the wire. So the admin at the beginning is obviously the user. Password is the clear text of the password. And then you take that session ID that I just talked about and you use that as a nonce. Now the nonce, the number used once, is passed back and forth. And the idea there is that you can't replay a hash from previously, right? So that also probably affected why they used the uptime. And there it is on the end. But you'll realize that by putting the password in the middle in between these two colons and then MD5ing it, it's not a very large space to have to brute force. So you can snatch one of these IDs off the wire, one of these hashes off the wire and brute force it fairly quickly. Like 16 characters I think took maybe 15 minutes. Something like eight characters would take a minute, right? And then on top of everything else, it was MD5. So continuing the theme, the open gearbox here also has a weak session ID. Essentially it's the number in red. Eight characters long. It's basically a hex decimal value. It is semi-random, it doesn't increment. But beyond doing any more analysis above that, we didn't really look into much further. However, there was a nice configuration item built straight into the switch that allows you to extend it. So this one's already fixed and it was fixed from the start. It's just not on by default. And this highlights some of the problems with these switches is that they're not always deployed in the most default, hardened state. Some of these reasons are to do with backwards compatibility. They've been deployed in the field with this previous state. And in the ICS world they tend to be very risk averse. If there's one less thing they can test and just do a quick swap, that makes their life so much easier. All right. So after finding these issues with the session IDs, you can do some side-jacking on a switch, both on the open gear or the Siemens scalance. But if you work regularly doing pen testing, you'll realize from an operational point of view this is not the greatest attack. Like, how often do you guys log into your switches? So I have to wait for that to occur, right? That's probably going to happen every couple months at best, maybe once a year. So we didn't like those attacks. We wanted to go for something a little bit more robust in terms of being able to use it any time. So we were focused on authentication bypass and particularly on firmware. If you go over to my GitHub, you can pull down this script. And basically this is a CSRF, right? But the CSRF is, it makes it possible for you to download a configuration file or download the log file of the device. That would just be a get. Or if you do a put, then you can put the firmware or you can put the configuration or the log file. And I find this kind of amazing, right? Because you can change the log file on a device before you've even broken into it. And this creates an authentication bypass in the sense that I can have a known, good configuration with a password that I know. And the passwords are sort of hashed in those files, by the way. And I take that known, good one. Well, before I use the known, good one, I download the configuration of the device. Right? So I have the passwords that used to be there. I upload my own configuration. I log in with my password, which might be finally waxed mustaches or something. And then do whatever I want with the device. And then re-upload the old config. And no one's the wiser that the password has ever been changed, right? So this forms an authentication bypass on the device. And it's also brilliant that you can just post a firmware image to the device. So Siemens have fixed this in newer versions. But if you do pen test in this environment or you have access to these switches, feel free to download the script and prove to yourself that those versions were vulnerable. I think this one's you. So I essentially got involved in this project about a month after joining my firm. I'm active. And Aaron said to me, I've been breaking these switches. Do you want to take a look? I said, okay, fair enough. He said, we've got the latest version of the firmware. Download it and take a peek and see what you can find. So the tool of choice more often than not is bin walk. It's a nice wee tool. It basically looks through an entire binary, looks for sort of headers of particular file types. In this case, we can see there's a large compressed file. And so you can use bin walk to essentially pull it apart. So taking the large compressed file, we can see the first four items there, closer. Is that better? Sorry. So we can see the first four items there relate to the certificate and private key of the Siemens Skellant. And indeed, there it is. So we reported this to the vendor as you do because, you know, we're nice like that. The vendor came back and says, yeah, but you can change it in the web interface. Okay, that's fine. We're in the documentation. Do you say you have this key that needs changing? Otherwise, that's just like an undocumented login password back door. Okay, we'll change the documentation. So self-sign certificate, it can be changed by the web interface. Nowhere was it in the documentation. So guess what? Vulnerabilities in these devices tend to be pretty common and pretty in the same style, in the same pattern, right? Default credentials, default keys. All of the OWASP top 10 are in there. And that's important actually because we like to do a bit of embedded and we like to do binary analysis and we're teaching ourselves to be better reverse engineers. I'm just looking at ECOS at the moment, trying to build flirt signatures and stuff. But we didn't want this talk to be like that. We wanted this talk to say, look, if you do web app pen testing, every embedded device now has an embedded web server.  So we could use your help if you want to come and look at embedded. Yeah, you need to learn some stuff about control systems and hang out with some engineers. But frankly, that's really good for you. They tell some of the best jokes, as you heard earlier. And in this particular device, I had to figure out how it was compressed. I carved out a firmware upgrade. Actually, it's worth backing up a little bit. Someone sent me a PCAP, right? There are various engineers who know that I work on this stuff and they'll send me sometimes anonymous letters, sometimes not anonymous letters. And one of them sent me a PCAP of the firmware upgrade of this device, right? And said, you know, here's this HTTBS traffic. But then once we do the firmware upgrade, it goes over FTP. And it uses the same credentials, right? And he was like, this is pretty bad, right? And I was like, yeah, actually, let me look at the rest, right? So I basically used TCP trace and I carved out all the files. And one of the files was much bigger than the other and it was over FTP, right? So now I have a copy of the firmware image just from the PCAP. And I looked through it and BinWalk didn't recognize a lot of it because I hadn't recompiled it and upgraded it and some other things. So I had to figure out that a section of it was compressed, which you can do using entropy and other stuff. And then I ran a little script over it to figure out where the compression lied and managed to decompress it and found a ROM image. And inside the ROM image, there are these private keys, right? And I know you guys like private keys, right? Everyone has a private key collection? Yeah. So these are, these private keys, the first one is for the HTTBS interface, which has just been ruined anyway because you've seen the credential go over in HT, sorry, in FTP, at least if you've ever witnessed a firmware upgrade. But if you don't witness a firmware upgrade, you can take the key from the ROM image that I just described and install it in Wireshark and you can read all the traffic. And the passwords underneath are unhashed. So you get those as well. And I think this is a really key, sorry for the pun there, that was entirely accidental. But I think this is a real key issue in the sense that once, if you only have one encrypted interface on a device and someone breaks that, you now don't have a secure channel in which to up-no-load new keys if attackers are persistent on your network. And I think that's an issue. The second key belongs to SSH, but couldn't be enabled on the device. And another researcher has since brute forced the key because I couldn't be bothered. So the password on that is Magnum 6K, which is a little clue to something else we'll talk about in a minute. Oh, I guess we're talking about it now. So it turns out that this switch is, you know, sold by GE, but it's manufactured by Garakom. They make a line of switches called Magnum 6K. So this key belongs to the Garakom switches in the same way as the GE key belongs to the GE switch, right? So you basically just take a different firmware, you analyze it again, the key has changed, but it's the same. And I should clarify that these particular keys affect seven out of nine of this switch family. So there's nine of them that are managed switches. Well, one of them is not managed, so therefore this stuff doesn't apply. Then seven are managed and the biggest one has a different firmware image.  Just continuing a thread on that. Whilst we found these keys in a GE switch and we reported it as such, essentially it is a rebadged Garakom. And so when you see the reports coming out saying there's issues with GE switches, we don't know who else has rebadged these Garakom switches, so who else is affected? Is there a Siemens switch out there that's effectively a rebadged Garakom? And how does that essentially get out to the wider public? There's a sort of web of network vendors and rebadgers or resellers that it makes it hard to do proper sort of incident response and incident control in that regard. So flush with our success pulling keys off Siemens switches and GE switches, we moved on to the GE MDS-wise. We don't actually have this switch, instead we just pulled down the firmware. This sort of investigation sort of highlights the problem of not having this switch. What we found was not really problems. We went to GE about it and they said, no, they're not really problems but we'll go through the process of what we found anyway because it's interesting if nothing else. So binwalk being our friend, pulled it down is essentially a squash FS. Binwalk knows how to extract this stuff and it happens to be a nice essentially Linux system, file system at the end of it. There's a search directory so you can applaud that later. But there happens to be a .password file which we thought was very interesting and indeed it contains the passwords and hashes. So, you know, it's got the admin one, it's got the guest, it's got auth code which is used, if you forget the password you need to reset it, you ring up GE and say blah, blah, blah. It's got a factory password which was not documented and it's got a root password that I got bored of waiting for Jack the Ripper to break. So if anyone has a faster password cracking rig than me, take a note of it or find the firmware and go knock yourself out. We did report this to GE, the guys rang me back, I think I had one of their head product cert people and about their entire development staff and they said, yeah we looked at it, we don't believe it, doesn't I think. Now I don't have this device to test that so if anyone's got one I could borrow or can lay their hands on one and wants to find out, that would be ideal. But the other slightly bizarre thing is this is an industrial network switch, why is there a games user? Oh and they've got private keys too, yay! So yeah, key management in network equipment, you are going to find default keys all over the place. If they're undocumented that is bad, if they're unchangeable that is worse. We have not found unchangeable keys however other people looking at switches have. Self signed keys are more confusing issue, yes you have self signed keys but you lack the ability to revoke them but if you're in an isolated network environment that's fine because you shouldn't necessarily be connecting your industrial system to the rest of the world just so VeriSign can verify that you have the right certificate authority and key installed. No, so but if you've got an internal one you can probably get away with doing that sort of thing. These switches tend to lack the sort of processing power and any sort of entropy to create their own. But if you're going to set these things up you will plug them into your laptop and your laptop tends to have this sort of capability. So to the vendors out there maybe they need to consider a sort of initial step process that helps lock these things down from the start rather having one key that we can now talk to every switch on. That's not helpful. Essentially the problem with key management is you have to manage your keys. Can't just leave it to the vendors or whatever crazy mechanism of just leaving it there. All right, so you know let's get back to some web app phones instead of some you know scraping, gripping keys out. Actually this is a good story about that.  How do you find so many vulnerabilities? And he said I just gripped for them. And I thought you know what he's right like if I'm going to write one script for industrial control system devices that does something what would it be and it'd be like gripped for private keys and hex. And it works. So this switch also amusingly has a flash interface if anyone's really into flash. I didn't explore that. I just went looking for cross-site scripting and I hope that GE will forgive me for using that phrase in my proof of concept but I was listening to DJ Kubert and that was a sample at the moment that I found the vulnerability. So it just seemed appropriate. So there are eight types of cross-site scripting on the management plane of this device. They also apply to Garrettcom and you don't have to put them in specific parameters. I don't have to tell you which parameters to put them in because you can make up parameters for the web server to put your cross-site scripting in. Hello gentlemen. Hello. How are you? Very good. So we have a thing we do for people who have never spoken at DEFCON before. It is called shoot the noob. We won't actually shoot them. Don't worry. Jeffrey, I think they're familiar. How many of you are not familiar? One guy. So I'm talking directly to you. So sir, were you born yesterday? Have you been to DEFCON before? No. Okay. Is that your real name? Yes it is. Wait, what's your name? Ryan. Is that your real name? Yes it is. We don't use real names at DEFCON. Your hacker name is Arapastoli. So we're going to explain to Ryan if that is your real name that it's very hard to become a speaker at DEFCON.  That was hard. I'm sorry. I'm sorry man. I really am. That was uncalled for. Ryan, I wasn't talking about you. You're the smart guy. Anyway, so just a quick toast for our new speakers. Are they doing a good job? And Ryan is up here representing all the new attendees. So thank you to DEFCON. Thank you.  The bear didn't get a shot. By the way, you guys got two minutes. Okay. So then we'll have to make that pretty quick. So we promised to come up here and drop private keys like DJ Jackalope drops breaks. I hope you've appreciated that. I think she's awesome. I come here to drink and dance. And this is an excuse to do that. But I do have to do a lot of hard work for it. So let's go to the next slide. So GE firmware integrity. Remember earlier, I didn't want to brute force that password that Ashish Campbell of QALIS brute forced. The reason I didn't want to do that is because I can patch in my own keys and firmware and then run a CRC Adler at the end and then try and reverse the algorithm at the bottom. So there's basically two checksums. One's code signing more than checksum.  So top tip there. So this is one of my favorite vulnerabilities. Okay. Colin once asked me why do you hack switches? And I said because that's where the packets are. You know paraphrasing, Dillinger, right? But also this is important because industrial system networks have a little bit of authentication and a little bit of crypto. But for the most part they don't, right? Most of these protocols, if you can put packets on the wire, you can tell it to do something. Occasionally you see something like a, they'll prevent a replay. But essentially if you understand the protocol well enough, you can just tell it to do what you want it to do. And you can even perform things like recording traffic and replaying it, right? So once you have control over the switch, you can exfiltrate traffic. Now, that'll work in many environments but on some real time systems, let's say electrical, you have like 500 milliseconds of time in which to perform some of your attacks. So if you route your traffic halfway around the world to another country, you're going to fail to reach your timing constraint in that system. So you need to be able to alter the firmware to perform some of the process control in the style of Jason Larson or something. So then I'm playing around with the switch and before authentication to the actual web server, I find this DOS, right?  And I created a no-cache parameter and just, you know, incremented or randomized the number and requested this particular config file 2,000 times very, very slowly. And the web server of the switch rebooted, right? It was just initial fuzzing when I first started. And I thought, oh, it's a DOS. So what? DOSes don't matter. But they do really matter in industrial systems, particularly when the web server going down causes the switch to reboot for two minutes and all the traffic to be dropped in that time period. So this is the fix, the current fix from GE. They had to go to Garakom and ask Garakom to fix it. And I think originally Garakom said, sorry, this is end of life and now they've changed their mind and now they're patching it. But I want you to read that very carefully. Their mitigation is basically turn off the web server. I'm not too happy about that. I think we can do better as a community. But how do you configure it now? Well, you would configure it through Telnet, which as we all know is an awesomely secure protocol. So why does DOS matter? Why does DOS matter? In the ICS community we try and be really good to each other and reference great work where we see it. You may have already seen the talk of Marina Crotaful and Jason Larson. Amazing stuff. They dig really deep into the physics of damage and exploitation in industrial control systems. I stay a little bit higher up and then try and focus on vulnerabilities and cost of attack. But basically they showed in this paper that having a DOS in certain types of chemical process control is enough to give you almost complete control over the process. So go and look that paper up and you'll understand why we're trying to raise DOS as a much more serious vulnerability in industrial control systems. All right. So I'll do the first one. There was an old day essentially, the SSH username enumeration if any of you remember it. I found it when I first started looking at the open gear. It's worth saying open gear gave me this switch. I met them at a conference and they were like, oh, you research switches. We'll give you one and you can test it and we'll get free testing and you'll get publicity and that's pretty cool, right? And I thought that was remarkable, especially in the ICS space. So they fixed it in one week. They patched this vulnerability in one week. Now I grant you, it's just a PAM configuration change. But still, GE took eight months to tell us to turn off the web server. Siemens took three months to change some of their documentation and fix the CSRF. So one week, I think particularly deserves a round of applause in this case. Yeah, so I spent most of my time looking at the open gear switch and it's out of the few switches we've looked at, it's probably the most secure default state. Everything we found essentially was post authentication. You had to be logged on to the switch to be able to do anything. So it makes a lot of the, whilst the impact may be high for some of these issues, the actual likelihood and realistic sort of impact is much, much lower. And in our conversation with open gear, they were upfront and told us when these fixes were planned, they had scheduled them all in, get them all worked. So they really did help us out here. But we'll carry on with the issues we did find. So that is the open gear switch. It looks like that. It's that on the website. You can download the firmware. They also have a open development kit. So you can basically build your own firmware for these switches. So if open gear ever go out of business, you don't have to replace your hardware and you can keep things running over. So that's pretty cool. First issue, they have on their web page this nice support report, which gives you all sorts of useful information. It's probably really tiny, even though it's blown up there. But they have a link right at the top that allows you to download the support details for essentially offline viewing. But in this page is normally only accessible by the root or admin user. However, you can directly navigate to the page as any user and pull down all the information. This includes things like the Chrome tab, the init tab, the entire syslog, a support text file, and that contains the IP tables configuration, the location of all the SSH keys, which becomes useful later on. The IP tables config and the config XML file which includes contains all the user names but no passwords. So immediate user name enumeration. So you can pull down the theater of everybody else. The next issue they have, we found, was get file. And essentially this allows you to get any of the files on the device. Useful if you don't have SSH or telnet access, but you can still essentially pull down any of the Linux files you've got there. So you can pull down their password file, which looks slightly more sensible than the GE password file in that it has no passwords in it and only contains sensible users. However, you can also download their private key. Again, it's their default key. You can change it. It is documented. And then we tried the traditional sort of cross-site scripting. Can we get cross-site scripting in place against this switch? No. Impa validation denied. You know, they've actually thought about this stuff. Yeah, I'm as shocked as you are. However, we decided what about outbound validation. So they have this config XML file. You can log on to the device and you can change a config XML file and basically just input your own XML that looks like HTML. And it executes. Oops. But again, you know, cross-site scripting on these devices, but you have to already have sort of permission to do this sort of thing. So yes, it's nice and quirky, but realistically, it's not the most brilliant of attacks. However, this leads us to the last pack, which is essentially cross-site request podries. So this is a burp suite dump of the sort of creating a new user. And so this is all the bits. It's got all the parameters. It's a post request. So we have mocked up a small web page that contains just that code that gives us essentially all these users. We deliberately asked ourselves to give ourselves admin user and if you can have somebody logged into the device and get them to navigate to your evil web page, you too can create your own login backdoor onto the account. And we can probably demo that if the demo gods are good to us. Can you make that look easy? Can I make that look easy? We'll find out. We'll find out. So at this point in the presentation, you're probably wondering what the hell this guy's on stage for. So why don't we give him a little chance to explain that while Colin sets up the demo? Cool. Thanks. So mostly I stand up here with good looking guys and I don't know if you're familiar with this but what I really like focusing on is the defensive aspect of it. So these two found a bunch of vulnerabilities actually 11, right? And five different switch families. And at that point, if you're an industrial control system person, you're probably kind of depressed. So I'm the kumbaya defenses doable kind of person to talk about how attackers can take advantage of these things, but more importantly how defenders can look for those things. So the unique thing about industrial control system networks, are you ready for the demo? Ah! All right, so this is the login page. Well, it's not the login page, but this is the user groups page. And we can see down the bottom we've got a root, a testing and admin user. The root is the default one. The testing and admin I've just created. So our test script looks shockingly like that, which is exactly what we had in the first place. And so we're logged in and in another tab, if we have another tab, we can navigate to our evil page. Our evil page does stuff. It thanks for a while. Thanks for a while. Please look. Song and dance, anybody? And we can go back to our user groups. And there's a new user. Yeah! And it's admin. Yeah! I know it was frighteningly so. But again, the ability to get that sort of timing attack is probably really low.  You either have to have local control, local network page somewhere or they've configured their own device to connect to the internet and that's probably wrong. So yeah, it's reasonably high impact but likely had way down there. So yeah. So in summarizing, it is a strong default state compared to their peers. The issues have been thought about and they have remediated them. The session ID is a default thing. You can just switch on. The configuration's there. They've got excess input validation. They do have potentially high impacts but realistically low and it's all post-authentication. None of this was pre-auth and these people have essentially fixed these issues in what I like to develop a sprint time, so two to three weeks compared to the vendor patch time of eight to nine months and I think that deserves a big round of applause because these guys are on it. All right, back to Defenses Do-able. Build it back up. All right, so there's a couple themes that go throughout the research. The first of which is as I was working with these gentlemen looking from the defensive standpoint of trying to analyze the post signatures and things like that. There was an interesting amount of vulnerabilities they found and only a level of them made it into the presentation because that's how many PowerPoint slides we could get through without killing ourselves. So when you look at it there are huge issues in the community around these vulnerable infrastructure. So if we see GE or Siemens or Open Gear, that's not the only ones that are bad, right? It's across the industry and we're trying to make it better. But the second theme to come out of the vendors, right? So we want the vendors to do better. That'd be great. But unfortunately if it takes eight or nine months to make a patch it usually takes another year or so to actually implement the patch in these facilities. So we don't want upwards of two to three years of vulnerable infrastructure where adversaries are trying to target these facilities. So we as a community especially for those of you in the ICS community can do a good job of monitoring for and defending these pieces of infrastructure without the vendor. So that's what I want to talk about a little bit. An interesting aspect of industrial control systems networks that I really like is that they're fairly static. If you think of your enterprise IT kind of network you have your users going to Facebook and LinkedIn and all sorts of weird places, right? And in ICS network that shouldn't exist they should be relatively segmented networks. You should have multiple places to capture data on the environment, right? So this screen up here an ideal network is you have your processes separated out from each other to capture data but they're static. If they're using Modbus TCP, DMP3, OPC whatever ICS protocols that's the protocols they should be using on set ports. There should be relatively static bandwidth usage and these type of environments. Now unfortunately we don't really usually do a good job with the networks. This is how a network should look like and this is the typical layout when you walk into an ICS facility. It's all flat properties. So the downside with that is adversaries can get away with doing a whole heck of a lot without ever being noticed because on the network it's really easy to spot. When we're talking about industrial network vulnerabilities adversaries would use that kind of thing like what does it matter? Well adversaries use that type of thing to learn the system, learn the process. There is no getting root on the PLC or the control system itself that doesn't matter. You can change control information do whatever you want just by having access to the network. So the ideal set for any sort of national adversary or anybody in the government that would want to get into these facilities across the world is to get the network. That's it. But we as defenders can see that and I always find it interesting that there's these narratives that build up about how amazing adversaries are, how amazing the quote-unquote APT or whoever is and it's funny because those are usually high. I think it's kind of interesting that folks think there's less power point and less bureaucracy in the government. So we actually aren't necessarily as good as people think we are. And realistically we can do a good job as defenders. So when you look into the ICS environment there are problems. I want to note yes, the vendors have good excuses sometimes about legacy equipment or who owns the infrastructures at the union as the actual vendors themselves who can go ahead and mitigate those. So the first one I want to talk about is network security monitoring. Now what I've done is I've thrown HabEx up here as a quick example and some of you are probably thinking well that's wire shark data, what is wire shark data doing in this talk and realistically after dropping 11 vulnerabilities on stage I feel we've earned the right to talk about basic security shit. So when you look at an ICS network it's really easy to map it out. So pre HabEx this second piece of malware ever to specifically target it. When you look at what it does on a network it's very easy to see it looking for ports and protocols and things that don't normally exist on that network. If you're inside of safety critical systems or inside of an ICS you should know your network traffic and it's very easy to spot these things. So from a safely capturing environment especially for the recorded aspect when there's my ICS sisters and brothers watching in about what do we do in our facility what can we do today without waiting on the vendor. And go safely capture data out of your environment. One of the things that I dislike is there's an atmosphere in the culture of don't touch it. If it's not broken don't touch it. If the lights are on and the oil is pumping please don't touch the network.  So as we mentioned during the press conferences leading up to this there's high confidence that there are folks there's adversaries that are interested in that kind of thing and we need to go collect the data. So again especially for the ICS folks tuning in we want you to make the infrastructure better. This isn't about shaming vendors this is about saying we have a problem in the community and we can fix it. This is about raising that awareness. So throw up some tools in the slides you can easily go through and look at it there's YouTube and defcon and black hat it doesn't look that simple right and in terms of malware on the network yes and the pre have actually have that nice little heart beat but post have actually don't but in a static environment it is really easy to see things like firmware modification.  You do not have to be a trained security professional to do this. The interesting part is it's very very similar to a control system engineer system architect process guy. When Aaron and Colin made this comment before and black hat and that is that the same skill sets that these folks have looking for abnormalities anyways around alarming conditions and things like that inside the environment is the same skill set it takes to look for abnormalities of the network. It's just different sets of data that you have to look at. So today your ICS engineers and architects go back to the previous one just for a second any engineers in the room excellent right so the point here being we're talking about doing change control using wire shark it doesn't have to be security stuff but you should know if anyone tried to change the firmware not just sign a piece of paper that everyone in the facility agrees to not change the the firmware unless they go and speak to Ted or Scott Adiva or whoever it is to use the wire to see if these upgrades are happening at times they shouldn't be and I think there's something to be said for merging the idea of some security monitoring and your change control processes that already exist in the plants all right thanks and so we're not going to go through and say like here's how to defend every aspect I know that would probably be kind of boring for the audience and the environment went through this up here kind of as a joke when we're looking at security inside of ICS it takes a whole lot of resources for an adversary to actually break in and do something neat inside of an ICS when you think of that power plant or that nuclear facility they're all unique there is no just one vendor takes control of one area or there's no one network configuration it evolves and it grows to do to even break or steal data or do anything interesting inside of an ICS so if we just do our jobs inside those networks I truly believe that ICS isn't this super vulnerable thing compared to IT I think an ICS network is one of the most defensible networks that we have and we can actually do that security well and to add to the kumbaya flavor yes IT and OT or operations technology need to start working together better with OT folks and break down those barriers that are required to go out and identify these things because all of us want to have secure infrastructure all of us want to bring that legacy and I think that's really where we want to focus the end point of the talk is saying I'm ashamed really when we look at these infrastructures that we're leaving behind that we are not doing a better job of security that we're not actually developing things out for critical assets human life trains power plants we are ashamed we understand the cost of failure in society of banks we know how much money to invest in banking security but we don't necessarily study the cost of failure of industrial systems infrastructure and we don't necessarily think about the long-term cost of buying things that can be OEM and not patched when we need them to be some time in the future so we are ashamed and we would like you to be ashamed as well we can all help contribute to this whether you come from a web pen testing background we've shown the web vulnerabilities whether you come from a strong reverse engineering background this is firmware you can start pulling apart the binaries and all the rest of it this is something we can all contribute to help improve essentially the infrastructure that's keeping the lights on in here that allows we don't mean this in like a def con you should be ashamed of yourselves kind of way we mean think about the infrastructure that was left to you right you were left a legacy of road infrastructure of train infrastructure of telephone infrastructure that enabled all of us to come into this room and talk about hacking and security and culture and technology we were left a legacy now when we're screwed so we want you to reclaim the word legacy we want you to treat industrial infrastructure as infrastructure and dream of a world where you leave your children a legacy of secure functional infrastructure


The story so far. I am attempting to put Fusix to the ESPA266. I have nearly all of it working except putting the file system on the internal NAN flashes achingly slow so I'm trying to make the SD card interface work and I'm having endless trouble. It looks like I'm correctly sending bytes to the SD card but the SD card never responds. Now since last time I had a thought which is I am using the SD card reader from the MSP430. This is a device I ported Fusix to several years ago. Now that worked. So why not simply hook up the MSP430 which I still have, plug it into the SD card reader, run it and then do a logic analyzer trace. That will show me, assuming it's still programmed into the device's flash, that will show me a successful SD card initialization which I can capture in the logic analyzer. Then I can compare that against what the ESPA266 produces. So I dug out the MSP430 board and completely to my surprise I actually managed to find the SD card for it. So in fact we have a complete booting system. If I hit the reset button here it is booting. So it looks just like it does on the ESPA266. The version number is somewhat advanced over this. This is years old and is rather bit rotted. But you can see that it's detected the SD card. There are three partitions. It's mounted the root file system. It's started in it and we have a shell. And in fact it is a completely working system. We can run programs and everything. So if you want to play Hunt the Wampus it works fine. We've got all the standard physics utilities. They all work. There's the fourth interpreter which works. And you can see that the speed is it's not instant but it's perfectly usable. The MSP430 has 64k of RAM more or less. The device I'm using is a bit weird. It has a few kilobytes of genuine RAM and the rest of it is F-RAM which is a strange form of non-volatile memory that I've never really seen used anywhere else. It's got the performance of S-RAM yet it's it retains its state when the power goes off. So in fact the device has no flash. It just happens to have the kernel loaded into F-RAM. But yeah that works fine. So here is the logic analyzer. Let's set this to something reasonable. Set this to something very big. Hit the run button then hit the reset button. Run. Reset. And leave that something came out. Yep. Stop. Did we get anything? Hmm. It is plugged into the logic analyzer. Yes. I'm looking at the wiring. Let's try that again. And hit the reset button. There we go. Oh I just wasn't quick enough. So let's set this up to 200 and do that again. Okay. That's it booting. So let's find the boot sequence. That's a command 62. Hang on this all looks very strange. Very strange. This is the reset command. This is the 0 4 blah stuff which is the actual command byte. Then we have 4 zeros and then we have the 9 5 of the checksum byte. So I think it samples on the leading edge. So 0 down up 1 down up 0 down up 0. Yes. So that should be going to Mozi bits here 0 1 0 0 1 0 1 9 5. So why is this peculiar? This looks wrong. So I can tell by looking at the what's happening on the different lines. This is chip select. So it's search the chip by going down. Then there is the command.  Then we de-assert chip select and prepare for the next command. Why isn't this decoding it? I mean the cards responding. Is this correct? Active low. Yeah it is active low. The clock phase wrong. It's interesting that the actual clock isn't grouped into bytes. This is reading bytes I believe one at a time. So I think the... Well that's inverted the clock polarity. It hasn't helped phase. No. Both. No. I am more or less just changing stuff randomly to see if anything pops out. I am wondering whether some of these lines are actually inverted. Which is why they are being decoded incorrectly. Word size has to be 8. So chip select is asserted. 1 2 3 4 5 6 7 8. 1 2 3 4 5 6 7 8. So it's reading bytes counting from when chip select is asserted. But the clock is ticking continuously. So why is the SD card responding to that command? Okay I was expecting to see something that made sense here and I'm really not. I don't understand why this is working and the decoder down here doesn't understand why it's working. I've got lots of samples. There's no aliasing problems causing a bad trace. I don't think the SPI stuff is configurable but let's just chop off the SPI decoder and actually add our own SPI. Clock, MISO, MOSES, yes. Right this is in fact the same as we were seeing before. So nothing wrong there. So this this zero should be the first bit of the command packet. So this is the MSP430 code. Set our slow, actually set a constant clock speed. Disable interrupts, MSP1st. I'm not sure how to configure that actually in this. I can configure it in the SD card layer. MSP1st. Here is the clock configuration to set it to fast or slow mode. Interestingly, SPI mode zero here is, let me see, KPH. I don't actually know what that is. Sync mode, clock phase, clock polarity. So on the ESP8266 we are not setting either of the mode bits. Of course the mode bits are unlikely to have the same meaning on the MSP430 as it does on the ESP8266. And I am still baffled as to why this is not decoding correctly. It seems everything is one bit shifted. You can see this should be 4-0 because the zero that's on the end of FE should be part of this byte. Well this is at least, if I ignore the decode, this is a logic trace that seems to be correct. So the next thing to do is to create a new session. Let's rename this. Apparently we can't rename it. Oh well. So to create a new session, swap out the board so we have the ESP8266 back in again and do another trace and then compare the two. So I'll assume this is right even if it's not decoding correctly. Interesting. Okay so I will do that offline. Okay here we've got that trace. So this is the ESP8266. This is the MSP430. Now one obvious distinction is that on this MOZI is left float high when it's not being driven. But if we look at this, this is our command. So CS goes down. Let's actually add the decode shall we? SD card SPI mode. So clock MISO MOZI chip select. Yeah okay. So this is decoding correctly. So here is the command. Groups of 8 from the clock. Not sure where there's a pause between each one. That's worth looking at. So that we start here with 01 and we start here with 01. Now let's look at the details of the clock phase. So you see this gets asserted high as the clock goes low. Which is exactly as happening here. And then it drops low again at the end of the next clock. So that's that is equivalent. And again at the end here is our 95 byte. 0 1 0 0 9 5 sorry I'm counting this incorrectly. 1 0 0 1 0 1 0 1. Yeah it's decoded here you can see. That's on the MSP430 and here on the SP866 we've got the same. I did as of yesterday figure out how to get MOZI left floating high when receiving bytes. So that works. This is where the chip responds. And no response is seen here. Okay so what are we doing differently? Well if I put a marker there. Can we get a length between them? I'm not sure. Ah here we go. I don't need this marking. See I'd rather like to be able to count how many clock bits that was to make sure it's the right number. But I do know for a start that this is on the lower edge of the bit that's here. Now the clock speeds are very different. So this is taking 150 milliseconds to do. Let's actually put that to. Can we put it to there. And we want this to be the last leading edge of the first clock, trailing edge of the last clock. So in this here is our first clock. We want it to be there and there. So on the MSP430 this is 115 microseconds. On this it's 156 milliseconds. This is because we've set this to the incredibly slow mode. The MSP430 should be running at by default 400 kilohertz. Okay well let's try and do something about that. Let's set this. Oh yeah I did actually figure this out before. 200 should be about the right number. So let's burn this. I've tried this it doesn't work. Here we go. So let's rescan the button. Stop. Okay now this looks very strange because we're getting bursts of clocks at very high frequency with long periods in between them. I can add a timing. We want this to be attached to s-clock rising edges. If we go over here and do the same thing. Not that one, that one. Delete some of the things I added by mistake. Connect this to s-clock rising. So the time for a single cycle is two and a half microseconds or 400 kilohertz. So the MSP430 is doing it right on. Here it's faster. Okay I thought I got this right but apparently I haven't. So 160 should give us 500 kilohertz. So let's just see if that produces the right number. So here's one of our clock words. That's hopelessly wrong. This 40. I wonder if that's related to the clock speed which is defined in here. Let's put that back to 80 shall we? That should restore the peripheral speed to 80 or not. That's completely garbled. I need to clean. Okay well I've broken the UART stuff but I do at least now have a trace of this stuff and I do in fact now see a command zero go idle. Oh yeah we were seeing that before. It was the MSP430 that wasn't decoding properly but this speed's wrong. Okay now I do not know how this works. These speeds do not appear to apply anymore. So let's just pick something by trial and error. That should be quite slow. So that's 800 kilohertz. According to the ST card spec we need about 400 preferably a bit less. 700. I am suspecting that my tinkering with the clock speeds has broken everything. So let's just take that out shall we? That's still way too fast. Okay let's just chop this stuff out. I'm going for globals.h. We want this to be 56, 56. Hopefully the UART should work now. Okay we've still got four megahertz for the clock. I did not touch this code. Still four megahertz. I am not convinced that the clock is being set correctly. Is this code at all right? Well I can I can set it to very slow mode. So let's just see if that makes a difference. Very slow mode is indeed very very slow. So here are my clocks full of glitchiness again. This is what I was seeing the first time I tried to do this but then it went away. So the decode all fails. This is mysterious. I have by the way got a second ESP8266 board coming in case it turns out that that was what the problem was. Interesting. Very interesting. Let me just try power cycling the board and doing that again shall we? That's not different. I was wondering whether I might have incorrectly set up the clock. So this is the this is the code that was setting up the various PLLs. So the peripheral frequency of 80 we set it to 88 and we need to set the second one to 91. So so for 80 we want to just set the clock frequency function. Don't bother setting the overclock bit. Change these. What have we got? Sane looking and I see that this is decoding correctly. There's still a long way apart. I wonder if a lot of the trouble I'm having with performance is just that running code out of flash is really slow. 0 1 0 0 0 0 0 0 yes. But 500 is still a bit fast. What's this set to 160 which is in fact the right number. So let's try 200 400 which is approximately what we're getting here. Okay. So 200 is indeed 400. So we are now more or less correctly setting the clock speed. I don't like these very long delays. The SD card should not care frankly. You see that's nearly a millisecond between bytes. What is it? It could be in fact it probably is that if it's my debug tracing was causing the problems I'll be very upset. Okay well it failed to find the card. So okay here are our bytes closer together. That's much more what I expect. However the card is still not responding. Well it's now operating at 400 kilohertz. So the timings we see here are fairly similar to the timings we see here. So if I set my cursor to last clock down there, first clock up there more or less. That's in the middle of a bit wanted to be well there actually. So 175 microseconds, 114 microseconds. This is a little bit faster but not much though. This is 400 exactly. This is 400 exactly. Oh right there are more gaps here. All right so what else could we be doing wrong? Well a chip select is asserted and then we get two bytes worth of zeros. FFs. Then you can see them here. Well 15 bits worth actually. Well here we're not. So that's easily testable. When the chip select is lowered we want to transmit two sets of FFs. This isn't in the spec anywhere that I could find but let's try it. So chip select goes down. We output two FFs. Then there's the command. Now what's different here is there's 15 FFs. That's 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 and then the command starts. Here there are 16 two groups of 8. Do I want to try and write out 15 bits? I mean I might as well. The code to do so is straightforward. So this is the number of bits minus 1. Output buffer. Yeah. So chip select goes down. 15 bits go out. Then we get our command bytes and you can see that the decode here is now showing the incorrect command 62 we were getting over here. I mean to the bit. The card is still not responding. So what else could be going on? I did find some references to needing pull-up resistors which I have enabled but maybe maybe they need disabling again. The biggest difference I can see is the way that Mozzie gets de-asserted between bytes. But the card should only be sampling on the leading clock edge. So when outputting data, the microcontroller on the on the trailing clock edge will set the bit and then it will leave it until the next trailing edge where it will set the next bit. The card waits for a leading edge on the clock to sample the bit. So this means alternatively on each half clock we set the bit, sample the bit, set the bit, sample the bit and so on and everyone's happy. So this is what the clock phase stuff is all about. However here I can see that this is exactly the same behavior. Trailing edge is when Mozzie changes and the card will be sampling and leading edge as usual. Okay I did see that reference to the SPI mode bit so let's see if I can figure out what that meant. So these are the two bits that control the clock polarity stuff. Okay pH, zero. Data is changed on the first u-clock edge and captured on the following edge. That's not the behavior we want. We want this one, data is captured on the first u-clock edge and changed on the following edge. So this is what this is setting up here. This one I'm intrigued about. Three pin SPI. I think we are on three pin SPIs. These three pins. Clock, Mozzie and Miso. I'm not sure what four pin means. Let's have a look. Yeah this allows multiple devices on a bus to originate clocks. Normally on the SPI bus the master emits all the clocks and the slave returns data on the clock pulses emitted by the master. So I can tell by the the traces that we are actually emitting the clock bits in the right place. So this is one, two, three, four, five, six, seven, ten, 11, 12, 13, 14, 15, 16, 17, 18, 20 bytes. So that's 160 clocks. That should be sufficient to initialize the chip. And in fact, if we go over here, this stuff along here is emitting exactly the same 160 clock bits because this is coming from the same code. The card itself is being powered off the 3.3 volt line, which is correct. It's a 3.3 volt device. If I peer at the MSP430 board, the only power line is just labeled VCC, which I believe is 3.3 volts. There is a 5 volt line next to it, but that's not being used. So I think we are powering it off the right thing. It would be hideously embarrassing to discover that the card was not actually powered up, which is why it's not working. Well it's worth checking with the voltmeter, so I'll just go and do that. Well that was worth checking. It was in fact connected up to the 5 volt line, but I don't think that makes a difference. Let's hit the button. No, it doesn't make a difference. I mean SD cards are safe to use at 5 volts. So here's our capture again, set to 5 volts, and it is not at all different. Initialization, command goes out, nothing comes back on me so. So what could be different? We're getting our bytes in burst, but that shouldn't matter. It's an asynchronous protocol. Well I do not like this piece of code. I know it was appearing in the MSP430, but I can't actually find a reference to it in the spec. And if I look at the actual initialization code in the physics source, we can see here, so we raise CS, we switch to slow mode, we send 160 clocks, we then send command 0, and if you look at what command 0 does, so we raise CS, where CS was already raised, this is happening in this gap here, we send 8 clocks, which is this one here, we lower CS, which happens there, and then we get this. If it's not a command 0, then wait for a FF byte. That, I think, is what's happening here on the MSP430, but it's no longer happening here. So let's just see if there's, this is something new, change 7116, how this works, support SD adapters without pull-ups. Let's see, right, so this only does, this change only does the wait if it's not command 0. And we are indeed a, we don't have any pull-ups because I just turned them off up here. One thing we could try is just changing this back, but I think that's basically what we just did by modifying, by modifying lower CS. I do need to be careful changing this code, as this will apply to all users, not just us. Well that still did not work, but anyway, you can see, uh, chip select goes down as a pause while we think, then we clock out 8 FFs to try and read in a response, then we write the packet. Let's put that back the way it was. I will admit that I'm rather surprised at these gaps. This seems to be code being executed in between bytes, and this device should be a lot faster than that. So I'm wondering if there's something wrong with the NAND cache code. I mean between here and here, let's turn these back on again, is 64 microseconds, and we know that we are running at 80 megahertz. So that's 5000 clocks. I mean it's not doing that much work, so I'm wondering if these things are wrong. Well we can always try, I know the second parameter here is speed, so let's just try randomly changing it. So this gap here was 64 microseconds, so let's try this from here. Interesting. 28 microseconds. 17 microseconds. That's faster. I can't honestly tell whether the gaps are noticeably smaller, and I cannot seem to duplicate a session, unless there's a way to do it here somewhere. Apparently not. I miss programs that had menu bars, and a user interface you could understand. So what's this one? So we're at 17 microseconds, still 17. Yeah I think this is a mode enumeration rather than a any kind of speed thing. See SPI flash config seems to be a frequency value. So I don't know why this would be four, because SPI interface is usually is usually either zero to mean the the main SPI, or one to mean HSPI. Well SPI flash config is 4568. Of course now we have to wait for it to find all the fours. That's better. So this is comparing A2 against zero. One, two, three, four. So five would fall through. Yeah I have no idea what that does. Now the other thing of course it's worth doing is just to turn this code off completely, and then see what the timing there is, because maybe the default value is the one I want. 32 microseconds. So setting it is actually faster. So we might as well leave it as that. The other thing that can be done is, so in this code, we're waiting for the interface to stop being busy before and after every operation. So this means that we're never doing any transfers in the background. Now when transmitting, we don't need to wait.  When we change clocks, we do want to wait to make sure that anything got sent. Likewise, when we raise or lower CS, when receiving, we need to wait until the transaction has finished before we can read out the value, not that we're getting anything to read out. This code here can be dramatically optimized by using the bold transfers. So that doesn't look any different. Yeah, still 17. So that's not really helping, but it's at least something. Well, once again, I am stuck. I cannot think of anything that could possibly be wrong here. My great idea of comparing the MSP430 trace against the ESP8266 trace hasn't really come up with anything useful. I'm just looking for any config things. It's just got just config sd. Yeah, this, by the way, is a really good set of documentation as to how to initialize SPI devices, which is a horrible mess. So let's just work through this to see if we're doing it more or less correctly. So software, here we go, power on, set SPI clock rate between 100 kilohertz and 400 kilohertz. We're at 400. So set di, that's data in. That's what the card calls Mozi. Di and CS high and apply 74 and more clock pulses. So Mozi high CS high. Now Mozi is dropping between bytes, but that should be fine. The card will enter its native operating mode and get ready to accept native commands. SD cards support two different command sets. There's the native ones and the SPI ones. We want to use the SPI ones. The native ones allow parallel transfers and the faster. So send command zero with CS low to reset the card. Don't know what this sentence means. I think that should be a when. If the CS signal is low, the card enters SPI mode and responds. Well, we're never setting CS high. So if the command gets there, it should be sampled. I do wonder, do we need to try and make sure that Mozi stays high throughout all of this? If so, I wonder how. But yeah, this is the point at which we are stalling out. So I don't know. Wait for one millisecond. Well, the card is permanently powered on, so that shouldn't be a issue. Do we need to make the clock slower? I don't think so, but it's worth a try. So if I set this to like 250, it should be even slower. I mean, very slow mode, it still doesn't work. So that is 320 kilohertz, which is between 100 and 400. This indicates the speed that high speed mode should be at, which wants to be 20, 25. So we don't want to run it at 80 because that's too fast, but 10 should be fine. We could probably use four to be even faster, but let's wait until that bit actually works. So I changed receive byte to work in duplex mode. So what this does is it, rather than just toggling the clock and waiting for data to come back, it writes out the content of the buffer at the same time as it reads in. What this ends up doing is here, where we're waiting, here's a better example here, we're waiting for a response. Mozi goes high. If we did this in simplex mode, then Mozi would be low, which we can duplicate by doing that, like so. So here you can see that both the init sequence and waiting for the command response all have Mozi low. I thought this might be upsetting the card, but it appears to make no difference. Now, with regard to keeping Mozi high while between command bytes, I am not sure if there's a way of doing that. Possibly this Mozi delay thing might help. This appears to be slave related. Mozi signals are delayed by this number of cycles. I don't think that's what we want. So this is the clock configuration. If the top bit is not set, then it says it's derived from the 80 megahertz clock, which must be referring to the peripheral clock. So we have multiple complicated dividers. Okay, user. We want the read data and write data phases. SIO is when they share the same pin, which is not the case. These are for complicated parallel signaling systems. Byte order. We always want to send the most significant bit first. User one contains the length of the various phases minus one. So for us, it's eight and eight, which means seven. And because we're using the Mozi phase in full duplex mode to receive, we set Mozi in both cases. We can actually put this up here. User two contains the command value, which we don't care about. Write status. This allows you to select which of the three devices is used for the chip select. This is all for slave configuration. Likewise, likewise, likewise, 64 bytes of data. Let me get down to the UART stuff. Yeah, I am once again stumped. I do wonder whether the problem has been caused by either dropping Mozi between initialization bytes. Can we do something about that? Yes, we can actually. So all we need to do is our transmit actually, and I am going to put, I'm going to put these back the way they were because I want to copy this code and stick it in here. So sdi, sbi, raise, cs, put cs up. We want to write out 160 bits. So 159. This is going to be 32, 64, 128, 160. So that should be 160 bits of ones sent out in a single chunk, and we wait for it to finish. And over in devsd.c up in devsd discard, which I never got around to loading. Here, we just want to disable this slot. Don't think it would make any difference, but let's run that. It didn't run. Is this hung? It's hung. Fantastic. 16 megahertz, 200 million, run, reset. Well, here's our init, which did not do what I wanted. So that's just wrong. So I told it I wanted 160 bits and filled the buffer with ones. We did get 160 bits worth of delay, but nothing showed up on the output. I never configured the clock. So here are 160 output bits, and then nothing happens. So that made no difference. I don't think the grouping is, in fact, important. So let's put this back the way it was. Yeah, I've got no idea. So I'm going to have to go away and try and do some more research to figure out what's going on. I'll be very disappointed if I have to call it a hold here. There is some work that can be done in configuring the TTY. I can avoid needing to wait for the flash, but it's still not right. It really isn't. And I still think it's running awfully slowly, so maybe some work needs doing here. We have 64k of code RAM. We want to use half of it for user programs. The other half of it is the kernel, which is 48k. So it's too big to wedge into the remaining 32k of code space. I might be able to put all the discard stuff into the flash. Then we can run all that, and then the rest of the kernel runs in RAM. That's worth a look. But I don't think there's that much of it. I don't have a useful way of getting the size of functions. But we can, for example, put all of start.c, which can all be discarded into discardable memory. I wonder if I copy this over, and here do start.o, start.o. I don't want this anymore. I cannot find start.o. Just want to be just start, which is there. Maybe it does need to be like this. I don't think so. Okay, so let's do, okay. So now if I look at the map, look for discard start. This is appearing first in the ROM, and discard end is here. So all this stuff can be discarded. It's 64 rebytes, which is one and a half k. Okay, so devsddiscard.o, so that's up to 93b. See, I very much doubt whether we'll get 16k's worth of stuff here. So we could move some of the platform code into a discardable resource, but frankly it's going to be about 3k. That's not going to be enough to make everything fit within RAM. One thing we are doing is that all our read-only data is in code. So we could put that into RAM. However, this means that we then run out of space, because we run out of RAM. I don't know how big the read-only data is, but let's just do... The problem is that our RAM is pretty precious, rather not use it for read-only data. ROM nought, data start, a90-4a5, one and a half k. So 3k of code plus one and a half k is four and a half k. We can save 3% by putting the 3% on the code size by putting all the read-only stuff in RAM, because then we don't need to use the 32-bit aligned accesses for everything. But I think it's still not enough. The kernel is just too big to fit into 32k, even with the fairly dense code this thing produces. So I do know that the Dara library is, I think, it's 7k, so it's not small. No, no, it's smaller than that. But we would still need it. And we will need some extra code to make the TTY work. Yeah, I'm going to have to go away and do more research. Well, this makes two episodes in a row where I haven't actually done anything other than fiddle with the SD card stuff. I think next time I will try and make the TTY work, I can avoid horrible delays by simply configuring it to use the shell as in it. In fact, I can do that. So I just need to change this to bnsh. And then when we run it, let's try using the right comment character, it will eventually fail to mount the root because hdA is the flash dev init is here. This should be registering a block device. Interesting. Well, if it wants hdA to be the SD card and hdB to be the flash let it. No, that's not right. Why isn't that working? Well, let's put that back the way it was. I haven't corrupted the NAN files flash again, have I? I think I have. That's not brilliant. Well, I'm going to go fix that. That will involve reflashing. It's not a big deal, it just takes a while. But once that's done, I should be able to do some work on the TTY. And that means once the TTY works, you can actually interact with the system. It should be finger quotes done. There's some tidying that needs doing because there's a fair bit of hackery in here and cleanup. The biggest missing thing is the SD card. The system running out of flash is just not usable. Yeah, well, I hope you enjoyed this video and please let me know what you think in the comments I suppose.