Thank you. So I guess we can just jump right into it. This talk is about scaling version control systems and if you're in here you're probably in the right place. A little bit about me. My name is Ben Carrow. I'm the SysAdmin, the primary SysAdmin for Mozilla, primarily dealing with version control systems. I interact with a lot with the release engineering team and this is a really old picture of me. And so what's this talk about basically? Well it's pretty simple. It's about version control systems but it's about scaling version control systems and it's about some of the stories and some of the examples from trying to do that. So it is primarily dealing with Mercurial. That's the main system that we use at Mozilla. We use a lot of others but and I'm gonna talk about more than Mercurial this session but that's basically it and as you could guess there's probably some Heisenberg related activity here. So there's headaches and Heisenbergs and there's getting kicked off of GitHub for excessive usage which I didn't know could happen but apparently it can. So a little bit of background on this. We'll start with some statistics. Like I said we primarily use Mercurial although Firefox OS uses GitHub a lot for development. Let me see. We have a couple thousand repositories many of which are unique and likewise we have a lot of commits but a lot of them are only about one in a thousand are unique. So well 1 million and 32 million I guess. And that leads to a lot of more interesting problems which I can get to in a bit. We do about two terabytes of transfer a day from these systems which ends up being a thousand clones and it's actually a lot more than this if you consider all the smaller repositories that we're serving. These are a thousand fresh clones a day versus several thousand updates and refreshes and things like that. We're our own biggest customer. The release engineering department is responsible for what would now be called continuous integration on all of the systems and we test on over 12 platforms. So several different versions of Windows, recent and legacy versions of OSX, different architectures on Linux, etc. We also use these things. We get quite extensively as well especially for mirroring a lot of our Firefox OS stuff because GitHub doesn't necessarily have the best uptime. The others we host but we don't particularly use them extensively and haven't really had to deal with scaling problems of them. Interestingly, we still have the same RCS repository that's backing the mail aliases file from when we were Netscape. That file just hasn't changed in decades. Bazaar still uses or Bugzilla still uses Bazaar but they're migrating over to Git and our update system still uses CVS although we're trying to kill a lot of these with fire. So next up this is basically as complicated as that I'm going to detail for Mercurial but there's two SSH servers and there's 10 hosts that mirror HTTP traffic. Unpictured is a load balancer that actually balances all the incoming traffic and strips off the SSL going to these hosts. Also unpictured is the NFS server that acts as the backend data store for the SSH servers. Recently we moved the web mirrors to local disk and that's created a pretty big performance improvement and greater availability since we're not just running all of these on identical hosts. So you're saying get to the story. So the first story is about knowing what you're hosting and the perils of what can happen if you don't. So I was just sitting there as you usually do at your best right operator control center and snooping on emails. I don't actually snoop on emails and deleting user files and I was minding my own business and this bug comes in and this bug says I used to host this repo on GitHub but they disabled it since Saturday and I need to host it somewhere. It's sort of big but I made a bundle out of it which is 1.7 gigs. Can you just host this for me? And I was thinking now why would GitHub kick something off that's 1.7 gigs? I've never personally tried to host something that big there but maybe they're just trying to crack down and resource usage. This should have been my first warning. So I go look at the page and I haven't seen this before but this is still up there. It basically says you used too many resources and we aren't going to host your repository anymore. And the funny thing is I talk to the developer afterwards and he can't actually delete this so this is just a cone of shame that he has to wear on his GitHub account for the rest of its existence. So I do it and I'm thinking maybe I'll just defer it for a while and then he responds to the bug. I think that becares the person who said get to Mozilla.org. So fine. I guess I have to do it. I'm being called out. Maybe I'll put it off until next week. But no, I should mention that this is repo being used to maintain our Gitmare. So this is pretty important. Now this is before and quite possibly the reason why I understood the difference between importance and urgency. So conflating the two, I did it right away. I looked at the host and I said, wow, this really isn't loaded at all. There's like a load average of .5. There's a couple of processes. There's 24 cores here with 60 gigs of RAM. There should be no problem for yet another repository that just one user is going to use. So I said, all right, here's your repository. Here's some details. You can upload your code. Let me know if you have any issues with it. So I dusted my hands off and I went away and I thought this was done. I'd never hear from him again, another happy customer, until this happened. And this happened on the very, very morning of April 22nd, which I think is a Saturday, but this guy is kind of crazy and he's European and he's working in a European time schedule. So this happened when everybody else was sleeping. And so we didn't really know what was going on. Nugget started reporting that there were critical levels of swap used. There were availability problems with the host. It was basically bringing the whole thing down.  So I SSH in and I do the repository and I see that it is not the 1.7 gigs promise, but 208 gigs used. So I do a little more looking. I do Git log and I see this kind of thing. And what this tells me is that this is kind of an automated repository that he's using and he's mirroring a lot of the mercurial commits in it. And this was the first attempt that we had to do a multi master set it between mercurial and Git. So he's taking a lot of these huge mercurial shame sets and moving them over to Git. And the problem is is that these these pushes had objects that were about 122 megs apiece. And it was doing multiple at a time. And then so that's fine. As long as they're not too often, but they are kind of often they're happening every 30 seconds or sometimes a minute. And so you can kind of see what's going on here. So this is basically what it was doing with the server. I don't have an htop interface to show you but it was basically like this. I the meme can be in the notes. The funny thing it wasn't the host wasn't very responsive but and you couldn't actually get an interpreter but you can SSH in an issue of command and that still had priority to run, which involves some really conflict like really weird debugging processes. So I disabled the repo I set it read only and then I started hunting for options that I could I could use to fix this. Now coming from a past of being a gentoo racer I'm used to reading nice big man pages looking for tuning options. Still this one was particularly dense that only the get config man page itself is 2600 and one lines and I didn't really know what I was looking for. I knew it had to do something with garbage collection or packing files but I didn't really know what. Additionally, this was happening on a live system. So if I mistune this I could just send the box deeper into deeper into problems. And this is when it was just running on one host at Mozilla. So we really needed this host to be up. So I started hunting and I found some things. I found this, which was the pack window memory operation. And this is kind of useful. It's pretty cool. But the problem is, is there's no limit on it. It defaults to zero, meaning that it can eat as much RAM as it can possibly use for this pack operation. And so this is something that if we're going to have commits pushes every 30 seconds, this is probably something we should set so we don't out of memory the box like we were doing before.  And the interesting thing about this is that it was defaulting to 6700. Setting it to zero disables it, which is good for performance, but it's terrible for disk usage and eventually will be bad for performance when you have to grab all these individual objects. One of the interesting things about this is that it determines when to run the garbage collection whenever you issue a bunch of weird various commands. Like you can issue a garbage collection if you're just trying to do get status on a repository, which is unexpected, at least from me. So we set this down from 6700 to 1,000. And then we r-synced the repository to a spare host so it wouldn't affect availability.  And then we waited. And we waited some more. And then 18 hours later, it was done. And we looked and it was much better. It was 28 gigs. So the pack operations had an effect. It's important to note here that if your repository is just one big pack file that also has a lot of performance implications, fortunately they're not on the server. So it's not our problem anymore. But it can increase clone time for people who are trying to grab it. Fortunately it didn't appear for us after this. And so if we looked at the load graph, we had this problem. And then ever since then, we haven't run into any hardcore availability problems with it anymore. So few. There were no more load spikes. I mean, there were issues because Git and particularly GitLite that we're using for this aren't perfect. But at least we didn't run into any problems like this anymore. So that's the first story of this. So what else do we got? So my next story, I have to go back a little bit into Mozilla's history. And this begins in, you'll see what I mean in a bit, but I have to go into history of managed code and CI at Mozilla. So the year is 2003. GitHub won't be invented for another four years. Just the idea of continuous integration was invented in 1998. And I think there was only one piece of software out there that you could use for it. And it was called cruise control or something like that. But developers still exist. And we had a lot of them. And they needed some way to be able to check their code in and have operations performed on it and run testing. Because with a small project, you can only have a couple developers. And it works fine. But when you're trying to manage 100 developers and the code that's coming in from them and making sure that they'll collaborate with each other's changes, it's a particularly difficult problem to solve, especially with software available in 2003. So we had a CI system like this. We learned about the new hotness. We did not have a light like this, which is kind of cool. But this is GitHub's light. And it's a pretty common project to do if you're at a software org that uses continuous integration. If you go grab my slides and click the link, this is to their build. That's a link to the blog post documenting their build. The way it works is a typical kind of CI workflow. Developers write code. They generate patches. These all get bundled in mercurial change sets. And then those get bundled into change groups. And then you push them off to a server. And then it builds it for you. This is weirdly enough there's a Wikipedia page for these build light indicators. I don't particularly think they need them. And apparently now that is Wikipedia. But I found it interesting that it was there. So anyway, this is why we called it TRY. This was a system for developers to host their code and then push it out to a server. And they don't have to introduce it into the main repository and then potentially break everything in the world. And then it gets pushed. And then it runs on something like this when a developer is not playing X-Plane on it. There probably should be some max in there because we do a lot of testing on max as well, which is another particular source of difficulty for us. So our developer checks some code in. And then it either comes back negative or it becomes positive. And then it's all good, man. They go to a site later. They look at their build status. And everything is fun. The problem comes from the server side of this. And its particular detail is about one of the mercurial implementation details. It's about changes being immutable. For the first probably eight years of the mercurial project, they decided that immutable history was a fantastic idea. And it meant that nobody could ever screw around with a repository and you would have to deal with the consequences of it as an end user. This has a lot of advantages like that. Unfortunately, scalability is not one of these advantages. So in Mercurial 2.1, they introduced this feature called phases. And if something is not in the public phase, if it's in the draft or private phase, you can delete it and not worry about people downstream of you.  Unfortunately, at Mozilla, we had a lot of difficulty upgrading this simply because our continuous integration environment uses so many different platforms that trying to debug a new version of Mercurial on Windows and Windows on ARM and OSX and Linux and dealing with case insensitivity was really difficult. So we were stuck on a version that didn't include this feature for a very long time. So when a developer creates new changes for this, they push it out to the tri-repository, they create what's called a Mercurial head. And that is the equivalent of a get branch. But for this talk, it's equivalent. There are some technical differences like they don't have to have the same root node. But for this, they don't matter. But as you might expect, the people who originally created this system never really gave thought on how to clean this up. That's IT's problem. So you end up with something like this. This is a tri- repository after a little while and it has the equivalent of 29,000 branches on it. So I don't know about you, but I'm not sure that this is an expected use case for version control systems. And it's, I don't know if it's an acceptable use case from the upstream project either. At first, when we told them about this, they didn't think it was an acceptable use case. But I don't know, I think they took it as a personal challenge that they should try to improve this and support this. So now it's kind of becoming accepted. So the way we arrived at this number is basically we have 100 developers, which was true in the mid-2000s. And let's say 30 of them push four times a day, or four times a week. And those people are pretty hardcore. They're pushing quite often to see the results of their changes. And there's 70 that are pushing twice a week. If we sum these up, that's 260 a week, which is about a thousand a month, which is about 12,000 a year. Now the problem comes at 10,000, 10,000 heads. And then the problem looks like this. At around 10,000 heads, some of the operations just start to fail. Now it's interesting because most of these heads have no more operations that happen on them. They just are pushed to once, and then they're left idle. They get checked out once, and then afterwards they don't do anything unless someone's doing some code archaeology on them. So they have some problems but not others. Back when we had the web hosts and SSH hosts running on the same machines, or when they were the same machines, they had availability problems for developers to push to, and they couldn't load the pages to go pull the code either. When we separated those out, only the push problems remained, but the mirrors that are using the local disks on the separate hosts don't have any problems serving us up anymore. So we have defeated the problem with this. The problem was that pushing would take so long, and it would upset so many developers, and basically give them, they weren't very confident in the system that we had to actually create some documentation for them, which looked like this. This was an entry, and it still is an entry in the wiki page that I've blurred out a little bit because I'm going to explain it later. It says if you experience excessive wait times exceeding 45 minutes, please file a bug to IT. And I don't know about you, but 45 minutes is slightly excessive just trying to do a push operation. But this happened so infrequently. It only happened once a year that it was never so fit to put forth the engineering effort to try to solve this. Whenever it happened, we would just, IT would just go manually and fix it. But because this happened once a year, they poked IT, and IT became a grumpy bear because no one around really remembered how to do this, so that I have to go look at documentation. And then it caused release engineering to be a grumpy bear, too, because they had to go cancel all the build jobs that were trying to go check out code that wasn't there anymore. And it made developers grumpy, too, because they had to keep their changes around and push them again, just because we deleted it last time they tried to do it. And so this is basically how it worked for several years at Mozilla. But the problem was, is that the code grew over time. So this is a graph of the source lines of code in the Mozilla Central Project, which is a Firefox browser and Thunderbird email client. And from about 2012, when this was true, to about now, the code doubled in size from about 6 million to about 12 and a half million. So the symptoms of what this kind of looks like is that developers have problems when they're pushing to 10,000 heads. The symptoms are 45 minutes to return, sometimes they never return. Some more details are that the when you SSH into the server and look around, there's one HD serve process running. It has one core peg, and it's just using CPU. There's no S-Trace outputs. You can't really see what it's doing. There's no L-Trace outputs, so it's not making any sys calls or library calls. Killing it yields no nice trace back that you could use to figure out where it was. And if it killed, if it's killed, the exact same thing is going to happen on a, on most subsequent runs. Sometimes it didn't, but we're not quite sure why that, why that was. So this is kind of a Heisen bug. We didn't, it kind of defied scrutiny. We wanted to get to the bottom of it and solve this problem, so we needed to try harder. So what we did was we had to compile some packages. These are painfully rel6 hosts, so we had to have this debug info package, which conveniently was not provided to us. We need to have GDB installed, and I wrote a nice little GDB script that we could use to dump the S-Trace from it. GDB script looked like this. It attached to a process. It ran the BT, which just gave us a system backtrace. It ran the pybt command to give us a Python backtrace, and then it attached from the process and quit. And this mostly worked, and this is what the output looked like. You could ignore the first two lines. Those are just system things that say it is running inside the Python interpreter. It's doing some compares on it, but if you look down below, you can see the ancestor line here, and it's performing some iteration options in there. It's using the iter generator, and it's happening inside the, the branch map function, and it's trying to update cache. So, we eventually figured out that what it's trying to do is it's trying to update the cache. Let me see. Yeah. And with 10,000 heads, this is not very scalable at all. So, now it can go unbler that little section of the wiki, and it says that if a developer, if a fellow developer has canceled their push as in control seed it, then they've just saddled you with the cost of rebuilding the cache. So, what was happening was that if a developer control seed, it was invalidating the entire cache for the repository, and on the next subsequent run, it had to generate that from scratch each time. When you have really huge repositories like this, that can take upwards of 45 minutes. So, then we asked ourselves, why is it updating the cache? And it turned out to be a, or why is it invalidating the entire thing? And it turned out to be a mercurial bug. So, we put this bug in with mercurial back in May, and we've been working with them to try to fix this. It's confirmed status. So, we're still trying to whittle this down to a very good example case. So, what do we do after, because we can't fix this, or we're in the process of fixing this? We filed our upstream bug, and that's good. We switched to a better compression format. This general delta format was introduced in mercurial 1.9, and it provides about a 10x increase in speed and compression for some of the operations that it's doing, like storing metadata. We have to find some ways to change the caching behavior. And the way this has worked now is that we have some extensions that we're using on top of it to be able to manually control this caching behavior. And that's worked very well for us. And then we wanted to plan a new and more scalable system. Since this was good, but this was the solution for Mozilla in the 2000s and not the mid-2010s. So, what we've essentially done is added some duct tape to our wheel. So, you hit that bug for 10 years? Yeah. It was a very long-standing bug. And it didn't actually crop up for us. It would crop up about once a year, and then when we hit it in the head, and no one really thought about it very much, because it didn't think it was worth the engineering effort to fix. And now that we have more than 100 developers, we're hitting this bug every couple months. So, it's actually worth our time to go bother and fix it and create a more permanent solution, which is what we're doing now. So, instead of duct taping our lunar rover and getting coaxing more life out of it, we can go and create something that is more analogous to what's used today. So, this is a new hotness. We're going to use it to replace the old system. It needs more web scales, preferably with some MongoDB. I'm not actually going to use MongoDB for this. It's going to use something closer to a pull request model. Mozilla was avoiding this for a very long time, but we're basically creating this homemade solution with review board and a couple other tools, because we wanted to integrate with Bugzilla and be multi version control system compliant and then have multi master with Git and Mercurial for the repositories. We want it because we can do easier multi homing for more disaster recovery. NFS shares aren't particularly known for working very well across data center, which is something we want to avoid of. This is going to leverage Mercurial bundles, which are analogous to Git bundles. So, instead of a developer taking their changes and then pushing it to a remote repository and it living live in that repository, we've created a custom server that takes their bundles and then pretends to be a server and says, all right, cool. I got a dude. And then it goes and shoves it into an object store. And then we can reference that later and trigger build jobs off of it and then it can live there forever because this is a purpose built object store for storing things like this. And ideally, this should require minimal tooling from other groups. There's a lot of legacy platform code because we do a bunch of testing on like Windows 7 and Windows 7 Pro and Windows RT with ARM and some smaller platforms like I think we're still building on Mamo stuff. So we don't want to have to go change a lot of this code to get it to work there. So this is something that's really important to us as well and is part of the reason why it's so slow going. So in review, know what you're hosting and ask questions until you're certain that you know exactly what you're getting yourself into. Don't host everything together if you can help it because oftentimes your developer can give you something. And if you try to host it on the same server as production code and his thing has a problem that affects the host, then you have problems affecting your production hosts as well and nobody likes that. Don't put all your eggs in one basket. Don't assume the approach you're going to use is going to work forever. Like this approach that we had with the tricerver, it worked for the better part of a decade, but it had some problems that started becoming more aggravating over time and over time we had to replace it and now is about that time and it's run a little long to be honest. And then the last one is you don't live in a vacuum. The Mercurio community has been really helpful for us and they're really communicative on IRC and on mailing lists. So you can just join the channel and say, hey, I have a problem and give them a pay spin or something like that or describe it and they'll actually work with you on it. And oftentimes that can be so far as to go through stack traces with you and tell you to add breakpoints or tell you to actually profile it and then do this interactively with you. And particularly in the Mercurio community, there's a lot of other users that are trying to scale this out like Facebook and Google are also trying to do this and they're creating they have a lot more engine engineering resources than we do. So they're actually working with the code itself and creating a lot of speedups due to faster operations and things like that. So for some further reading, if you're interested in these sorts of things, you can check out the Planet Release Engineering blog at Mozilla. It's a blog planet.  There's maybe 25 people working and maybe 20 contribute to this blog and there's posts every couple of days. This one is the latest and it was only five or six days ago now. And it's a lot of stuff that happens there. The one other person on my team, Greg Zork, he writes about this fervently. His his big thing is developer productivity. So he tries to streamline the whole process. We had a blog post pretty recently about what the difficulty in the pain points that developers go through every time they want to contribute to a project, which is something a lot of us don't think about. Things like you need to have an account here and you need to have an account here and you need to have check out code here and what's SSH and things like that. And it's a lot of stuff that as developers we don't think about. But if we're trying to attract new new people to contribute to these projects and it's something that I think we do have to think about. Lastly, I have a blog up at bke.ro. I write about some things. Sometimes a lot of it's personal, though. So if you're interested, check these things out. This is me. The slides are up here. And you have feel free to get a hold of me if you have any questions. Thank you. Thank you. We actually have quite a bit of time for questions. Feel free to ask anything that comes to mind. If anything does come to mind. This might not be something you've had much involvement with, but. Have you actually found that some of the tools coming out of the Mozilla services group help with all of this process? Like I know that in particular, HECA has been a huge improvement in the performance and general overhead room to play on for servers and processes I've been working on. So any of those things been particularly useful that we may or may not have found because they're small and new? Yeah, definitely. We use a couple of tools from them. HECA is one of them. We use it to aggregate aggregate a lot of logs. And instead of the typical Elk elastic log search Kibana cluster, we're using HECA in there instead, and that's been really useful for us. Additionally, their circus thing, their circus tool, which is kind of like Supervisor D, but a little bit on steroids and a little bit more intelligent and plays nicely with HECA is something that we use for controlling these WSGI processes. Does that answer your question? Oh, cool. Hi, one of the things that you didn't talk about too much was DR and backup. How do you handle those? So the question is, how do we handle disaster recovery and backup type situations for this? And the way we do it is we have incremental backups hourly through the NFS share. It's this big scary NetApp thing. And sometimes we can just and those are for the canonical copies of the repositories so we can wave at that and say this is good enough. Additionally, we have seven copies of it on all of our web mirrors that we can use if we need to use those. And from that, we also have backups that run on those hosts as well. I think we're using we're using Amanda to copy all that off. What kind of storage engine are you using for hosting all these repositories? And have you done any analysis on file systems or what other system actually works better? Yeah, so the question is what kind of storage systems we're using. And if we think that something would be better than we're currently using. So right now, these are just local file systems on the web mirrors. And those are just using XFS without too many tuning options on them. The the hosts themselves, the masters are using basically in NetApp backed NFS mount. We have experimented with and we might go in the direction of using one of Facebook's products, which is called HGSQL. And it actually keeps the repository data inside an SQL database, which is good. Unfortunately for us, it doesn't really solve a lot of performance problems that we've been having recently. But nonetheless, it's still a pretty cool project. Yeah, Megan, I just wanted to ask about what's it called global replication. So obviously, Mozilla, you've got offices all around the world. And you mentioned, you know, with the new architecture, you hope to sort of, you know, do data center replications so you can, you know, hopefully make it easier for those devs to work. Well, how are you going to solve that problem? So back when all of the hosts were just running all of the things and they all had an NFS mount and they all had read, write access to the data and everything ran on those four hosts, this really didn't scale to different data centers. One of the reasons we switched to doing local discs on the mirrors was that we would be able to throw it anywhere. The system that it's using right now for that is just a little clustering system I built for myself that is basically it's literally SSH in a for loop that goes through it and it maintains some persistent SSH connections and it triggers and SSH is in with an identity that then pulls the information and it uses local discs for the storage. So the theory was that if we wanted to have a repos or if we wanted to have a mirror in say the Paris office, we would be able to hand them a small machine among other things. It would serve this and then we could just add it to this list of mirrors and then it could go mirror the data over there and then using some load balancers or some tricky DNS views they would be able to pull from their server. Whoa. Okay. If there are no further questions I and everything's falling apart as your talk suggests. Yeah, I get to thank you. And you, yeah. I'll take both of them. Yeah, I don't think that will.


No, not really. Well, let's see how many people already see you actually revise their mom's text. Oh, god. And where were you? Allie. Allie, can you read out the last text your mom sent to you? Yeah, another ask the video. I'll put money in your mind. You're going to have to be like, anyways, it'll be there if you need it. Oh! And you said? And you said nothing. Good luck to you all, son. Have a great 24-hour broadcast. Love you so much. That is heart-wrenching. And what did you reply?  He didn't reply! Hi, Ashley. How are things with you today? It is very stormy evening here, given it bad for the next two days. And what did you say that? It is like a wee, sweaty face, weird emoji with sunglasses. You know the cool emoji? Mom sent me. Sorry, Mr. Cole. Yes to Lyft, text me when you're ready to leave. And what did you respond? I didn't. We are missing your sense of humor here. Better let you get back to work by LOLX. And what was your reply? My reply was, by wonderful stuff. No LOL or X. At least he replied. No, that's it. Bye.


In a way, music today is a utility, it's a public utility. It's like electricity or like water. Everybody listens to music, everybody wants music. So we can't really act like music is a product in the same way because it has to become completely liquid. Any way on the web you can get music. The direct connection between the creator, the composer, the musician, the band and the consumer or the user as we call them now. That's the future because that's what the web allows us. My connection with the music is going to be embodied in my community around the music. The new friends are making because of the music and other media. My rating, my tagging, my keywords, what people call the folksonomy. Essentially connecting to folks and people around it. And basically I'll be able to use all of the stuff around the music as a social environment rather than taking a piece of plastic and only listen to that and it's only me. And I will make my selections of what to listen to based on what I see in my network. So the entire way of marketing, advertising is going to return to a place where what matters most is other people recommending. That's already the case, right? But the word of mouth will be the word of mouse as well. So that will be very powerful for direct marketing musicians and bands. Music like water is a term of course taken from David Bowie in a New York Times interview. But basically what music like water means is that we're all connected to music just like the house is connected to water. We don't need special permission. We can turn on the faucet, we can play. That's the way it should be for music. If we want higher definition, we want extra service, we want last night's release, we want last night, we pay extra just like at home, we pay extra for the pool. So let's say 10 years from now we're always on 24-7 high speed. So then I don't actually carry the music, I carry the playlist, the data. And even that I can connect somewhere else. Our consumption will be done through the network. And for now over the next five years because we don't always have the network we're going to save some of it like Spotify is allowing us to save 3,000 songs but not 3 million. So that we can listen on the airplane or so. But consumption is moving entirely to the cloud.


We're now at the end of the session for migrating from 8 to 32-bit microcontrollers. So hopefully you've learned during the day the basics of the SDM32 microcontroller, which contains the Cortex M0 core from ARM. You now have some experience of using the SDM32 ecosystem, which is the Cubamex tool with the HAL and low-layer libraries, and the Kion microvision tool where you can edit and debug your code. And you've now done multiple hands-on with the SDM32. So you've had a play around with a selection of the peripherals available to you on the SDM32. And over the free hands-ons, you've gained some practical user experience of assembly, direct register access, and using the add-on libraries of HAL or low-layer. Small but real-life application. If you'd like to find out more information about the SDM32, we have different courses on the MOOC. One of the useful ones, if you're getting up and started, is the Cubamex and the Cube HAL basics. So you can register for this as you did with this current course on www.st.com. And learn more about the ins and outs and how you do various tasks using our Cubamex tool. There's also the SD communities. So there are lots of extra tips, tricks, and hints in the SD communities created by fellow engineers like yourselves to help use the SDM32 in very different applications. So there's a wealth of experience there for you to have a read of and add to this various community. We'd also like your feedback on the session so that we can help improve the sessions for yourselves in the future and for other new engineers starting out with the SDM32. The survey can be done anonymously if you want to, so if you don't want to give your name, you don't have to. The link for the survey is there on the screen. Most of the questions are covering what you thought of the material, what you thought of the hands-on, how it was presented, things like that. We're always open for feedback. We would like to improve the courses to make them more beneficial for people like yourselves. Thank you for listening to the migrating 8 to 32 bit workshop. I hope you've enjoyed the workshop today. Please keep your eyes on your inbox for new workshops coming along in the future.


All right, how's everyone doing? My name is Steven Adair, and I am here as a security researcher from the Shadow Server Foundation also for the day job. I do cyber threat work for a large federal agency. I'm supposed to be co-presenting with Matt Richard, who's obviously not up here. I encourage you, if you were going to throw anything or heckle, to save it till it's 2 o'clock talk. And even if you weren't going to, I'd really encourage you to harass him or do whatever, for him not being here. Other than that, our presentation today is on the whole, a big Adobe vulnerability that was back in, well, technically January, February timeframe, and our presentation is entitled Zero Day Ghost Net with a zero, the Adobe JVig 2 decode disclosure debunkle. And presentation is not as exciting as it sounds, but I find if you set low expectations, it's easier to exceed them. So a quick standard disclaimer, this applies to both Matt and myself. Anything we've seen here, anything dumb, anything smart, it's not on behalf of our employers, doesn't represent them in any way, shape, or form. None of the information in here, since we talked about some targeted attack stuff, stuff related to cyber espionage, this has not come from our places of work, this has not come from other people's places of work. We're not taking any kind of data we shouldn't have in presenting it here. We have it from other sources, legitimate sources. And we have permission from the sources to present on these topics, share, and do everything we're doing today. So we didn't hack anything, do anything illegal, we didn't steal anything, we're okay with this. So the agenda, a quick item, we're going to do a quick background. I know there's a certain number of people here, probably have no idea what the hell I'm talking about right now. So we'll give a quick background into all the Adobe stuff, what we're actually talking about. The inside story being how this really came to light from our end. I don't know the inner workings of what happened at Adobe or how a source fire did all that stuff, or any of that good stuff. We'll tie that right into a timeline, largely related. Do a little bit of the analysis of some of the malicious PDF, why it's relevant, the evolution of them. This is really Matt's slide, so I'm going to talk to them. It sounds like I'm making stuff up, it's because I am just making up stuff. Then we'll go into disclosure, the classic partial, full, no disclosure. We're not trying to reinvent the wheel or do any kind of crazy new debate or new take on it, but this whole incident ended up being a really good example of a good case study. If you want to look at the different types of disclosure options and how it went through all of them, it really comes to a conclusion whether it's good or bad, what's the right one, but we definitely had some interesting results. Finally, before we conclude, we'll take a look at what we're calling GhostNet in our presentation with a zero. GhostNet being the report that was put out by the Information Warfare Monitor in March of this year, and it ties back into cyber espionage. Here's a quick background for some of you who don't know what's going on. I'm not going to go into too long of the details, I'm going to try and speed through these slides since I tend to talk a lot. We're talking about the Adobe Vulnerability and Adobe Acrobat and Acrobat Reader. This is what is known as Adobe's Apple Product Security Advisory, their first one of this year. There's a CV number if you wanted, there's tons of other reference numbers, that's what we're talking about. I'm an expert on JBig2 and that stuff, but it's basically JBig2 is an image compression format that's used in code images so a lot of times you get a PDF with a fax or something's been scanned, like a black and white image a lot of time. It really decreases the size. And then JBig2 to code. And here in Adobe is the reason we're here today, it's the vulnerable function that resulted in this pretty bad exploit that led to this presentation. So the inside story, so how did it all start? Basically it kind of started with rumors. There was some stuff going on, some people mentioning a few things, some talk on some lists, there was some discussions about a new vulnerability. No one had any details. It was just a talk of Adobe vulnerability that's affecting multiple versions, that's a zero day, but really, really, really lacking any kind of details on what's actually exploited. Someone pointed to a semantic page at some point that we found had a write-up that was really vague. The write-up had a mention of this JS001.3322.org. That's a host name on a Chinese CN99 dynamic DNS provider frequently used in malware authors. And that's really all it had. It referenced the PDF, it didn't say it was a zero day, it didn't say it was, and it was very vague. So I was talking with Matt and realized I'd already had a sample of this. And actually, because I get a lot of samples from different sources, looking at targeted attacks, this is like your PDF files, Excel spreadsheets, PowerPoint, Office docs, all kinds of stuff, Windows help files, .chm. I realized I'd actually already analyzed a sample on February 12th, the day of that advisory. So this is around February 13th, 14th that this is like coming to light. So I looked in my DNS monitors, I take all these targeted attacks and throw them in the DNS monitors so we can actually watch where the IPs bounce around so they will change where they're hosted at. This one is already in there, and I could quickly go back through my repository and see, oh yeah, here it is, I found it, grabbed a copy of it. And then I say I'm not technical, but I'm not the reverse engineering type. I don't know the breakdowns of like JBigTutu code. I don't do a lot of reverse engineering. That's Matt's area. We're working on that. Send it over to him. Didn't take him too long to figure out it was a JBigTutu code vulnerability. This is details that were not out there anywhere. We just got it. We just found it. We found it. I mean, someone else obviously found it. We didn't invent it, but we found out what it was. So now we had some information other people didn't, and we started looking through our other samples, started working backwards. We started going through even new ones that were coming in, seeing what other ones referenced JBigTutu code that are also involved with exploits.  But if you open the PDF, it exploits and puts a trojan on your computer. That's probably not what it's supposed to do in a good way. So once again, at this point in time, there is no advisory. There's no mention. There's nothing public from Adobe. We pinged them and kind of said, hey, are you aware of this? I said, what are you talking about? I said, hey, this. And they said, yeah, we were working on that. And that's around the 14th, I believe, 14th or 15th. So we found some more samples. We started working back. Ultimately, we'd come, you'll see in the timeline, which needs to be updated when we found some of the earliest samples. But we found more and more people are looking for, more and more rumors. People didn't know what to look for. It really blended in. There weren't a lot of them out there. This was not a widespread exploit. This was being used very targeted attacks. And even then, the targeted attacks are extremely limited. So people didn't know what to look for. So they just opened up all these Adobe files and it would look just like any other one. So we then became aware of more attacks. So from our sources, we knew that other people were being hit with this. We knew this was, we already knew it was in the wild, but we knew that people were actually being hammered with this exploit. So got a little antsy about that. We're not a big fan of different institutions being hammered with Zero Day. Especially when no one has any information out there to protect themselves. So this is where we opted on the Shattered Server blog to do something we call the partial disclosure.  One, we didn't mention the function. We didn't know any information on how they could recreate the exploit. But we gave them a solution. We said, disable JavaScript. Now you can read tons and tons of people will say, oh yes, yes, we already knew this. I can't remember if we addressed this in the post or not. You could circumvent JavaScript. And I think someone, I don't know if it was immunity or in the Canvas frame, someone actually created something that can do it. But to date, we have never seen an actual targeted exploit that has actually been able to, that actually used a method that circumvented JavaScript. So this solution would work 100% for you, the one that we provided. To this date, we're not... I'm sorry? Well, yeah, that's another issue, I guess. Well, there's lots of Adobe fun out there. But yeah. Another solution that was really good is having, if you had DEP on, having that enabled would pretty much stop the exploit as well. Having that on, it would crash and everything, but it would actually would not exploit the system. Possible also to probably write something that could circumvent that. We still have yet to see anything like that in the wild. So this results in Adobe publishing an advisory, which is definitely a good thing. It's kind of what we wanted. But like I said, they can't put a lot of the details out there, but they didn't put a workaround. And the key component is that it would be patched in about a month. So that's a long time to wait for something that's already been out for a while. So here's the timeline. We mentioned earlier the JS001.3322.org. That's on January 6th. We looked through that same host name, which we hadn't seen before in any of our other attacks on our other files we've analyzed. And found a thread expert. They had a sample from January 6th. And then I think another one on January 12th. We don't know that that's related. But I suspect that it could be. There's a couple of people who said even going back to December, this exploit existed. We haven't found any evidence of that. The first hash from all the files we had, we didn't know the date, but we punched them in the virus total. That date actually needs to be corrected. Matt's not here. I blame him again. January 16th is the first J-Big2 to code vulnerable file attacking file that we found. So we know as of pretty much about a month before, we posted the advisory that were active live exploits that were working in the wild. February 12th, we talked about the semantic post at an advisory. Didn't have a lot of details. We suspect they may have been under an NDA. 19th was our advisory. Same day, Adobe advisory. Then SourceFire put out a blog, which basically detailed what the actual function was. They then got a copy of it. I think they had it in their repository. Yeah. Okay. I did not use J-Big in my post. You're thinking of the McAfee advert log, but the Shattiser did not have J-Big2 in it anywhere. Well, it's incorrect, but thanks. Anyway. The exploit, you can pull it up right now. Feel free to pull up the post. It's right there. Feel free to pull it up and show me where it says anything about J-Big. It doesn't. Anyway. So we got the Millworm proof of concept came out. Someone else put up the proof of concept. Got picked up on Millworm on the 23rd. The SourceFire, they put out a homebrew patch. That was actually something that really helped. They took the DLL file from, I don't know exactly what they changed in it, but they were able to make it so if the exploit would load in Adobe, it actually would not exploit the system. So Adobe, excuse me, SourceFire put out that and shortly after they also put out some clam AV signatures and sort signatures that would help as well. Then also March 5th, we started to see a large increase in J-Big2 exploits. This is a number of different modified ones. We'll touch on that in a little bit. And then finally, Adobe patched shortly after. It actually came in piecemeal. It was one after another for different versions. It took them a little bit of time to patch the different eight, nine and so on and then different OS flavors were patched later on. So this is starting on matte slides.  So some of the key items you probably wanted to add, I'm not going to touch on, but I'll do this best I can. So kind of some of the key components in analyzing the malicious PDFs, especially if you want to start doing like attacker attribution and comparing. A lot of the metadata is pretty useful. There's some interesting information in it. You've got the encoded JavaScript or other attack code. And then you've got a lot of the other object tags that are in there. So you start to notice things that aren't in other files. So all the exploit files may have J-Big2 to code in it, for example. You might have the other JavaScript tags, which media or flash things in there. And you can look for other unusual parts in the PDF, for example. If you see an embedded executable, either in ClearXSword or otherwise, that's probably a clear indicator. This is probably a bad PDF. You start to match when you start comparing them to other PDFs. So what Matt has on here, he said PDFs are probably derived from any attributes. He calls it like the, in his terms, the DNA. There's some parts that you start to see in PDF files that you're not going to see in a lot of others. And if they start matching and you have a lot of matches, they're probably derived from the same initial exploit pack or exploit developer. There are things that you just don't see in the wild or normal PDFs a lot of the time. And then a lot of these also, if you start modifying a PDF like in Adobe, you can keep the original content and start adding onto it. So the old stuff's there. And one of the cute items they always have are these timestamps. Now timestamps can always be bogus, but we touch on this because you start to see this, we see this in a lot of targeted attacks that have to do with Adobe, but especially here too. You start seeing GMT plus 0800, which is Chinese time zone. So that's a pretty interesting item that you start to see when you start wanting to correlate where this stuff's coming from. So early PDFs, in his slides, had one or two payloads. You see on there the metadata at the top. It's pretty unique. All these Xs, XXXE for the email, and then you have Xs and Ds. This is not something that you see very normal, very normally, and then you wouldn't get this on accident. So you start to be able to compare the exploits and the different files out there, and you see a lot of matches. They also use the octal encoding for the JavaScript, which was very large. That's where you have the backslash and then the three numbers. That was a big point that made these files around a little over 800 kilobytes. They're pretty large files. They had, like I said, a lot of unique metadata, and then they had a couple Xs or keys that were the same throughout all the different exploits. And the initial sample that we found, that we initially knew of, was the JS001.3322.org. And this is an example right from it. So from the semantic advisory, this is what we found. We've got the original JS001 from. Didn't have a lot of details. But it did have on here, down on the bottom, you can see the command and control server that it uses. That's how we originally found it. Comparing the virus tool of data before the 12, there were any virus vendors that would pick up these malicious files, but it was not because of the specific detection for this exploit. It was picking up on other things like odd JavaScript or shell code or other weird things during the variation. So starting about February 12, we believe it may have been because of under NDA, MacAfee and Symantec, I believe, started detecting this. And soon after, the disclosure and additional details from SourceFire, others were able to start picking it up and adding better AV detection. And at that point, the files got shared around a bit more. So the first known malware, like we said, was a Ghost Remote Access Trojan. It became the JS001.3322.org on Port 220. So the Ghost Remote Access Trojan is a pretty powerful Trojan that's got a lot of attackers. It's out there and it's open. It's available. You can download this yourself and set it up, but it's very frequently used by Chinese malware authors and attackers. And what they do, it allows them to get on the system. They can run a lot of functions. They can set on the computer as if they are sitting at it. They have full control over the system. It's pretty powerful. And like I said, Symantec mentioned it on the 12th, and then we can see going back to the 6th of January that this very sample, the very same Trojan, existed. Not sure exactly when it first started using the PDF, but we know as of the 12th it did. So the earliest sample we had goes back to January 16th. This is the earliest one we could find. A few people said maybe a week or two before this. They had samples, but we haven't seen any. This is the best we can do. It's pretty much identical to the JS001 sample we were talking about. Got the same author, the same email, the same web. The encoding is the same, the same XOR keys. So it's obviously derived from the same kit, except for here it is nearly a month earlier. So this thing has been around for a while. It's changed hand a few times. And we found some other samples from different attackers that are using it in this particular one. So this is what I'm talking about, the AV detection. This is that sample from virus total on the 16th. It had 5 out of 39 could detect it. What they weren't looking for was things like, it wasn't specifically detecting this exploit. It was flagging on other things. So you were still somewhat protected, but you didn't have very good protection out there, and you weren't specifically protected from this vulnerability. And you can see the file size. It's nearly 830 kilobytes, which we'll see in a minute gets dramatically reduced. So this earliest sample, once again, just to talk about the malware behind it. So the first known attacker that we have with this that we know of, their malware dropped the Poison Ivy Remote Access Trojan. Very similar to Ghost, it's pretty powerful. Has a lot of root kit functionalities. A lot of different things you can do with it. It's also favored quite heavily by Chinese malware authors, but it is, once again, open and available on the web. Anyone can download it. You can go out there and do it yourself. In this sample, it connected to a host name called yo-yo.networksec.net on port 443. In this sample, we talked about how some are a little more advanced than others. This one, when it crashed, it did not open up or provide any kind of replacement PDF. So if someone opened it, they would see a crash and they wouldn't get anything new. Not as advanced. Probably just a bad choice on the attackers part because some of the other samples did do this. But this is also an attacker we've known previously as of December using Office Zero Day. We believe it's some few different exploits. We also had the WordPad text converter exploit. So this is not the first time these guys have popped up with some form of a Zero Day. So they're probably not just some random person testing or playing around. We don't have any information in the back end of their server, but it starts to lend an eye to cyber espionage-related activity. So February samples. This is where the code starts getting a little better. From February 7th and 10th, we start getting some new samples that we looked and see how they start to change. This is a few days before the February 12th sample we had that was identical to the one in January 16th. So once again, working off matte slides here, you start seeing some of the differences. PDF header changes. It has a similar heap spray, the same comments in it. So if you looked at the code, the actual comments in there were the same. They were just reusing it, but now the big difference, they were able to dramatically shrink the file because they just used flake decode. So now instead of having all that gigantic octal encoding in there and the clear, it now is all encoded. So it took up a lot less space. So it went from 830 kb to about 108 kb. So the question is, are these new attackers or just people that took the exploit and are trying to refine it? We never saw any overlap from our initial attackers, the command and control servers to the new ones. So we don't know, there could be new people with new command and control servers, new Trojans or just someone who got a copy of it and started messing around with it to improve it. And then some of the PDFs had multiple versions showing what changed in each revision. You can see some of the differences. And once again, if you take a look at the bottom there, you see the attackers when they started modifying it now have a mod date that involves plus 0800. So once again, obviously it could be anyone could say that to anything, but that's where we are once again. So code got a little better. ABtection did too. This time they flagged shell code signatures for a few of them. The other ones pretty much remained the same for the JavaScript. Files a lot smaller. You can take that for what it's worth. So now we can see the attackers are evolving. So we had the initial encode was improved. We saw the size shrink a lot, and then we saw better attacks. It's pretty standard operation for an attacker with a targeted office or PDF or doc file exploit to have a replacement document open. The initial couple we saw didn't do that. They got a little better. Probably someone who's a little bit smarter had this happen. And still some attackers don't have the exploit. So those that didn't have it were still looking for it. So this is the point at which the Millworm exploit becomes weaponized. So the Millworm proof of concept, the proof of concept itself came out on the 22nd, posted Millworm on the 23rd. You can see this is actually run through all this. The formatting you see here is a tool that Matt Richard wrote, calls a PDF parse. He said it'll make it available. He's not here to speak to that, but I don't know where to tell you to go to get a copy of it. But he said it'll make it available if anyone wants it. So feel free to email him at the end of this. So he scans it, looks for some key objects and the red is kind of what changed between them. So if you approve a concept on the left and then on your right you have the PDF in the wild. This PDF in the wild, we don't detail the attack on it. It's one we're actually very well familiar with. It's an attacker group that's been around for a couple of years, heavily into cyber espionage activity. They actually took the code off of Millworm and they edited it, modified it, and made it work for them. And once again the mod date of 0800 appears in there. They edited it with Adobe. They did some stuff on it with GhostScript. So the attacker actually took it, and that proved a concept and weaponized it. And I know there's lots of other people who were playing around with it, but this is the one that was actually used in real live attacks. So at this point it brings us to the whole disclosure. So this is a regular old debate. It's a long running debate. No clear answer. There's pros and cons of pretty much all the ones that we saw. So we didn't want to disclose. Here's a quote that Matt liked that he put in here. He thinks he kind of liked it too. This is from H.D. Moore. He was actually talking on the Adobe issue from the 23rd. He said, the strongest case for information disclosure is when the benefit of releasing the information outweighs the possible risk. In this case, like many others, the bad guys already won. Exploits are already being used in the wild and the fact that the rest of the world is just now taking notice doesn't mean that these are new vulnerabilities. And he says, at this point the best strategy is to raise awareness, distribute and to patch, especially with the last line that we agree wholeheartedly. I think everyone involved in the disclosure process here was pretty much on the same page with that. So no disclosure was a major fail. We're not here to beat up on anyone, so we're not beating up on Adobe. We've seen a lot of things just from the casual observer improvement that we like. For whatever it's worth, our opinion. So obviously Adobe, they didn't disclose it right away. What did it make a big difference? I don't know, and maybe they kind of worked with more antivirus owners. I don't know the case, but obviously no disclosure didn't help here. And he said the AV vendors, we think some of them may have been under NDA so they couldn't put a whole lot out there. Are there others who could have noticed it or found it or done something? That's a question. So basically in this case from no disclosure, the only ones who were benefiting were the attackers. They were running rampant for at least over a month before anyone even started to do anything about it. And they had another month from there before there was any kind of patch from Adobe. So like I said, we were in favor of partial disclosure and we only give it a partial win. So they had issued an advisory, that's great. What did we get out of that? So Adobe said there's an issue out there, so I guess people know we're not lying. That's the only real plus of it. Potential workarounds, we like that because we had potential workarounds. They could be circumvented, but they were valid, they worked, they prevented stuff from people who were being exploited. So we had a form of fighting chance and we're just complete sitting ducks at this point. And then also it led to the full disclosure which is kind of like a gray area depending on who you talk to. So full disclosure, we said it succeeds. So there's some things we liked about it and there's others we didn't. So Where's Fire pulses the volume details basically a day later and it's not complete and I think they actually, I think McAfee actually let some of the cat out of the bag from one of the presentations that's all in a form of a hex editor, so you could have actually taken that and figured out some of it. So it was kind of out there. Someone would have gotten out there one way or another anyway, but they were able to release an unofficial patch a day later. So we consider that actually a pretty good win. I had no idea how many people actually were able to apply that, but at least like I said there's another solution out there. But then the Millworm exploit came out pretty much the same day and that pretty led to some new detection. Many virus detection started improving at this point. More information is out there. There's more people know what to look for, but this also leads to the quicker weaponized exploit development. So like I said, disclosure kind of leads to new attacks. This is going to happen regardless, but it happened a little quicker we think in some cases. So like I said, the proof of concept code the 22nd, Millworm shortly after. One of the codes references the Sourcefire blog. That would detect some of the attacks, which would later get changed and they wouldn't be as effective, but that's pretty much the cat and mouse game of antivirus detection. So it takes a little over a week from this point where we start seeing derivative attacks that are modified more and more. So start changing little pieces, start doing different things to avoid detection.  I think a lot of people knew the impending patch of firing them out there. We saw a massive increase in the number of exploits coming in. So signature power. So this is some of the signatures. I believe Matt posted these on here. I think they're the ones Sourcefire put out for the ClamAV and I believe this may be their sort signature. So what it does is they look for the a lot of the time they look for the J-BigDudeGode, the stream followed by the carriage return and the line feed. But the PDF spec actually allows just a carriage return without a line feed, a small issue for things that started evading detection. As you can see in the red, that's the carriage return line feed. Same thing with the snort signature. So we started seeing evasion signatures. This is always the case with antivirus though and other things. A lot of them are unless they're fully heuristics based they pretty much match exact known attacks the best they can, which in this case would probably for a while get the bulk of them. They can quickly change it. Like I said, they can use just the carriage return. They can exploit. That would cause it. That would no longer be detected. And then some other things replace the letters with ordinal values or they start finding creative ways or trying to change the case for some things and not detect that. So basically, right now our patch is our main solution. This is a picture of the DOMO-KIN. Every time you open most PDF, God kills the kitten.  So, final tie-in. This is where we start talking about, GhostNet. GhostNet, the report that was put out by the Information Warfare Monitor. So, is there any connection between the malware and the GhostNet report? So that's a good question. Is it cyber espionage or just one in games? Is this random malware? Is it something like, it's really not a big deal. So, March 29th, 2009 this is when GhostNet report came out from the Information Warfare Monitor. When they put out this report, I think maybe one or two of the guys here talked with them, I want them pretty extensively and another one a little bit. What they did is they looked into some of the targeted attacks using a lot of these same type of attack techniques using like different Adobe files using office documents and things that were targeting the Tibetan community and they had some pretty much unprecedented access to the office of the Holy Dalai Lama and what they found in the report, it was a pretty long report, they found a network of systems being controlled by these guys with different attackers. Some of the attacker sets on there, it's not real clear whether or not they're actually related it's probably more than one group attacking Tibetans in the office of the Dalai Lama but they had some pretty good information in there and some of the questions are being raised this report came out in March is what's the relation between what we're seeing now and the stuff that was put out by the GhostNet report? Is there a connection? The best answer is not exactly. I'll put out a question mark because there are some loose ties to it. So to keep in mind one thing is that the GhostNet research was from 2008 so this being a 2009 vulnerability the vast majority of what's in GhostNet report was from the previous year. This exploit is not known to exist in 2008 or excuse me, in the wild exploit is not known to exist in 2008. So our title GhostNet actually with a zero is kind of initially, so Matt wanted to gather his presentation and so I agreed to present with it and to kind of late he just took the GhostNet with a zero just off the bat based on the initial malware that we saw. So it was the ghost remote access erosion which was built with a zero it's kind of a play on GhostNet without it. So we weren't really expecting to find any close ties to it it's kind of a nice cyber espionage is related it's interesting so we just kind of put it in there but we found a loose connection after all so we see from the report there's three different there's several different commanding controls listed in there but there's three of them in particular that we're familiar with so a lot of the attacks from GhostNet we've seen have copies of it we're aware of but in particular three of the commanding controls listed in the GhostNet report mcafferyresponse.org, msnxy.net and lookbytheway.net These are ones that have a loose tie to what we saw in the early attacks in January. So this malware is often called TrojanInfo what it does is it steals different information from the systems it'll rip off files it'll take off credentials, it'll take off emails usually attackers specified they'll beacon in they'll beacon in with the commands that look like this they'll post different data they identify themselves with the computer name and the MAC address and then the attacker will respond back with what they want from the system and I think I actually owe if Martin Van Hornbeek's here because I had the Trojan name wrong so it's actually Info and not Rihler which they're easily mixed up so it steals various information this one's been around for quite a while I think the first mention of it on an antivirus page is sometime in 2004 on one of the, I see the storm center, the sands blog they mention this a while back for Tibetan attacks using this very same thing so this malware is not very common it sides directly back into the ghostnet report this is what the malware looks like they have this in their report so we went back and looked at our earliest samples we actually only had four that we could find that we could confirm were from January the first one we already talked about it beacon to yo-yo network sec everyone that went to mail.nti-mobile.com one that went to any mail that's so web mail.com and then we have the last one in the red search.touracealv.com so this is one where we actually found some of these similarities so these four different trojans are completely different trojans different command and control servers but one of them has a similarity to ghostnet so the first one that we had a similarity was, or the only one really was search.tourace.clv.com not only is this one that had a similarity it's a known command and control to us it's one that we've seen before we've seen it in other attacks it's the endfall trojan like I said this is a very uncommon trojan in terms of total number of IP address it's the only one we've probably seen at a given point in time active less than 10 this is pretty much seen in a lot of attacks originating and coming from China usually against the Tibetan group so we also started to see some other stuff like we said it shares IP address so this host, we have several other command and controls we know of that are on the same IP address it's search.touracealv.com and these are all hosts that we know on a couple of them have been involved with attacking any of the European Union so they've gone directly after EU and European hosts so this starts to be a decently interesting and strong cyber espionage time because this thing is being hosted on the same box we know has been attacking EU entities with different targeted attacks different exploits so like I said search here's from our DNS monitor search.touracealv is hosted at this very moment I've had a few different IP addresses over time at <IP_ADDRESS> Chinese IP address there's a few other hosts blueandnt.com ggsddup.com if you do a lot with targeted malware attacks these are probably ones you probably come across at some point in time this is that Trojan, very uncommon sharing IP addresses with these so just taking some of the file names sometimes we have file names sometimes we don't you can look at some of them having to do with Brussels exhibit imitation suite in public planning for the public procurement these are not like some of them are not run in the mill a lot of them if you opened them they would crash like I said they'd open a new file a lot of those new files had specifically to do with European conferences or things going on in different parts of Europe whether it was Sweden Belgium, other places Germany so these are very in line with one another from across the different domains that were attacking other EU entities so we saw only a thing a lot of them were like jbig2 decode exploits and one of them was an older powerpoint exploit we go back further I think we have some samples that attack like MSO6 different word doc vulnerabilities so some of these have been around for quite a while and they've been attacking quite a number of different things so we started to get to cyber espionage and jbig2 we know that even the earliest of jbig2 samples can be tied back to what we are sure cyber espionage so there's a number of them where there's even a few samples where we've been provided information on the back end we know who's been targeted we've seen the infected host so we know some of these have definitely been used to attack financial systems or foreign governments things like that and many other samples are related or other known attacks like the past screen we're going to talk about in a minute I'll go back to the JS001 another item we've seen from that different groups appear to be operating kind of with the same exploits we didn't really see a lot of overlap in the initial stuff was kind of surprising it seemed at least four or five different groups had the exploit for about a month and they didn't really fire it out a lot like we didn't see massive influx of it either out of people who weren't realizing what they had and it just wasn't getting reported and down through the chain so we're not sure how it stayed so it stayed relatively quiet yet it seemed to be in the hands of multiple different groups so another interesting item is all the initial command and control servers pointed back to Chinese IP addresses so this is coincidence that's possible so interesting enough that JS001 that 3322.org connection how does that tie back to cyber rescue now well it currently sits on a Chinese IP address like we said it's up there that host also has another domain called cyhk.3322.org on it they share the IP address no who cares why is that interesting well the reason that's kind of interesting is there's multiple reports out there one of them is put out by Symantec that in April 2008 there were email attacks against Japanese businesses they were purporting their B they were spoofed from the Japanese government and if they succeeded they beacon back to the cyhk3322.org host name which now also shares an IP address most likely run by the same attacker that runs JS001 our guys that had the one of the earlier people to get the jbig2 to code exploit so this is the look at it took a sampling of about 82 different host names that were involved in jbig2 host names that came as a result of jbig2 pdf files that exploited and took the IP addresses that they bounced around to over the period of time so sometimes some people will change every couple of days some of them since February have been on the same IP address till today as you got a mixed bag but if you take a look at the samples you see that excuse me China 202 of the 301 different IP addresses were there so about 67% you go Korea, Hong Kong US, Taiwan and Japan and it just kind of works down with the intermingling between different countries I mean bad guys or anyone who does this they can hack a server anywhere they can take a credit card or they can buy a server anywhere so it doesn't definitively prove anything once again but we can start to come to some conclusions so obviously our first conclusion is that China is behind all attacks involving zero day exploits and cyber espionage okay you might hear that in the news or something okay that's obviously just a joke so if you're going to quote me on that please add the just kidding at the end but no in seriousness so we know the conclusions as we say there's a group there's multiple groups out there there's people that do it for obviously to sell it or profit there's other people who do it to get what they need from systems so they're making some of them manage to stay pretty quiet with their exploits another thing we need to realize from this we can't fully rely on the security community to identify and protect this from zero day and it's kind of common sense but it's just another item that's kind of highlighted out there there's a lot of people out there looking at the same file like I said if they are getting I don't know how many pds to come in a day to different antivirus vendors but if they're getting like hundreds of them thousands of them and one of them is slightly different like this like I said and there wasn't like hundreds or thousands of jb2 to code adobe exploits coming there's just a couple it's kind of hard for them to identify so I guess it's probably someone got hit with it reported it and that's how the initial detection and people started looking into it adobe and the antivirus vendor so it's not a real good way out there for security community to protect this right off the bat it seems like full and partial disclosure fail us to some degree but they also succeed but what's very clear is that no disclosure obviously was the worst option of all these so if people start to find out about stuff and like I said I don't know I'm not in adobe shoes but it seems like if they could sing something a little bit sooner probably would have been a better solution so then our bad actors whether it's ghostnet or ghostnet our way or anyone they're pretty much busy at work we know this for a fact there's zero days pretty frequently in different file formats that we see they're always looking for new ways around new exploits it's been happening for a long time but they've been pretty busy at work adobe's had a few other issues recently such that real quick on the last slide and another thing is a key item Matt really want to put in your pds carry a lot of useful information so like all that like you start editing it keeps old revisions all the header information the mod dates if you if you can trust the data or it's valid there's actually a lot of useful information there especially if you want to start doing tracking back different attacks seeing how they evolve so for example the one taken from Millworm when it was actually weaponized if you decoded it they actually left in all the same exact same code a lot of the words down to the words so you could see that the weaponized version was without a doubt taken from Millworm so that's what that's what's kind of interesting some of the data all the data that's inside pds and they start adding it with other file they started adding in other timestamps you can see other information you can start kind of trying to figure out who it ties back to so final some of the final conclusions so kind of this it's certainly I think it really served as a lesson to learn how the community a lot of people came together a lot of people argued a lot of people did different things a lot of people pointed fingers but in the end I think a lot of people learned some lessons that helped with the next time so this is not the first time this has occurred it's not going to be the last time I think we've already seen some improvement from Adobe for example the recent flash vulnerability they seem to have got maybe it was easier for them to patch actually but they got some stuff turned around quicker it seems like people are starting to to learn and get some better feedback and response from it all so that is that is our presentation thanks for attending I said Matt Richard didn't make it you can ask him for his tool you can harass him at two o'clock feel free to do whatever feel free to come find me later thank you


That was me, so I like to solve problems, so and I was a good student, but enjoyed math and science the most, so engineering seemed like a good fit for that and I also liked chemistry, so I chose chemical engineering. Georgia Tech was in my backyard having completed high school in Atlanta, so it was a natural choice for me to go to Georgia Tech and my path to NASA was not a typical one. I wasn't always dreamed of working for NASA or always dreamed of wanting to be an astronaut. I had met my first husband while I was at Georgia Tech and he had co-opt with NASA's Marshall Space Flight Center in Huntsville, Alabama, so I was headed to Huntsville and there aren't many chemical plants in Huntsville, so I was fortunate enough to also be hired into Marshall Space Flight Center. Had that not happened, I probably would have ended up in the oil and gas industry, which at the time was hiring a lot of chemical engineers, maybe ended up in Houston. So I was really fortunate, I got in at NASA and initially into the propulsion group but only spent about a year there and found my way to a fledgling life support group, which had just been given the responsibility for the space station job way back in the mid-80s. So it was a really good fit for my chemical engineering because life support systems are chemical processes and so I really enjoyed that and enjoyed starting working on the space station way back in the beginning. It sure is an exciting time. I just came from the Flight Readiness Review for our next crew launch to the space station and we'll be launching next week. So it's a really busy time, really exciting time for NASA and our international partners. Not only are more countries interested in space but more private interest is coming from the private sector, commercial space as well. One of the key missions for the International Space Station is to be a test bed where we can learn about the systems and about the effects on the human body that we will need to do these future missions beyond low Earth orbit. So things like the life support systems but also the effects of weightlessness on the human body over long periods of time and we need to develop those technologies and do that human research so that we know how to keep crews healthy as for the missions to the moon and Mars. So the ISS is a really critical test bed platform and how's it changing over time? Gosh, we are entering our third decade on ISS. Our first decade was all about assembly which we completed in 2011 and then the last decade was really learning how to use and expand those capabilities on the space station to do research. And now, especially now that we have our commercial crew vehicles flying, we're really maximizing the full potential of the space station. So we have more crew. That means we can do more research. We have more partners participating, both international and commercial. And so all that adds up to really maximizing its full potential. Our international partnership is the reason the ISS has been so successful. And we are seeing more and more interest in other countries, not just our regular partners but other countries and nations wanting to participate and even fly their own astronauts. So that's a great opportunity. It's also a challenge because we have limited flight opportunities even with more vehicles and limited space on the space station to do things. But I think as we and other countries expand our commercial capabilities, there will be more opportunities. So one of the areas we are seeing promising markets for in tourism are these flying astronauts from countries, from sovereign nations. And we think that's going to be a promising market for these private companies that want to have platforms in low-Earth orbit and do activities. So I think it's a challenge but also an opportunity. Our vision for a low-Earth orbit economy is hopefully multiple human or human-tended commercially-owned and operated platforms. One day we will retire the space station. It won't last forever but we don't want a gap in low-Earth orbit and human space flight. So we are doing all we can to enable a future where we have privately owned and operated platforms where NASA can purchase services and other countries and entities can as well for what we still need to do in low-Earth orbit. And so we call it, we want to be one of many customers. So we, with the growing number of commercial Leo satellites and constellations, it does increase our efforts to ensure there's no interference with the space station. We work that closely with the FAA and those providers when they're going to be doing their launches to ensure that we stay out of each other's way. And we have had to do some avoidance maneuvers occasionally to avoid some objects but that's pretty rare. So yeah, so far so good. Well I would say it's a great time to be starting a career in space and there's lots of paths to take. Not just the traditional STEM, don't have to be an engineer. Don't get me wrong I like being an engineer but there are all kinds of careers in the space business and not just with NASA but exciting opportunities with our commercial space providers as well. So when I started there were very few women in leadership positions. That landscape has definitely changed and when I started working on the space station I never imagined one day I'd become the director. So my journey has had a lot of twists and turns and I said yes to opportunities and jobs when they were offered and those things led to the next thing. I would say don't try to start with the perfect job or map the perfect career just take an opportunity and look for ways to contribute and find your niche and let it lead you to the next thing. So pretty much everyone I've talked to has not climbed the ladder straight up. It's more of a jungle gym and like me I started out in propulsion and that wasn't really for me and found my way to the life support system area which I made a career on. So I would say just being open to that and look for that first opportunity and go from there. It has been a challenge boy. Initially our onsite work was restricted so some projects were especially the non-critical projects were delayed because we couldn't bring people onsite to do the hands-on work. Critical projects have been remarkably successful though. The teams have found a way to safely continue to execute. I know at Mission Control there are multiple Mission Control rooms and they work from one and clean the next one and do shifts that way. So we've made it work and we've kept this had to keep the space station flying and all of those activities. So the team has really risen to the challenge now that the vaccines available and people are really starting to get vaccinated. I think things will get easier. Some of our centers are going opening to the next stage and allowing more people onsite. So we'll start to see things hopefully get back to a little bit more normal. I would love to sure but I'm also happy at this point to enable others to go. I think at this point in my career I probably destined to stay on the ground. We've got a whole cadre of younger astronauts. So developing life support systems in space like I said it's a lot of chemical processes that we use and it's a combination of liquids and gases and solids which as you can imagine a microgravity behave differently together. So phase separation is more difficult. I'll give you one example. We process urine into drinking water and you might imagine if you were to do that on the ground it would be a fairly simple process. You're distilling urine, you're boiling a pot of urine on the stove and the steam rises and you collect the purified water and the solid is left behind. Well in microgravity the way we do it we still boil urine but we have to do it in a rotating centrifugal use centrifugal force to rotate it so it slings the liquid to the outside walls where it's evaporated.  So everything is more complicated. But we recycle over 90% of the water. We're getting close to not recycling 98% of the water. And on the carbon dioxide that we remove from the air we are able to recover about half of the oxygen back. So recycle about half of the air and we're working on technologies to go farther than that. You mentioned the effect on the human body. We're doing a lot of research on ways to counter the effects of weightlessness. Exercise is a very important countermeasure we call it. And it has to exercise a couple hours a day just to counter those effects of a long-term weightlessness. But we've gotten pretty good at how to do that. You do notice when they land they need help getting out of the spacecraft. You can imagine a mission to Mars where they're in space for six months or longer and then nobody's there to help them get out of the spacecraft. They're going to have to get themselves out. So we have a ways to go to figure out ways to make that work. But the space station is a wonderful test bed for learning. We know that no one country can do everything by themselves. If we're going to explore space we need to do it together and that's been very important to us. And so our partnerships are very critical. So groups like the ITU not only do you provide collaboration opportunities but develop these important interoperability standards. And we've developed other interoperability standards so that different countries can bring contributions and we know that they'll plug and play together. And I think that's really important to enable everyone to participate and it also drives the global commercial partnerships as well. So I think it's a very important role.


You don't know anyone who's lost a pig, do you? Hold on now, hold on. Is anyone here lost a pig? No Donald, sorry. Harry, you don't know anyone out your way, keeps pigs, do you? Huh. I just heard they taste alright. Yeah, I am falling about Sue. Yeah, just a pig. I could have her back to you today.


I am Dr. Mitheshanti, Head of Department and Consultant, Department of Medical Genetics, Manipal Hospital. This department is 22 years old in Manipal Hospital. We are having in-house facility of counseling as well as a high-end genetic laboratory. So for the autism spectrum disorder, child psychologists, developmental pediatrician, pediatric neurologists and genetics, we work together as a team. Autism spectrum disorder is a neurodevelopmental disorder characterized by social environment, restrictive and repetitive behavior. The autism is a multifactorial condition where genetic and environmental cause plays a role in the etiology of the autism spectrum disorder. In the Center for Disease Prevention, CDC incidence of autism is 1 in 59, which is very high. The presentation of autism spectrum disorder is variable. The usual presentation is speech-delay, regression of developed milestone. But it can be identified as early as the third month of life when there is no social spine or there is no babbling at the age of 1 year. The other features are the repetitive movement and social skill deficit, likes to play alone, not to mingle with other children, hyperactivity and more boundless to objects. All those are the peculiar features of autism spectrum disorder. For genetic diagnosis, the chromosomal microarray is first line of investigation. There are more than 1000 genes which are associated with autism spectrum disorder and more than 2200 copy number variation. This chromosomal deletions and duplication has been reported in autism spectrum disorder. The other investigation which is required in the autism spectrum disorder is fragile screening and red syndrome screening in case of a female patient. And metabolic screening if the clinical features are indicated. Gene-level testing, so we have more than 1000 genes which is associated with the autism spectrum disorder, so that can be screened. In literature we know that the intervention if it is given before the age of 3 years it has more impact. Intervention is in the form of multiple techniques together physical therapy, occupational therapy, speech therapy and combination of medicine. This intervention helps in the children to develop their combinations and speech development. And if we identify the underlying genetic cause it can identify the other underlying genetic systemic abnormality. And once we identify the genetic theology it becomes easy to give the recurrences for the next pregnancy and we can help the family for the future pregnancy.


I am Dr. Hemant Kalyan, consultant and head of the Department of Orthopaedics and Sports Medicine working at Manipal Hospital, Bangalore. This is a question that a lot of people ask, Doctor, what exactly is a knee replacement surgery? Now, in a knee replacement surgery, we hope to accomplish three things. First of all, we said that the joint surface has become rough and irregular. So we replace it with a perfectly smooth joint. So if you just look at this model over here, you have this part of the knee which is artificial, this part which is artificial and the white plastic which is in between. So the surface is replaced by this material, top and bottom. This part goes on the surface of the high bone. This component goes on the surface of the knee bone, the leg bone and in between the two we have a high density plastic. So you also have on the inner side of the kneecap a plastic component. So as we see here, there is a combination of a metal with plastic or high density polyethylene which replaces the surface of the knee and forms the new knee. So all the ligaments of the body are preserved. The muscles are the patient's own, naturally the bones are the patient's own but the surface of the joint which was originally rough and irregular is made perfectly smooth and regular. The second thing we do in a knee replacement is to correct the alignment of the leg. We mentioned that in osteoarthritis the person's leg often becomes bow leg or not knee. This is corrected at the time of surgery by balancing the ligaments in such a way that the person gets a nice straight alignment of the leg after the surgery. The way this helps is that when a person stands and walks the body weight is evenly distributed across the knee and doesn't fall more on one side or the other side. The third thing we do in the knee replacement is to recreate the gap of the knee. So as I mentioned in osteoarthritis the gap becomes narrower because the cartilage is lost and the bones become closer to each other. When we put a metallic component on the two ends and a plastic component in between then this will recreate the gap in the knee joint. So these are the components of the knee joint. This is known as the femoral component which lies on the thigh bone. Over here you have something known as a tibial base plate which goes inside the leg bone and in between the two you have the white here which is the plastic component and together they form the new total knee replacement. So this is of course a very skilled surgery and we have a lot of tools including aid with computers, navigation and of course the surgeon's own judgement and eyesight which is the most key factor in giving a well-aligned, well-balanced knee which should serve the patient well.