Okay, please welcome Michael Bebel, who will be giving us an update about system D during the last release cycle, and also what you can expect during the stretch release cycle. Okay, does this work? Okay, great. So, hello, my name is Michael, and isn't it a nice coincidence that the system D talk was scheduled after K3BC? So, yeah, I've been a package system demaintainer for quite a while now, and today I'd like to give you a short overview of what we did since last year, since our talk last year at DevConf, what topics we worked on, and I think it's mostly three issues we concentrated on. It's polishing the existing package and making sure packages we upload to unstable are always in a releasable state, but also we put focus on cutting our way with excesses dependencies and making our base so as smaller, because that's useful in a lot of cases. And also related to that is getting rid of legacy stuff, mostly related to system five in it. So, since the release of Chessie, we had four stable point releases, and those bugs that were fixed weren't always grave bugs, but smaller bugs and just stuff that annoyed people and documentation updates and stuff like that. So, we use user tags excessively and track bugs that way, and if you have any issues with system D in Chessie, please talk to us, please file bugs, we try to get them fixed in Chessie and keep that updated. For new releases in unstable, we started to work pretty closely with upstream because we wanted upstream has a really rapid release cycle and we wanted to be able to ship new releases pretty quickly and unstable. So, what upstream already has is a pretty elaborate unit test suite, but Martin Pitt worked on auto package, on the auto package test suite. So, we run that on every pull request that's actually hooked up upstream in the GitHub CI. So, basically, every commit that goes into upstream system D is tested on Debian or specifically on Ubuntu, on various architectures. That means we could actually release a package to unstable at any moment, and that's actually what's happened. When there is a new upstream release over the last year, we just shipped it more or less the other day, and it's worked out really well. I mean, we need close interaction with upstream for that to work, and we need to make sure to push up our patches upstream as much as possible, and we worked really hard on that and shrink our Debian-specific patches to a minimum, and so far, we are pretty happy with the workflow that is established out of that. So, trimming down the base OS, what does this mean? Sometimes it's better if you install less stuff, and one of that use cases is, for example, the init package itself. There are use cases where you don't want to have it installed. For example, the init package in a built-in environment is not strictly needed. So, we actually made the init package that was introduced in Chessie non-essential. This means you can app get, app remove init, and system dsv and system d along with it. Actually, that's not really possible because we use this flag important colon yes, which means once you installed a system, the init package, it can't be removed without that d-package warning. That's just a precaution, so users don't accidentally remove it. So, what you typically do is you run a reboot-strap variant build-d or min-base, and what you get is a truth environment without system d installed without the init meta package installed. It's just a really minimal build-the-environment, or truth environment. For that, we also not only had to drop the essential flag, we also made sure that the udef is also optional. So, we dropped the dependency on udef in system d because sometimes you have containers where you don't need udef. Docker containers are a good example for that. So, we still want to ensure, though, when you install the package, the system that udef is installed, so we use priority important for that, that makes sure that a standard installation has udef and init, but as I said, you can remove those packages. There was some fallout from that, for example, packages expecting has been run-leveled to be around, but it was really minor, it worked really well. We also split the package for various cases where we thought that the additional library dependencies weren't justified. We tried to keep splitting up the packages as minimal as possible because we do not like to split it up in 20 packages for every binary and have a coherent base system. But for those cases, system d container is mostly for system d and spawn and system d machine d, that pull in additional library dependencies. Journal remote is a special case where we thought it pulled in curl, curl lip curl, I think. So, we split it out as well. And we actually tried to coordinate the cross distro. At last year, system dconf, I talked to other distro maintainers and wanted to know if they're interested in sharing the same names. So, they actually did split out of the Fedora people, split the package in a similar way like we do in Debian, which is really nice. Something that just landed a few days ago is system d, lip system d shared. So far, the approach that system d used is was to link all the object file statically into all the binaries and use some compiler tricks to remove those symbols that weren't needed. Philippe Sattler worked closely with upstream and turned all this common code into a shared library, private shared library. It's not an official API, but that had a really nice effect. It cut the package size in half. So, the latest upstream, the latest release that landed in Debian a few days ago dropped the package size in half. And he worked upstream first on that. So, we didn't ship that in Debian, but worked upstream and once it was accepted, we pulled that into system d, which is really nice. And upstream was really helpful in that regard. So, what we have in, we also tried to get rid of init scripts in this VR scene. Because under system d, the init scripts that are shipped in the init scripts package aren't really used anymore. So, why should we install it? At the moment, it's pulled in because of its priority, but it is close to possible to remove init scripts. For that to work, we had to move that libinitvar.shell, shell library that is used by a huge amount of init scripts. We moved that to the CSV init.util package. We first considered moving into init system, init system helpers, which is an init system agnostic package, but then we decided it was too CSV init specific and kept it in that package. So, we are on the closing stretch of getting those removed. Those are the user tag bug reports. There are some of them left, but for a stretch, you should be able to just not have init scripts installed. The same is true for this VR scene, which in Jesse and previously shipped update RCD and invoke RCD, those we moved to init system helpers because they are really init system agnostic. There is support for open RC in them. There's support for system D in them. So, those move to init system helpers and it should be possible to remove those as well. At the moment, they're still installed. As said, they still have priority important, so are pulled in by the installer, but that's something we work on and there are still some corner cases we have to fix, but that's on our to-do list. Moving on, I'm not sure if you know what INSERV really is in our old CSV init-based init system. INSERV was a tool that generated all the dependencies between the init scripts and calculated the priorities, made sure that the one in script runs after the other and there are some system facilities defined by LSB which are due to a limitation in INSERV have to be specified via INSERV config files. So, what we did in Debian to support that, make sure that the ordering between the init scripts and the system D service units is correct. We shipped a generator, which amended the generated unit files or service files with drop-in snippets and pull in those targets because system facilities in the CSV init world are mapped to targets in the system D world and it's really a Debian specific patch and we really like to get rid of that and it's really simple to do it in your service file if you already shipped that. You just declare once before a target name and even if you ship a CSV init script in your package and you don't want to migrate to a native service file yet, you can still amend your CSV init script with a drop-in snippet. So, there are not a lot of packages left which need to be fixed. Those are tracked with that user tag if you want to take a look there. The biggest topic is the removal of RCS init scripts. Those are really tricky because the dependencies defined by those init scripts, RCS, maybe I have to quickly talk about what RCS.D is. RCS.D is Debian specific and means those init scripts are started during early boot and they are pretty prone to cause dependency loops under system D because the generator which runs for those init scripts doesn't know the specific dependencies those init scripts have. So, sometimes the dependencies are too strict and for example we have hooks in DHC client if up down which start RCS init scripts and those can lead to deadlocks. So, it's also Debian specific patch we would really like to get rid of and the best solution we have is just ship native service files for your Debian and in that service file you can declare very specifically which dependencies you need, which orderings you need and then this way we can get rid of the patch and we can get rid of all those dependency cycles that happen because of the RCS.D init scripts. Finally, I've taken a look at how well system D is accepted in Debian and what you can see here the yellow line is the number of init scripts we shipped in Weezy, Chassis, Stretch and Sit. The green line is the number of packages we have shipping init scripts so it's pretty close. I mean there's one system five init script per package and what you can see the blue line is the number of system D unit files or service files and the red line is the number of packages with service files. Typically what you have in system D or packages shipping service files that you ship one service file per Debian will not start multiple demons in one service file so that explains why there are that much more service file compared to the number of packages shipping them and what you can also see that the increase was pretty nice over the Chassis release cycle. We over doubled it and we already have 50% coverage of service file in Debian which is really great and we didn't expect that, that people are so eager to work on that so huge thanks to everyone. So I wanted to leave time for questions. So this is where we track our bugs. We use, as I said, we use user tags pretty extensively. If you want to have a look and help us fix those bugs look at either system D specific box or box and packages which are user tagged. If you want to get in touch with us we do have an IRC channel, an OFTC, it's called Debian minus system D. We also have a mailing list. So just come by and talk to us if you want to know more. I will also be around the next couple of days so just grab me in the hallway and if you have any questions specific to system D just talk to me. We are friendly people and if you actually fix a bug, come by. I didn't bring cookies but you get a free hack. So thanks everyone and I left time for a few minutes for questions if there are any. I think we should have five minutes, right? So I'd like to go back on the RCS problem. Yes. So about two years ago when I attempted to port open LC to Debian, I've hit the same problem which is that... Which package did you convert? Open RC. Okay, open RC. So I also found out that the RCS scripts have many dependency loops. So the way we did it was that we have some logic to break these dependency loops in some way or another but I had the strong feeling that the problem was the RCS scripts themselves. Wouldn't it be healthier for Debian if we just attempted to fix these rather than going around the problem? Well, what do we do? Because K3BSD port and her ports still have the issue. Okay, so what we do in system D is actually we don't want to work around those issues anymore but make the package ship native service files. So I'm not sure what you mean and what we should do differently. Do you want to abandon RCS in its scripts altogether? No, I'd like these dependency loops to be fixed in the RCS scripts so that non-Linux ports could still enjoy having a healthy RCS set of in its scripts rather than these that we currently have with dependency loops. Okay, so thing is system D in the early boot sometimes behaves differently. So some of the dependency loops you don't get with Sys5 in it or the Sys5 in it just doesn't care so it doesn't happen. And I'm not sure if we can easily detect those dependency loops with system 5 in it. So I'm not sure if we can, or how we can fix those. We have the code to detect them and they print when you start OpenRC and it says, here's the loop, I'm breaking it. If someone wants to fix the RCS Sys5 in its scripts, I mean all power to them. The solution to system D for system D is really to ship native service files and we can't really fix that with RCS, keeping the RCS D in its scripts. So we will push getting that fixed and it's actually looking pretty good. I mean we are on the closing stretch. Christian Hofstädler who is here worked really furiously the last couple of days so we get that for a stretch for system D at least. And if there's interest for OpenRC working on that, I mean, great, why not? Any other questions? It's network, no it works. Maybe I wasn't paying enough attention but did you mention system D network D? No, I didn't because there are no specific plans for stretch yet. We ship that within the system D package. It's not enabled by default, you can try it. You can use it if you want. It works nicely for different use cases but we don't have any immediate plans to make it, well replace if up down for example for stretch. I think it's too early for that. There's still some missing bits and pieces here and there. For example, there is no hook interface yet and option is not really so eager to add a hook interface so this needs to be worked out first before we can propose that. And are there any other things that are available in system D that should be in the same situation? I mean I do know that Ubuntu is pushing for system D resolve D which is a really nice technology but I'm not sure if we are there yet in Debian. I mean I will just keep looking how it works out in Ubuntu and maybe Ben for Buster we propose the same. Okay, I mean there's also system D boot but it's also optional at the moment. It's for UFI booting, not something we push for Debian 2 to make the default. Yeah, one question. Do you guys plan to update the version for the freeze? We are in 2.30 for some time now. The version is unstable? Yeah. Sure, I mean as soon as there's a new upstream release if you follow the Debian change lock we basically uploaded a new upstream release the other day. That's possible because we build it every day anyway and we test it every day anyway and I actually use a trunk build and it just works, it's really nice. So yeah, before the freeze we will try to get in the new version this. Well, we try to if there is something upstream and which breaks then we probably will defer a new upload but sure, we track new upstream releases pretty closely. More questions. Hi, is there any plan maybe by the security team or somewhere to go through and add some of the security flags to various services? So for example, you can add flags to remove use of home or add second one for, there's a whole bunch and over time it's beneficial to add them to various services. Yeah, that's a good point. Actually one of the benefits that system D provides. I don't know there is a coordinated effort at the moment because it really needs input from the individual maintainers. They know best what their service needs and how you can restrict those. But sure if someone would want to work on that coordinate that maybe put together a wiki page documenting that stuff that would be really awesome. So if that's something you're interested in, get in touch. Okay, yeah, so system D has a lot of flags where you can restrict the capabilities the service has. For example, you can use private temp then your service gets a temp directory which is not the real temp, but a private directory. There are lots of simple switches you can enable in your service file which restricts our service and makes it as likely that open box and in the service affect your system. Okay, questions? Oh yeah, we ran out of time. So we're out of time for this session. So thank you already, thank you. Thanks for coming.


Hello DefCon, my name is Ken Pyle of Cyber and this is what I call my Blue Monday series which is a series of new attack types, techniques, vulnerabilities, and web application exploits that I've been working on for quite some time and really this presentation is going to focus on mapping exploiting vulnerable devices and appliances at scale via self-registration services. So again my name is Ken Pyle, I'm a partner and exploit developer at Cyber where I do a lot of reverse engineering, offensive security stuff, a lot of refinement and development of new attacks, exploits, techniques, and pen testing, that sort of stuff. I'm also a graduate professor of cybersecurity at Chestnut Hill College, Chestnut Hill College is particularly notable as we are the college that produced the ENIAC programmers way back in the day, so a very awesome, interesting little facet of the program. So at Cyber what we do is we are a manager as this provider, data-secured privacy compliance breach response, digital forensics, e-discovery, data recovery. So we're a full spectrum security forensics and IR shop and obviously my primary focus is offensive security tactics, exploit development, reverse engineering, refined methods of exploitation and attack. So I am also a professor of cybersecurity at Chestnut Hill College where I teach a lot of the techniques, new ideas and whatnot that you're going to be learning today so we have a number of different programs, master and undergraduate and we cover the full gamut of education and training at an educational level. So what I'm going to talk about today is really what I like to think of is what I think of as real threat monitoring, real threat monitoring which is attacking common or shared vulnerabilities, supply chain, software, hitting things basically due to bad design, poor decisions and a lot of just accrued and accumulated vulnerabilities and issues that come from a lot of places over a long period of time. So what we're going to talk about today again is appliances, IOT stuff which are designed for ease of use and access. A sample of the devices we're going to talk about today have a couple of very interesting features, mainly a find me feature which whenever you're in a network or a segment where a device such as one of the ones we're going to talk about is, you can go in and use their service or their website to find them. You also have appliance deployments and scaled use. So we're going to abuse these things to identify map networks, identify vulnerable and exploit vulnerable devices and basically turn a lot of stuff into botnets and large scale attacks. The reason for is very simple. These are architected for mass deployments where the fully qualified domain name, the DNS regime, is unknown, unreliable or totally independent. What that means is these devices and appliances are designed where they're aiming for the lowest common denominator and the designer really can't tell you or provide any assurance that the environment they're going into, if it even exists at all, is reliable. So this creates a number of weird sort of lags as I use for this right here. Splashback properties as I like to call them because it's a lot of poor design where people aren't talking or just jamming things in and it ends up causing some very serious fundamental problems that we don't really see until you get out into the field. And some of this is due to IPv4 address exhaustion. We have created all kinds of workarounds over the last couple of decades for IPv4 address exhaustion, convenience and a number of other things that have created these fundamental flaws such as, like I like to say, splashback flaws in everything else. And one of the core issues for that, one of the reasons is, again, we have this constant fight between DNS, IPv4 and humans. And it causes all of these problems. And what I've seen, I've been doing this a very, very long time, application and IoT developers rarely ever see what I say below layer seven on the OSI model. They see the network stack and IPv4, IPv6, whatever the transport mechanism is as the exit ramp for them and they don't really need to consider what goes on. It's a network stack, it's a call, we just drop it there and it goes off. And then engineers, conversely, rarely ever see above layer three or four. They say, hey, it's the IP address, it's up, I can ping it, ICMP works, I can pass traffic, TCP, UDP, what have you. And they don't think about what happens above that. And there's this gap this all sits in that I love to live in because it makes it really easy and not a lot of people bridge those gaps. It's one of the places I live. I understand both sides of this. I've been doing exploit development for a long time. I'm a pretty damn good engineer. And this is what I see and I'm doing this stuff all the time with my team and my company in the field and this is just a recurring problem. Which creates a cloud and appliance vendor problem because they rarely ever consider metadata leakage, advanced tactics or hybrid attacks ever. And I have plenty of proof of this and maybe I'll come back to DEF CON next year and show you all the emails and conversations, what I have had over the years even related to this where you find out that a lot of vendors aren't doing the things they're supposed to do. And even worse than all of this is no one ever cooperates. Everybody white labels. And this is how supply chain attacks happen. This is how MSPs get popped at scale. How big attacks happen, like SolarWinds and Kaseya, because all of these things happen. And again, it comes down to for what we're talking about here, IoT vendors and integrators. And it allows for scaled attacks and supply chain exploitation. Vendors collect metadata to market and enhance the reliability and usability of their devices. They refuse to coordinate or consider any of the implications of kill chains or the aggregation of vulnerabilities. And again, I'll show receipts some day on all of this where I have either spoken with or engaged with some of these vendors and they go, well, it's not my product and it's only really affecting this little part. I don't care. Or they are quite simply not aware of or don't consider because you're either talking to an application developer or a security person who doesn't understand all of this. Even worse, vendors control the ecosystem of patching, installation, integration, and knowledge. They are the gatekeepers of their own vulnerabilities. And this creates a huge conflict of interest because how many companies are going to openly come out and say, we didn't think about this and didn't design it with this, especially when it's an issue or a problem that at now or in the time isn't scored or considered very high on the vulnerability scoring matrix. And they're relied on to know better how many of these devices and appliances and whatnot are designed to take the thinking out of the hands of the end person. They're frequently bought so that somebody doesn't have to think about a lot of these things. And again, at the end of the day, vendors want to make money from everybody, especially in COVID. They want to get vertical. They want to sell at every level. MSPs, individual users, large corporations, what have you. And they try to be everything to everyone. Like here's the Meraki website. Meraki does Wi-Fi, switching, SD, WAN, security, wireless WAN, Meraki Insight portal, smart cameras, mobile device management. These things are just absurd to me that you want to try to do all these things. And even on top of that, I mean, here's something I pulled out we're going to talk about later.  Who would want to put WordPress on anything? It's not designed to support it. I hack WordPress all the time. It's actually remarkably easy. And people are going out of their way to put this on vital infrastructure. It's crazy. And all of this creates what I like to call the Dennis system of IoT and self-registration. The thing to remember, and I'm going to show you individual vulnerabilities or whatnot, but here's the crazy part about everything I'm going to show you. Nearly every one of these devices or appliances with this service model or design concept might suffer from a common set of very easily exploitable conditions. Because, again, DNS regime is unreliable. The FQDN and host headers of any new individual host are never really known by the developer, and they have to make it work with everything. So it creates a couple of very interesting exploitable conditions. One, almost every one of these things suffer from a 302 abuse or redirection flaw where you can just simply feed it the right input craft that only can send somebody wherever you want. This also creates a number of cross-site request forgery abuses. Because, again, if you're depending on certain information, submit it from the browser as part of authentication, and you are openly disabling or dismantling these protections, which they do because of these things, you can come up with a bunch of CSRF abuses that are abusable via DNS. This, in turn, creates a lot of cross-site scripting abuses because you can combine these two up here, most of the time the 302 abuse and host header injection, to inject arbitrary HTML content or trigger cross-site scripting. Because they accept almost all kinds of client input on their side and integrate it unsafely and build content relatively based off of that, which creates, again, what's in right here, both server and client-side exploitation of privileged users working en masse on appliances and devices designed to take the thinking, the bad part, the convenience part, out of their job. And again, it creates the Dennis system. The Dennis system of IoT, and I like to consider IoT, I say, it's instead of thinking marketing because they don't want you to think at all. And in fact, they're marketing and designing around that because here's your Dennis system. You can't trust anyone's infrastructure so they implement dynamic DNS services. You can't trust anyone to be compliant so they implement encryption and external PKI services like certificate stuff. You can't trust anyone to install software, particularly not your client. So they either don't allow any software installs or they do and they don't allow any external oversight of that or they publish an ecosystem. You can't trust user knowledge because a lot of these users or integrators are buying this to not have to worry about having that knowledge. Security is difficult, it's hard to do. So we're gonna want to nurture that dependence, nurture it via MSPs and white boxing.  And lastly, you can't trust your clients employees to know what they're doing. And again, they're marketing based off the fact that these people, integrators, engineers, small IT companies, medium IT companies, corporate security, what have you can't possibly know all of this. So it integrates silently. You can't trust them to do anything. So we're just gonna slipstream right in very easily and do these things. And on the right hand side, it's a very good example of this. This is the initialization page for a data device. I purchased off eBay. And again, what's interesting is you have to check the box to allow third party installation of drivers and software but you can't actually trigger any of this in the device without allowing it in some cases. And again, if you look at the functions here, I use device for migration, things like that where they're basically making this a cattle shoot of exploitability and vulnerabilities. Even beyond that, I'm gonna only spend a couple seconds on this but one of the crazy things in why supply chain attacks are so big and they're going to get bigger and bigger and bigger and I'm gonna give you right here a technique for doing this. Remember this, all of these devices and appliances obey the same laws, everything else does. Anything that exists on this device is a file or content that was put on through a firmware update or OS install. They're almost always running Linux. They're built for efficiency to be cheap and for storage considerations. So if something resides in a firmware update or on storage, it is either essential to it because they wanna reduce cost on storage or they forgot it existed and it's in there by accident and that tends to give you some pretty good information. And again, think of it this way. Firmware updates, no matter where you're getting them from are usually just flash file system updates or drive information rewrite of whatever's over there. And the cool part is you can carve these things like any other file system. In fact, I do it all the time. It's one of the best techniques I have for finding vulnerabilities. It's how I found a lot of these in here. Obviously, I like to use BinWalk because BinWalk rules not only because of the IoT though which sponsored but they're just fantastic at doing this stuff especially when you do the extraction and what they call the Matrioshka which is going through and water falling through recursively extracting all these things out of these updates. So if you look right here, this is an update I pulled which is literally a file system that gets pulled out of this image. And from here, you can pull out static credentials not expiring credentials. Static passwords or program credentials and passwords which are MD5 and crackable, stuff like that. And again, even kind of the byproduct of this and I guess I only spend a couple seconds on this is that you can start carving these firmware for supply chain attacks. What you're seeing right here is internal development system and network metadata that I carved out of a firmware update that tells me the internal development environment and networks of the company that produce this. This is a very specific product that does this sort of embedded system build in our firmware update. This is the GitHub environment and I had to remove all this and redact it so that nobody actually sees what this is but this is the GitHub environment and details of one of these developers and one of these manufacturers. It's crazy. And again, where we really want to get into is carving firmware for scaled attacks. These two things right here are a private certificate, private key that I carved out of a firmware update and this is a static password used for encryption that I pulled out of a different device and you can pull all of this out very easily without even owning the device and testing it. So, what I'm gonna start with is here some dynamic DNS domains. There are a number and I picked out a couple of these for a reason and I'll get to it later but these are dynamic DNS domains that are maintained by each one of these individual providers as an independent self-registration regime that allows you to conveniently access things. Now, you can start farming these zones for a significant amount of information because of a couple of things. The providers are rarely restricting record registrations. There's very little policing of these zones that is actually done. Many of these things enroll by default and the name is algorithmically generated which means it's predictable, which means it's farmable, abusable, I can find it or even worse, they are user defined which opens up a huge area of attack for me. The zones are third party hosted. You'll see that this is data local.net. This is one of the interesting ones and not only that, it registers inside this domain or a couple of the other ones but it also registers inside the default DNS zone of wherever they are, so it ends up registering two DNS zones. What's really interesting is via passive DNS collection, all of this information is gonna live forever which means it's never going away. It's a map for me and a sort of codex of things to look at and there's significant metadata leakage. I can't stress this enough. Think about it this way, by passive farming of this DNS zone, I can tell you about how many or estimate how many of these devices are out there active local at any given time. Like there's probably about 140,000 DATOs that have been sold and activated over the year. Let's talk about these exploitable storage ecosystems. It's a good place to start. I'm gonna use DATO and Synology as my primary example. Now the first one you're gonna see here on the right hand side is this find me function and it's a very simple enumeration.  You type in find.synology.com or find.datocom.com or what have you and it's simple enumeration. You go in there, it runs some things, all of them do it a little bit differently and it will tell you exactly where any or all of your storage and backup devices are in this network. In fact, it's part of the setup routine and in fact marketed towards. They want you to do this. It registers with a dynamic DNS service and that's part of the setup. And these again, are either user-definable names and controlled input or algorithmically generated. This creates a number of very interesting scenarios. You have third-party applications and administrators who are typically using this and again, this is what they're designing for. And if you consider all this, this is the keys to the kingdom. If I am a ransomware threat, if I'm an advanced persistent threat actor, if I am somebody who knows what they are doing, this is the first thing I wanna look for because this is the keys to the kingdom. People are putting backups on these things. They're keeping their NAS, this is a NAS, it's keeping all of their data on it. This is what I wanna know about. This is my end goal and this is where I wanna be when I start acting in a real bad way. And even beyond all of this, the thing that's the worst part about these devices is that the data custodian typically, and if there's typically more than one, gets to execute a lot of user discretion. They get to say or do things or assign rights, discretionary access control, all that kind of stuff, which opens up offset failures. And again, I could go into just this part forever, but I'm not going to. Just think about that for a second. What gets even crazier is that most of these zones, it's nearly universal, do not scrub RFC 1918 private addresses. This is almost universal. So by farming these dynamic DNS zones, I can start mining huge amounts of private network information. That's really what I'm concentrating on today. The data is exultrated and updated to the provider and is available unauthenticated via direct DNS request. Have you ever been asked for a password when you do a DNS request? Because I certainly haven't. And this is a great example of it. This is a DATO device. This is an algorithmically generated name from DATO that I found out on the internet through researching. And look at what this record tells me. This company, whoever they are, are running a DATO. The time to live on the record is 120 seconds. The A record points to <IP_ADDRESS>, which is a private address. This is the private IP address of this DATO device wherever it may be. The problem is here. This is being hosted through their zone. And I queried it through <IP_ADDRESS>, which is everybody's favorite DNS server, right? Or 8.8.8. You can go out and test this exact record for yourself. I queried this, again, on July 15th, 2021, just to date to this, just so you could see that this really exists this way and has for a very long time. And you ask yourself, why does any of this matter? Because whether this company knows it or not, the internal IP address scheme of this target network is available externally. I have effectively mapped something behind their firewall, which may or may not even have a direct rule out to the internet to allow me to see it, which creates a couple of things. One, if it does, I have just mapped a NAT and PAT address translation through the firewall. If this thing has a direct NAT or PAT through, I have figured out, if I can reach on 8044325, you name it, that's a firewall rule I've just figured out. Plus, I know the internal IP address. That's a key piece of information. If there isn't, I just determined the egress IP address for this network. Because if they're registering and they're NAT or PAT-ed and it's something external, I can tell you where all the traffic leaves. And again, this allows me to determine firewall rules. This also allows me in mass to find vulnerable software appliances and organizations really easily. And what do I mean? If you look over here, this is a Meraki zone, dynamicm.com. And if you look over here, I am following a templated install from somebody over and over and over again. A VPN template, VPN template, VPN template, all the way down. If these are being deployed, and this is indeed a templated install, what I've managed to do is find and track someone cookie-cuttering these things out. Not only that, again, I have many of these. You look for template in this zone and you're going to see people repeatedly doing this. You find a common supply chain. If they're putting a static credential, which is a common thing MSBs do, to be able to administer NMAS or by default, if that doesn't get changed, I just found them easily. So why does this matter from an advanced standpoint? Because now I can create hyper-accurate maps of internal networks down to layer one and beyond. For example, this is one I found. I have the exact device. I have the layer two address, the model, the fact that it runs wireless, and it's external IP address. I can overlay these things and tell you exactly what layer one looks like, at least partially, from just this one entry. On top of that, you can hijack code execution and redirect clients. Why? Because of three, two redirects, dynamic DNS registrations, a whole amount of other things. You can also create extremely effective social engineering campaigns outside of any sort of technical exploit. I know if they're using Meraki or Datto or whatever else, if this is a external IT company that's servicing, I can call them directly. I can call the people directly. Hey, I'm calling from Meraki or Datto. We had a problem. Or I'm calling from your IT company, which registered this. All of this is out there. Again, this also has passive DNS collection, which allows me to stealth, ninja style, historically view detailed internal vulnerability information forever. I can figure out what internal controls are, what to target when it's there, even on shielded private networks. Think about this as a double NAT or PAT problem. I'll show you this, but think about it this way. If your device doesn't have a direct route, or direct PAT NAT, whatever, to the internet, but it's buried three levels deep in your core, I can read your core IP addresses from these registrations that are getting sent out whether you know it or not, which is exactly what I'm after. I wanna know where I'm gonna end up, if you will, when I land. When I exploit this device, I know where I'm gonna be so I can pivot or what to avoid if they have Meraki security products in there. I know where they are and I know not to hit them. In fact, I don't even really need to do port scanning to find this stuff, right? So let's start with Datto. Datto devices, they're backups, they do disaster recovery, there's appliances, infrastructure. And they focus on MSP and partnership relationships where they wanna say fire and forget. The self-registers, you don't have to worry about this. It's a backup device, it's a disaster recovery device. It takes all the thinking out. You have Datto dynamic DNS zones. It automatically signs up for a certificate. It white labels the software. In fact, their agent is, from my research, a rebranded storage craft agent. It also works on this hybrid and hosted model where you can put it in any, roll it any way you want.  These things also like massive amounts of metadata. To an absurd degree, I could do hours on just this device and ecosystem alone and why it's so poorly designed. And worst of all, they're vulnerable. There is continued persistent issues and exploitable flaws and design considerations that they don't really seem to wanna fix or address. In fact, I've had public work and CVEs registered critically against these things since 2015. Now, let's look at their DNS records for a couple of things and the way they design this. Their DNS records, like I've read most of them others, have very short time to lives, which ends up functioning as a beginning or a canary. It tells me is the device on, is it active? If I'm trying to DOS or take this device out, disable their backups, watching this externally tells me if I'm successful or not. Again, this does active external registrations, which give me the I egress IP address and firewall rules, particularly important if I'm reverse tunneling.  And again, these things run Linux, they're very simple to pop. And again, they have backup agents and software and these things are pivot points. There are other things, other software that gets installed as part of this ecosystem that's exploitable. It's pivot points, plus we know it's talking to other devices. It allows me to identify things or even better, I can flat out disable the ransomware controls to make sure you can't back up and you have to pay my ransom. Now, let's go hunting for these things using that some of the techniques I've given you. So we have TKI registrations, we can directly form the DNS zones, we can do passive collection via security trails and show it on. And if we're inside a large network, we can just use the fighter service. This is an SSL certificate that these devices automatically register for. And I've actually taken this entry out because I don't want you knowing what it is. It's a private piece of information. But this isn't a certificate signed up for by the data device that you can go find through any method of doing this. And again, here's the private IP address of here, of this device. I can map these two things together. I know this thing has SSL, I know that's the host name and I know that's the internal IP address. It's a very useful thing to have. But let's start thinking like a real bad guy thinks. Let's do some real threat modeling. And this is one of the most interesting entries I found. This is ostensibly a data inside a public Department of Defense network. And think about this, the only reason you're dropping a data in your network is to do DR or backup or what have you. This thing is beaconing home regularly. It provides status updates at regular intervals. That's what this is. It is active because once this TTL expires and it's not updating, it's taking it right back out. This thing houses, again, potentially use of backups and data. If this is a DoD network, really was a DoD network information center computer, this is probably holding some data I wanna get my hands on or hit. There are firewall rules allowing for it to tunnel back because it is beaconing actively. It's registering itself. There are other things in here that are happening that tells me this thing can reach tunnel back or there are firewall rules allowing it to reach something on the internet. Again, I can reach a lot of things about it and worse, it's very attackable. Why? Couple ways. I'm gonna use this different entry to show you this and again, I'm not gonna give you exact exploit information through tunneling through right through. I'm gonna show you how this all goes together. I'm gonna give you the pieces put to puzzle together. So over here is Shodan and over here is a dig entry for this more farmed out of the data zone. And you can do this either way. You can either look for data devices in the data zone or you can just go out to Shodan. What's great is this is reflected. It works either way. I have the external IP address, the external DNS name, the certificate and by the way, it can absolutely get through the firewall to this thing. And this is the IP address, 727522930. By determining this entry and comparing these two things together, what I have effectively done is figured out that there is a NAT PAT rule for that IP address for this and oh, by the way, this is their BDR appliance. From just this information, I have put together a huge amount of attack data and mapped things out just from that. Again, .30 has a firewall rule for 443, 80, who else knows going into <IP_ADDRESS>. 25.10 is actively beaconing outwards and sending stuff back through the firewall, which means I have determined both ingress firewall rules and egress firewall rules. That's all I need. Plus I know this company, whoever they are is running a Datto, bunch of other things that you can figure out from this. Now, here is another one to look at. And this is again, using security trails and look at what we figured out here. One, I can figure out DHCP scope features by just watching how long the duration is for or how long leases are held for. I can tell you that this thing was on either Pat, whatever, internally, went to an external IP address and then went right back in again. What's even crazier about that is I can actually map out IT developer, whatever, habits out of this. And what do I mean? This 10, the 10.network and the 192.network for here. And again, this goes back three years. Whoever is administering this likes to apparently put backup devices at 161 or uses that for configuration. I found this entry by looking for 1.10 because I know it's an IP address that IT people or engineers like to put important things on. So I looked for 1.10 and came up with one of these records. Oh my gosh, they also like to use .9 and 1.61.  I have now been able, I can now predict where things are on their internal network or where I want to go hunting. Again, if I want to avoid something or disable something or go after something, I already know in this network we're a pretty good idea of where things are gonna be. And again, I'm tracking their behavior and uses. Western Digital has the same problem. In this, you can use to map DHCP scopes, remotewd.com and westerndigitaltogo.com. This one is hopping someone's internal DHCP scope 24, so this is going through their entire DHCP scope and telling me what's the extent of the DHCP scope, what the properties of it are, how often this thing is getting rebooted, but not only that, again, egress firewall rules of this person's network and that they're running Western Digital device. All very useful to me. So let's look at Synology. Synology has a very similar design and but when you do it, they suggest an opt-out. And again, they have a third party application support in the ecosystem and I always say users install the dumbest things. They really do. Their DNS zone allows for a lot of user discretion and parameter injection. And I'll show you that second for, and this is where you can start taking it. Cannot connect to Synology.me when at home. I know why that happens. You probably know why that happens, but this person doesn't know why that happens, which allows you to abuse this very, very, very, very, very easily. And I said a lot of varies for a reason. This is the entry. This is the workaround. And you have people in these user environments or these communities giving out the worst possible things that other people are going out and telling them what to do. So they're disabling security controls for me on a device I wanna take a look at. And there's zones Synology.me and DiskStation.me, I believe a couple other ones. Give me all this. I can search for interesting keywords. I can search for user metadata or I can find vulnerable software and instances of user discretion. What do I mean? Go search this zone for what I did, Firewall. And see how many people have named their Synology or given me some sort of interesting information about their network, either directly or indirectly through these DNS zones. What kind of Firewall is probably there? Which one's the Firewall server? All of these, whose Firewall is that? All of these things are out there and you can find them. And again, you can find from the hosting provider. Even worse, think about this from a blackmail or a ransoming standpoint.  Why? Because people have Plex devices. They like to use Synologies for them. And you can get this information and you can find every pornplex on earth that's running on Synology. Think of that if I can get your IP address or you're laking other information out of it or it's been registered in a different name. You're now giving me who you are, where this thing is and if you have some compromising or interesting information on here you've given somebody how to do this. For example, here's another one. Look for backups. What do people name backups? RSA backups. SI Hodges Backup Server. We'll talk that one in a second. All of these things, Tom's backup server. WP might be WordPress. Lazy backup server. All these other things you go, oh my God. You can map these things out and they're telling you everything you need to know. And think about this from a user metadata standpoint. One, I can tell you there's about 675,000 registrations inside that zone. And look at some of these pieces of information. Barrett Nass, Harold Bogner, which is, I believe, a company territorial, that's probably a Italian speaker. You have all these things out here and you go, oh my Lord, they're giving me everything. You can use this in fact to de-opfiscate anonymity networks like Tor and other things. Why? If they're algorithmically generated they have an agent or some sort of software on there and they're using a full stack proxy that's going out and beaconing this or what have you. It is highly doubtful that anybody else on earth is going to be looking for these specific FQDNs, especially if you catch them coming across Tor or what have you, or an anonymity network. Even worse, you can track via this passive collection where this IP address is, all the metadata about it, and that this thing is hopping a dynamic cable provider network for their backups. Think about this if I wanted to DOS these people. They're giving me a path to find their IP address forever. It's like a giant beacon for me forever to track them across.  You can also map controls for this. This is a way to map IDRAX as well. Getting that later, I exploited them in a totally different way. But again, these are things we're registering inside zones, outside, and again, these provide some more information. And you can map people's entire networks out of these sorts of registrations. Again, this is a switch, a wired Meraki switch that also happens to have the same IP address, egress IP address, as this support portal. Here's Wards, Livermore's Laboratories IDRAC device. It tells you exactly what are on these things, what they're running. Look at these things. I can map all of these to these zones. You can even find vulnerable instances of software and applications from this, because again, people install the dumbest things. Somebody installed WordPress on a Synology NAS. Not only that, I get to map out everything from it. It's externally available. It's in another country. I have everything I need to attack this thing. And you can find them through their dynamic DNS registrations. Meraki does a similar thing. They have dynamic DNS registrations because they're cloud managed infrastructure. These things are designed to allow you to always be online, always updating, and manage your infrastructure from anywhere on Earth. And look at some of these zones and some of these names. People are telling me, MX84.1, Boston, Glenwood Springs, you're telling me exactly what I wanna know, exactly where it is, and exactly what the device is. And that's a problem. So I wanted to use this for a little bit different, and I decided to start exploiting and mapping developer habits and networks. And you can map the Meraki zone for registrations. You can map the entire infrastructure regime of internal networks. However, one little entry, and you kind of already saw it, was very interesting to me when I was going through here, this one, BXBRAC66. And Aaron's search of this tells you that this results back to Cisco. Yes, that's Cisco systems. You can further illuminate, rumorate, and passively farm this zone, and you'll find a lot of registrations that they have in Meraki here. Think like a bad guy thinks for a second. These devices are externally available. They allow for a micro-targeting of devs in future releases. I can track teams, devices, and habits. And think about this, this is a regression test for MX67. If you're changing something soon, or I wanna test it before it's, or something changes on the new one from the old one, it means it's a problem that I can exploit, and we have no idea when it's gonna be fixed. In fact, you can map these registrations and overlay update schedules, and start seeing where they're working on things actively and how long it usually takes. This is how you crack the supply chain. Look, BXB regression has moved several times. Remember that devs and engineers tend to forget the layers that do not apply to them. They don't really care. Now, I don't condone doxing. I'm not gonna dox anybody here. That's not why I'm doing this. The reason I'm showing this is to make a point. None of this will ever be erased, hence exposing some of this is not inherently evil. I'm telling you how to do this, somebody else is gonna find it. We need to fix this problem now. We need to start cleaning up on IO5 and fix this. And the first thing to do that is start thinking like a real bad guy thinks, and how you can abuse this. This information is obfuscated here, but it is persistent. By going back through this zone of Rack 66, and going through other IP registrations and dynamic DNS registrations, what blacked out here happens to also be the names of people and what they're working on.  Taking just one of those names out and overlaying the Aaron record and a LinkedIn search of one of these names, you turn up that this person is in Midway, Massachusetts, and they happen to be the engineering technical leader of Cisco, Maraki here. I can target them specifically. I can go after it. I can look at all that stuff. And it's very scary. Now, moving on, let's talk about GeoVision security systems. This is where it gets really scary. They have dynamic DNS zones, gvdip and dipmap.com. The developer is known, both by me and many a couple other people, to delay or improperly patch their software. They don't do it right. They never do, for whatever reason I do not know. Which opens up a mirai type botnet opportunity. Not only that, the target groups of this have very poor Opssec. The target market has poor Opssec and the distributors have poor Opssec. Think about what this is. It's a IP network camera, DVR, access control, license plate recognition, facial analytics, you name it. Now, look at this. And this is what really appalled me. They published this changeover notice for their dynamic DNS service that says all the things that are on this old zone that's getting retired on October 31st, 2020 are old things that don't get updated and they're vulnerable and you shouldn't use it. But we've got a new one, you can find all these things on, you should update your stuff. Now what's crazy is, this is dipmap.com, the zone that's supposed to be disabled. The problem with all of this is, it's not disabled. GeoVision DVRs suffer from several vulnerabilities I have disclosed to the vendor personally. If you'll cure June 16th, cross-site scripting HTML injection, local file include XML injection code execution vectors. I told them about this, very specifically. In fact, here's some of the exploit right here. They don't seem to have an interest in fixing this. In fact, thank you for your reply. We provided, they said, hey, we have provided, we created this update for you. And we did, and I updated it. We'll see what happens in a second. Please help us scan and confirm, which I did. As for your question about whether or not we plan on registering or telling anybody about this, we'll do this after we, my, myself, confirmed it was patched, which is not the way it should happen, right? It's not, but keep this in mind for a second. I tested it here and said, hey, my God, this thing isn't working. You gotta fix this. And again, here's what I'm asking. Their reply back to this was, and again, I said, this isn't fixed. Here is the way you exploit it. This is an LFI on a Windows-based host for this thing, you can go out there and you can pop all these things. Their reply to that was, we discussed with engineering and then we scanned with Nessus and it doesn't seem that Nessus seems or any problem with this, with version 5.33, which is what the version I tested here. They have attached the Nessus report and sent it to me and said, well, thanks a lot for everything, but here's the problem. If they don't report it, don't register it and whatever else. It's never gonna show up in Nessus, right? So what have they done? They don't seem terribly interested in it. They seem to be backing away from this problem and trying to talk over my head and talk over my team's head. That doesn't work that way. The way Nessus definitions get put in there is obviously through CVs, registrations, whatever, which they obviously have not done. Think about that for a second. Now, again, dipmap.com, if you look at that old entry, is supposed to be disabled as of October 31st, 2021. It's not. In fact, this is a company called 24 Hour Toe, Cobb Parkway, wherever that is, and this is the login page for their device. Even worse, the first time this DNS name was seen is well past their expiration date, but not only that, it's still active as of 719. This thing is exploitable. It is simply exploitable and this company does not have a very good history of patching and you've already just seen that. Absurd. Now, think about some advanced applications of all these things. What happens if you overlay multiple DNS zones in public open source intelligence sources? You can track vendor preferences, poor Opssec, and then the poor internal deployments. You can pivot through multiple networks via botnes and DNS zones abuses. Again, if you farm their entire zone and it's all public IP addresses and you can get to them, because again, this is a DVR or a wireless access point or whatever, and there's a common vulnerability through all of them, I can just immediately take everything over. You can use some of these internal IP addresses for some really crazy stuff I'm not gonna show here, but obfuscation, misdirectioning, signaling, exfiltration of data in ways I haven't even shown yet, but think about all the things you can use that for. For example, these things are operating essentially as fast flux DNS networks with very little control, but not only that, all of these zones are designed for a very specific purpose to allow you to get into them, plus these vendors go out and make sure that these things can get through firewalls, they're white listed, all these other things, and you can turn these into very big scaled attacks. And again, remember, 302 redirection host header injection to client-side code execution. I have turned every 302 host header redirection and reflection to potentially a client-side code execution vector. That's another paper at another time you guys can go look at, it's already publicly posted, and it works on multiple things, which turns everything device, especially in this scenario, into my own personal exploit factory, either reverse engineering or executing developer attacks. For example, you may be asking yourself why I picked out all of these specific devices in dynamic DNS zones, and I'll tell you very straight out for one specific reason. I have zero-day critical exploits against every one of these that allows me to take over their networks or their devices, I will. Again, going back to, I can't connect to Synology for my own network, and you go out on this Reddit and you find out these people have made all these changes to it to make it very exploitable, such as hosting a WordPress website on their Synology. Another thing on the Synology is, you can reliably unauthenticated enumerate all of the installed plugins via brute force attack, just web server content enumeration anonymously. You can go out and figure out exactly which plugins are out there and vulnerable if you know which ones to look for. And again, I go back to 302 redirects and what have you. The API of these devices allows you to determine exactly what the HTTP status codes for anything are, and you can increment and watch these, for example, if you were successful in bouncing a 302 redirect off of them. Think about that. Datto's have even more issues. And again, this is a symptom of a larger disease, if you will. But the local agent, again, is a rebranded storage craft agent that uses in all testing we had, an old cherry pie instance that's vulnerable. It gives you a whole lot of metadata, but not only that, you can abuse this for LFI enumeration and a whole lot of other things. Plus, this thing has some local privilege escalation problems as well. And this is a SSL server, web server running cherry pie 322 on a port that not many people know about. You can use this and a couple of other things, again, this applies across of other things, is start crawling things like alien vaults, OTX, open thread exchange, or other submission things where you can go through and look at entire conversations between devices on private networks, if you look for the things that are vulnerable and really start pivoting and mapping out networks or picking up credentials. I picked up credentials doing this on against another device. And the last thing I wanna hit on is turning these things into giant DDOS cannons or abusing them. And look at this right here. This time to live is 120 seconds. What is the implication of owning these zones or being able to abuse these zones with these devices? Again, these are fast flux DNS networks. You can abuse them. They don't do a whole lot of RFC scrubbing. You can use them for malware attacks, command and control what have you. You can abuse the TTLs and lack of controls to conduct, target, and abuse their DNS zones. And I'm gonna show you here in a second. Remember, these are user-defined DNS networks that operate under a trusted provider that somebody is probably whitelisted or won't think is a big problem. The question you have to ask yourself is, what is stopping you or me from creating a record that is false, nearly indetectable, abusable on their network, and that they can't change? And the reason they can't change it is the entire ecosystem works off of these dynamic DNS names.  An unfixable problem. This is an injected, valid, spoofed, if you will, because I didn't really spoof it. It really exists this way. DNS record on the DATO network. I just bought a DATO off of eBay that had never been registered. I set this up on my network and all I did was give it the IP address of <IP_ADDRESS> on my internal network, which happens to be a Cloudflare IP address. Why Cloudflare? I don't wanna point this to a live IP address. I needed to create a flag and maybe Cloudflare can take. Looking externally, and again, I created this on my network and this thing registered even before all of that outside. Now, this thing has checked in, and I can force this check in, but look at what I've done. I've spoofed this record. This is not really 1.1.1. It's <IP_ADDRESS> in the sense that it's the IP address of that. But now I've created a third-party DNS record on a fast-flux network that isn't scrubbed that I can use to aim at things now or exfiltrate data or perform any sort of crazy attacks using this. If I wanted to DDOS this thing and I wanted to point my botnet, maybe I took over a different network and I pointed at this address that I've spoofed and it's very hard to fight me off. There's all sorts of crazy abuses you can use for this. Think about this from a social engineering standpoint. If I can go out and change somebody's data device DNS record and they're using this to get to it, I can put them anywhere I want. And it's true. It's how it works. This is that easy. This is how abusable these are. So we've got all these different types of exploits and ways to attack these things because we have a way to map, abuse, inject, find, pivot through, large-scale provider appliance networks. Now, the question is this, how do we keep from getting redenised, if you will? How do we keep from having all this happen again? Because what we're effectively doing for me when I look at this is exactly that. I am playing exploit poker and everybody else at the table is showing me their hand. And all I gotta do is figure out if I get a better hand than they do. How do you stop it? The first thing I will say is stop using insecure self-registration services and practices and appliances. They're terrible. I, it's a bad design idea. I'm sure they're gonna come back and say there's reasons they do it or whatever. It's a bad idea and we've kind of absolutely closed the book on that. We need to bridge the gap between developers and engineers on IoT. Again, devs stop at layer seven, engineers stop at layer three and we don't really understand what the other side does and how it affects us. And this turns into, again, third-party independent oversight of vendors' appliances and ecosystems. I don't know exactly how to do it right, but what I'm gonna say is this. I don't trust them. And you shouldn't. And I'm not saying it personally. I'm saying I don't trust anybody. I'm a security person. But it's clear at this point that the people who are supposed to know better aren't thinking about these things or their motivations may not be to make the best decisions in the interest of their clients. I don't know, I'm just speculating, but it makes you think about it and makes you say, we need people who have no dog in this fight to look at these design decisions and arbitrate it in a better way. Better than what we have now. We absolutely need to migrate away from faulty legacy systems and fixes like IPv4, DNS, and a whole lot of other stuff because we just keep building on faulty foundations and bad information. We need to commit to moving off of HTTP technologies as a transport and API access model. HTTP is a library static hyperlinking transport that we have used for all kinds of crazy stuff that shouldn't be used for. It's why we have all these problems. And lastly, we need to manage IoT the right way. And we need to stop depending on computers, machines and definitions that tell us what's going on. We need better human analysis, period. We need to train people, developers, whatever else, whoever it is to understand the context of information, the exploitability of things, to understand why things work the way they do. We gotta stop taking and eliminating complexity and work out of this job. It's a very aggressive field that's growing quickly and we're trying to give people guardrails and it's causing these huge problems and we gotta stop. So I wanna thank you all for watching this and I'll give you a couple of hyperlinks to check out. Our Insights blog for cyber where I publish a lot of this stuff and a lot of my new research is right there. Feel free to check it out. In fact, after the conference when this gets posted, I'm gonna post some more additional analysis DNS zones and what have you. How to turn 302 redirects into code execution using Java is something I've worked on for a while and there's a link to it and it shows you how to exploit other types of IoT devices. And lastly, my college THC has a cybersecurity conference every year where we bring experts in to contribute, talk, what have you. If you're interested in joining us or contributing to the conference, there is a link for it, you can absolutely do so. So again, I wanna thank everybody for your time. Thank you for watching and I hope this was very informative to you. Thank you very much.


And welcome to today's site supervisor panel presentation. We're pleased that you're able to be with us. We have a wonderful group of site supervisors willing to explain the opportunities they have for you as interns in the spring of 2014. On the screen you'll see my picture, Dr. Pratt-Frank, studying the coordinator for the SLIS internship program. And on the right you'll see Melissa LaFranchise, our student assistant who is responsible for putting this site supervisor panel program together. What we're doing this evening is recording this for use later on as well as creating webcasts of the presentation. So everyone is able to eventually come back and view them again or send somebody to the site to view them. Our first presenter is Wendy Townsend, and she is going to speak to you about the opportunities she has available in the spring. And right now I'm going to turn the mic over to Wendy. Thank you. My name is Wendy, and I'm one of the librarians here at the Placentia Library District. We're located in Southern California in North Orange County, not too far from Cal State Fullerton. So this would be an on-site internship. You would need to be available to come to our location. The main thing about our library is we like to have fun in everything that we do. So whether it's the children's department, adult services, or technical services, we really like to have people around us who enjoy working in the library and enjoy working with patrons and having a good time. So as you can see, our Christmas picture from last year is not really a serious one. It's more of a funny one, and we'll have a little word cloud that you can see on our website as well, based on some of the things that people have said about the library and the things that we offer. We actually have three internships at our library. We're about a medium-sized library. We serve a population of about 50,000 people. That's the community size. It's mostly families in middle class, middle class people that come to our library. There is a large Hispanic population as well as a large Asian population, so we do do a lot of bilingual services. The first internship I'll talk about briefly is our adult services and reference services internship. When you're working with the adult services department, you will be working at the reference desk with library staff and then probably by yourself as well for a little bit of the time. And depending on your availability and when you would like to come in and do your internship and what you would like to work with, your internship might vary. We have a lot of programming here. We have a literacy program. We do a lot of adult programming like DIY, workshops and art workshops to holiday series. So we do have a robust adult services programming department. Internships with the adult services will also work on pathfinders and the collection development, so you would help the library and select materials as well as weed materials from the collection. And if you're interested, you'd also be able to work in the adult services department. So again, we like to leave it a little bit loose so that you can enjoy and learn whatever it is you would like to do as well as be beneficial to the library as well. The second internship is our children's and YA services internship. It's very similar to the adult services. It's just that it's in the children's department. They do a lot of programming as well. Part of the duties would be to sit at the reference desk and work the children's reference desk, like create pathfinders and of course work on collection development. Again, depending on when your availability is, will depend on what type of programs you'll be working with. Our children's programming is in the mornings during the week, but there is teen programming at night and on the weekends. So there's something for everybody. And the last internship is our local history room. And I'm actually the local history room librarian, so you would be working directly with me. We'd be working together on adult services reference, so out in the main library, as well as local history reference questions. We have a lot of projects to do here in the history room. We have a big project where we're going to have to go through all of our photos and determine copyright as well as age. There's digitizing of materials, processing of new materials, and of course we do a lot of outreach as well. So going out to the community and promoting local history. So it's a really fun internship, creating displays. So if you're creative and are looking for something that's a little bit archival but doesn't necessarily need archival training, this could be for you. Okay, so student qualifications. We really want you to be interested in whatever you want to do here at the library. So if you're interested in adult services, that would be the place for you. If you like children and really want to work with children and young adults, young adults would be good for you. Or if you're interested in local history or just history in general, that the history room might be a good place for you. We are looking for friendly, enthusiastic, and self-motivated people. So we'd like to give the SJSU interns a long leash. So we may give you a project and guide you through it, but it would really kind of be up to you to create it and to work yourself through it. You'd have to have good oral and written communication skills. There is a lot of writing, particularly in the adult services and the local history room. You'd need to know the Dewey Decimal System. Hopefully everybody knows that by now and be able to help us create documents using all of the Microsoft application. So that includes PowerPoint, Publisher, Excel, and Word. And of course, how to navigate the internet and how to use databases that the library has. See here, I think that that is pretty much it. I do want to mention that I was an intern at the Plus Venture Library just a few years ago and they actually hired me about a year later to be the local history librarian. We have several, we have had several interns here. Some of them have gone on to other jobs in other public libraries. Some of them are still here with us volunteering. So if you're looking to stick around for a long time and volunteer your time later, this could be a good place for you. Thank you very much, Wendy. That was very interesting and some wonderful opportunities for our students. We'll have an opportunity for question and answer in a little bit. But right now, I'll move on to our second presenter of the evening, Claire, who will speak to us about opportunities at the University of San Francisco. And Claire, would you like to take the mic? Thank you, Pat. Can everyone hear me okay? Get some smiley faces or, okay, looks like, okay, great, looks like a lot of yeses. Excellent. Okay, let's go ahead and get started. My name is Claire Sharifi. I'm a reference librarian at the University of San Francisco, which you're seeing a picture of there in the slide. And that's actually the library right at the center of campus. So a little bit about me. Like I said, I'm a reference librarian at the University of San Francisco. I've been employed at USF since 2011. And I'm a graduate. I'm a December of 2009 grad of San Jose State. So, and so a little bit about the University of San Francisco, not to be confused with UCSF or San Francisco State. The University of San Francisco is the Jesuit institution. We have about 10,000 students, about 6,000 undergrad and 4,000 graduate or doctoral students. We have five colleges, the School of Arts and Sciences, the School of Nursing and Health Professions, School of Management, School of Education, and then we have a law school. The library serves four of the five colleges. The law has, the law school has its own library. So, you get a nice variety of questions and students and faculty working at the library. So the Gleason Library and Geshie Learning Center, it's, like I said, right in the center of campus. Let's go ahead and move on. And I'll tell you a little bit about the facilities to get started. The library has four floors and it includes numerous computer labs, a dedicated electronic classroom for information literacy instruction, smart study rooms with wall-mounted LCD screens and art gallery and the atrium, which is what you're looking at in this slide. It's this beautiful glassed in, glasswalled study area. So, it's, the library is lovely. The facilities are really beautiful. So, in our services is what you'd expect at an academic library and my summary is going to be pretty reference-centric. This is a reference and instruction internship and, like I said, I'm a reference librarian. We have a very heavily used reference desk. We answer reference questions in person via IM, via text, over the phone and via email. And we also do a lot of one-on-one or small group appointments. We also do a lot of instruction. Mainly one-shots, about 150 per semester, a little bit fewer during the summer, of course, because it's a shorter semester, but we usually do over 300 a year. Everything from first semester freshmen to doctoral students working on their dissertations, lots of instruction. So, with this internship, the internship specifically, the responsibilities and duties would include, you'd be able to work at the reference desk, you'll start out by shadowing a library and then when everyone is comfortable, you'll start taking shifts by yourself with plenty of support and back up, of course. And this is a great way to get experience providing reference through various channels. So, it's a really good opportunity. And, like I said, it's really busy. We have a really busy reference desk, so you won't be sitting there kind of twiddling your thumbs. There's a lot of actual questions come in. So, in addition to the reference desk, you can observe research appointments or work with librarians who are doing research appointments. And then the library instruction sessions. So, this is, I'd say this, the instruction is a primary component of this internship. First, you'd observe a wide variety of instruction sessions. We have so many scheduled per week that it's pretty easy to pick and choose ones that would work out with your schedule as an intern and kind of fit in with what you need to learn. So, it'll be really helpful to see lots of different teaching styles and to become familiar with all of our resources. And then eventually, you'll co-teach or take sole responsibility for an instruction session based on your level of comfort and what you're interested in. And you'll definitely get lots and lots of support. So, the other bit is special projects. Well, attending department meetings, of course. I think that's a really good way to kind of get kind of a bird's-eye view of exactly what happens in the reference department of an academic library. We have interesting and pretty focused department meetings. So, special projects. If a librarian has a special project that needs to be worked on, if there's something you're really interested in, if you have a skill set which fits in with a need the library has, can definitely design a special project for you. And one thing we're trying out is having the interns provide a training to the library staff. So, if there's something specific that you learn during library school that you think would be a big, that you think would be an asset for the library, we'd love to get a training in it. And that's also an excellent, that's an excellent experience for you to be able to put on your resume that you've provided a training to library staff. So, that's kind of the last piece of the internship. So, prospective interns, the kind of what we like to see is what's required is basically someone who's interested in reference and instruction. And again, that instruction piece is big because it is a really instruction-heavy internship. We want completion of or enrollment in library 210 reference and instruction services. And what would be good to have is a flexible schedule of course, especially with observing classes if you have a flexible schedule and can kind of come in at different times during the week. That would be really helpful. But we can definitely work around your schedule. Comfortable taking initiative, so if you are comfortable identifying and speaking up when you have something that you're interested in and you want to take responsibility for, of course, that will make the internship more valuable to you. And a real interest in information literacy would of course make this internship more valuable. And I think if you're looking for kind of a one-on-one mentorship experience, this internship isn't really set up that way. You'd learn from a number of different librarians, which I think is really, really valuable. You definitely have a point person, which is myself, but you'll work really closely with the whole reference staff, which is really great. Okay, so if you're interested, email a cover letter and a resume to me, Claire Sharifi. And if you have any questions, you see my email up on the slide there. And if you have any questions, you're more than welcome to shoot me an email or give me a call. And you, like I said, we'd love to have some San Jose State interns. Thank you. Thank you very much, Claire. That was quite interesting and what a wonderful opportunity for students who are interested in information literacy and instruction. Our next presenter now is Michelle. And she's talking about work at In The Stacks. And I believe this is our only virtual internship of the evening. Michelle is on the teleconference, so Michelle, if you would like to present, so we'll be glad to move the slides for you. And I'm going to turn the mic over to you now. Wonderful. Thank you, Pat. Can you hear me okay? Yes. Oh, good. Okay. So, yes, my startup is called In The Stacks. It's a video book review program that I began as a part of my graduate degree. I went to the University of Pittsburgh and completed my master's of library and information science there. And as you mentioned, the internship is virtual, but we're located in San Francisco, so Bay Area interns might have the opportunity to attend events with us and volunteer with In The Stacks. I'll talk a little bit more about that later. So, I'd like to introduce In The Stacks. It's the first video book program produced by Led Brerian. We review books and interview authors to provide book recommendations. And we're expanding into a larger web portal that aggregates and rate video book content from libraries around the world to provide more reliable book recommendations. And on slide three, you can see this is our recent event. We curated a reading of librarian authors, reading from their latest books at Litcrawl in San Francisco this past October, 2013. So, I'll move on to slide four now. So, yes, as I said, the intern duties are definitely virtual, and it would be writing, marketing, and WordPress duties primarily. But depending on what the intern is interested in, there is some flexibility. So, we've messed up for discussion. But the intern duties might include writing book reviews and author interview questions, writing web and marketing copies, assisting and compiling lists of potential partners. And I just wanted to mention that our VP of marketing and community is actually writing the community building plan right now. So, the intern hopefully could assist in implementing the community building plan. And our VP comes from PlayStation. They built a huge community there. So, it could be interesting. And also, we'd be implementing a lot of social media. So, this internship would really focus on new media, community building, and publicity from outreach to partners. And also, that would have a chance to work with a content management person. So, sorry, we use WordPress. And also, you could have the opportunity to assist with our charity projects. And so, the intern is in the Bay Area. One thing that we've been trying to do is to get into 826 Valencia. A few years ago, we donated about 40 books to their library. And there was talk about us going in and actually creating a, like a cataloging system for them. So, that could be up for discussion. And the internship duties would include some research and also some editing, some video clips. So, as an intern, as an intern, you would have the opportunity to work on an exciting new digital library startup that's revolutionizing the publishing media and library fields. And during user testing this year at Stanford Venture Lab, we found that 50% of our bookseller and librarian partners said they'd be likely to add the intersects plug-in module to their website. And that's our book recommendation generator. And the partners are looking forward to testing this beta version. So, if you're interested in being involved in an early-stage startup, this is an ideal role for you. Student qualifications include good administrative and technical skills, including Google Drive, WordXL, PowerPoint, and some WordPress. Knowledge of available tools and platforms in the social media space. Enthusiasm, a great attitude, and communication skills, and ability to work remotely and meet deadlines, and good research and internet skills. So, if you're interested, you can send a resume cover letter and three references to me at MichelleIntheStacks.tv. And also, for more information, you can visit our website InTheStacks.tv to see more information and to get more information about our products. And as I said, before our team is partnered with Kate Peterson, who's formerly at Sony PlayStation now. He's an MBA candidate. He concentrates on community building. And we also are partnering with a programmer and NACC with named MFLock. She is currently our CTO. So, yeah, I'm looking forward to answering any questions you have and hopefully getting your application. So, thanks so much. Okay. I'm next presenter is Hillary Hart Castle from the University of California, Hastings College of Law, and so I'm going to turn the mic over to you now, Hillary. In order to speak, just click on the talk button and then go ahead with your presentation. If you need help moving the slides, let us know and we'll move to the next slide for you. Okay. So, I think I'm in the talk mode. Can you all hear me? Yes, we can. Okay. Great. Okay. So, I'm Hillary Hart Castle and the head of reference services at the UC Hastings Law Library. UC Hastings is the first law school in California. We're a little different than the other UCs in that we are a standalone law school. So, we're not part of a larger campus. We have about 1,000 JD students and a couple dozen foreign LLM students. And we've just started this new consortium with the UCSF Medical School. So, we now offer a master's of studies and law for health and science professionals. So, the law library, and you can see a picture of our circulation desk there, it's located, so the law school is located in three buildings at Civic Center in San Francisco. We're a block from City Hall and a block from the Civic Center BART station. So, we're easy to reach from the East Bay and the Peninsula. And the library itself is located on three floors of one of the campus buildings. And essentially, our primary mission is to serve the research needs of UC Hastings law students and faculty. And although we're close to the general public, we are open to attorneys. And a number of attorneys visit the library daily to do their legal research. But they're generally pretty self-sufficient. So, the library itself has about 650,000 print volumes. But we also subscribe to a number of online databases, most of which relate to legal research. But we do have a few interdisciplinary databases, such as ProQuest and various psychology journal databases. Our students and faculty tend to use the online resources. Many of our books are just gathering desks at this point. And then, as to the library staff, we have 11 professional librarians and five paraprofessionals. Our public services department, which is looking for interns, would include me, two reference librarians, a foreign comparative and international law librarian, and a faculty services librarian. And all of us typically spend two to three hours a day staffing the reference desk. So, we would love to have one or two SJSU interns who are interested in law librarianship join our public services department. When I was an MLIS student at San Jose State, I did three internships at academic law libraries and found them to be one of the most informative and rewarding aspects of my library school experience. And one of them led to this job. So, that was a bonus. So, our hope is that the internship would introduce students to the day-to-day working life of academic law librarians. So, that would involve sitting at the reference desk part of the day along with one of our reference librarians, where they would answer questions from law students, attorneys, and faculty. We would get most of our questions from law students. Our JD students are required to write a research paper to graduate. So, they'll often come to us for help getting started. And then, a strange aspect of legal academia is that the law journals in which professors publish their research are edited by law students. So, as part of that editorial process, students have to gather and check all of the sources footnoted in those journal articles. So, we tend to get a lot of questions about where to find some of those more obscure sources. Our professors don't drop by the reference desk that often, but we have a librarian liaison program. So, each librarian is hooked up with specific professors, and we work closely with them on in-depth research projects. And interns would help out with those. And not all of those are law related. For example, one of our professors is writing a book on John Marshall. And needed to know what the weather was like in Paris on specific dates in 1797. And it's been very, very hard for our librarian who's working on that project to find that information. He's been going through a lot of obscure French journals trying to find information about the weather. And a number of our professors also do interdisciplinary research. So, we'll often find ourselves doing health research or social science research. And then a huge component of our job is teaching and instruction, which we do face-to-face in classes as well as through our website. And on our website, which we designed ourselves and manage ourselves, we have research guides that educate students on how to do legal research in a specific area. So, for example, if a student is researching American Indian law, they could check out our American Indian Law Research Guide to learn what resources are available to them. And we would love for interns to help us update and create those guides. And then our website also has video tutorials that show students how to use our online databases as well as free web resources. And we would love to have interns work on those as well. And then sometime next year, we're going to lose our sixth floor, which used to be happening to a lot of libraries. We're losing it to office space as our faculty grows. So, we'll be undergoing a huge collection assessment project in the spring and summer to determine what we should keep, what we should move to remote storage, what we should toss. So, our interns would assist us with this project by evaluating parts of our print collection and determining how much they get used, whether they're available in electronic format, and the advantages and disadvantages of print versus electronic. So, in terms of the ideal qualifications we're looking for in an intern, we would hope to find someone with some background in the study or practice of law who has some legal research experience. So, someone with a JD or someone who has worked in a law library or in a law office as a legal secretary or paralegal. But that's not absolutely necessary. We would also consider candidates who simply have a strong interest in law librarianship with no prior experience. We do want someone with excellent communication skills. So, someone who can communicate easily with people face to face at the reference desk or who can write a good research guide or a video tutorial script. And ideally, we would want someone who is somewhat tech savvy. For example, we use Camtasia to create our video tutorials. So, you wouldn't need to be familiar with Camtasia when you start your internship. But you would need to be able to learn how to use it with relative ease. So, if all of that sounds interesting to you, please send me a short letter of interest in your resume at hardcast at uchastings.edu. We are very flexible as to the number of days and hours an intern would work. Most of the reference staff is, we all sort of work normal business hours from 9 to 5, but we do have a reference librarian who works until 8.30 in the evenings. And then we have reference on Saturday and Sunday. So, if it would work for you to work some hours and evenings and weekends, we could work that out. So, you can email your letter and resume to me and also feel free to email me if you have any questions. And good luck with school. Thank you. I was pleased to hear so many Swiss grads presenting today. Right now, I'm going to move to a slide that has all of your images. Well, Hillary's image is a substitute there. But I'd like to open the floor for questions so you can see where everyone's from and ask them your questions. And if you would put your mic off while you're not speaking, that would help us avoid an echo in the background.  Well, I have one while we're waiting in case someone's thinking. I was wondering, because our students must register for a certain number of hours and they are required to put in those hours regardless of whether or not there's a lot of work to be done at the site at certain times, I was wondering if you had any type of long-term research project or backup plan or if you are quite confident that you would be able to give a student a worthwhile work for the probably 135 hours during the semester, do you have any backup assignments that you usually keep there for students like that? I can speak for the Placentia Public Library. We are always busy. So, we really don't have any time where the interns are not busy and when if they find that they're not busy, we will quickly find them something to do. There's always program prep to do or to evaluate our programs. We've always got little backup projects if something goes wrong. That's excellent. That's good to hear. Anyone else to have something like that in mind or you're all very busy as well? This is Clare Sharifi from USF. We are also pretty busy, but if there is downtime, students usually, if they're once they get to the point where they're co-teaching or doing the instruction sessions themselves, they can use downtime to prep for their instruction sessions, prep for the training that they're doing and special projects do definitely pop up. So, yeah. Okay. Thank you. Now, does anyone have a question for any of our presenters? That means you did a wonderful job of covering everything. So, what I'm going to do now is just thank you very much for presenting. We will be making a recording of this available to all students. And we will also have a webcast available on our website in a week or so so that additional students can view it that way. And thank you very much. If you have any questions or comments for me, I'd like to hear them. Well, and if not then I'll say good night and thank you.