I'm a Senior Principal Engineer on the team, so no marketing, get nice and technical, hopefully. How many people use or know what RDS is? So most everybody knows, so we don't have to cover any of that. How many people use RDS today? Cool. Great. Yeah. So what we're going to be talking about today is essentially going over what we've been doing lately with Postgres as an RDS engine. It was our fourth engine, but it's very popular with our customers. And we've made a lot of improvements over the last year and a bit. And I'm going to talk about what's changed, as well as the lessons we've learned when working with customers running Postgres. So one of the big news was last week we did another release. So we released 9.36, which just has the normal 9.3 fixes in it. But we also had a small bug in how we do our security stuff, because we have a different super user role. We actually had a problem if you tried to do reset all, it killed your connection. So that was kind of bad. And so we fixed that in 9.36, so you can do a minor version upgrade if you use reset all. Lots of our customers don't and never notice, so we continue to not care. And then the other big announcement was that we now support 9.4. We released 9.41 with all that goodness that all of you probably know and love around JSON B and index improvements. We added the new PG pre-warm extension that came with 9.4 and support for it. And then we bundled new versions of PLV8 and post GIS with the version. So our kind of our philosophy is we don't tend to bump up the new versions a lot in each major release. So on the major releases we want to get up to the current versions on those. So let's talk a little bit about data movement. We get a lot of customers asking us questions about how to move data in or out of RDS. So I wanted to cover that. So one of the big ones is, how do you first load data? You'll see a lot of these remarks are similar to what you would see for any database and any kind of service where you want to turn off the things that are overheads while you're doing your data loading. For us, disabling backups, setting your backup retention to zero is pretty important. Disabling our multi-easy or high availability strategy, because that's going to slow things down a little bit when you're doing bulk loading. And disabling auto vacuum is always good to do. PG dump, standard thing. You want to do it compressed if you can, because that's obviously going to be faster. And on Restore, you can use J to do some parallelism, depending upon what your schema looks like. We don't set the maintenance work memory above the default today. It's mostly due to some sort of lack in our variables, but we're working on that. But you really want to set that to an appropriate size, depending on your instance. And then the other one I really recommend is increasing the checkpoint segments and checkpoint timeout during this, and I'll show you kind of some of the results there. Now, the other thing you'll see in a lot of blogs is this next item, which is Disable F-Sync, or I call it the feature that says corrupt my data, please. We don't believe in that. And in fact, in 9.4, we've actually turned that off as an allowed option in RDS, because we did see customers having problems when they did this. And I'll go into why we think that's not necessary, because with the feature that allows you to turn on synchronous, turn off synchronous commit, we think you get all the benefits of Disable F-Sync without all the tragedy of having a corrupted database. And folks say, well, I'm just loading my data. It's not a problem. Once I'm done loading my data, I'll be fine. But if the machine crashes in the middle of you loading your data, you have to start all over again, because you can't actually prove that you didn't have corruption somewhere in your data. So that's pretty key. So I did some tests just to show how this works. This was an insert-based test, which isn't necessarily a data load per se, but you might load data that way. And in the blue, I have it with 16 checkpoint segments, and the purple with 256. So we're looking at transactions per second on the axis, so bigger is better. And when we turn off F-Sync, wow, we get a big performance improvement, right? This is why a lot of people recommended this, because you get almost twice as much throughput. But guess what? If you turn that back on and you disable synchronous commit, it's even faster, which is kind of interesting.  That's kind of the obvious question. You get a slight bit of improvement over, especially on the 256 segments. But look at the difference. It's 4%. And you really got 104% just by turning off synchronous commit. So if that's what you're doing, you really want to think about, is it really worth 4% improvement to have corruption possibly happen on my database? We don't think so. Now people will say, what about bulk loading of data? Well, bulk loading of data has less sort of commits, typically, in how you do it. So F-Sync doesn't play as much of a role, but it still has some. So here I'm testing a load time, so shorter is better, so it's time in seconds. I'm loading 2 giga data, the blue again with 16 segments, purple with 256, and I have them both on. Now, see the big dramatic change when we go to 256 segments? How much improvement? We got at least a 20, 30% improvement by just changing the checkpoint segments. When we turn F-Sync off, notice we get a big improvement on the 16, but you probably weren't going to run that way anyway. And the 256 only goes down by a little bit. Synchronous commit, almost the same number with 256, so equal. And again, both off, we see a small, minor improvement. Again, 3%, almost within the range of kind of testing. I did repeat these, so I do believe it's faster. But it's not significant. What about on index build? Another thing you have to do when you load data, right? Well, in this case, again, time and minutes, so we want shorter to be better. I just did the left-hand side, which is F-Sync equals 1. So F-Sync's on, synchronous commits off. And then on the other side, we have both of them off. In the blue, we start with the default on maintenance memory, maintenance work memory, 16 meg, and a small amount of check point segments of 16. We see that we don't really have any difference between the two of them, almost equal. When we increase the maintenance work memory to a gig, we actually get the bigger change, right? So this is, you really want to understand your parameters for what you're trying to do and see which ones help the most. And again, if we go and add check point segments up to 1024, we get a further reduction in time, right?  Now sort of a follow-on to this, I was kind of like, well, how much memory should you use to build that index, right? So I ran a little test, which really has nothing to do with F-Sync, but it was just more of interest. I started again with the 16, and you saw that took 29 minutes. And then I went to 4 gig, and I only got 28 minutes. I only got a minute back. I was like, well, this is very strange. Now, again, I had a lot of IO capability on this box, so it's pretty good. But then when I use a gig, I get a faster time. And when I actually did 512, I got a better time. Now, I didn't dig into this a lot, and it's all going to depend on your configuration. But what it tells you is, you want to test these parameter changes a lot. And one of the nice things about RDS is, it's super simple to go take a snapshot, a copy of your database, fire it up on another box, maybe a different size one, and change your parameters, and play around, and do some testing. And when I do testing, I might even have 10 boxes running at once, all doing just different variables to see what is the best option. And you can do that in one hour, so you can get all your testing done and get back to doing what you want to do. So the other thing that we see that's really interesting is customers having issues with vacuuming, and not really understanding how it affects their instances. And one of the first ones that we ran into was a customer did a big data load, and then they were running read-only workload against that. And this is a test that shows the same thing. So we have IOP rates on the left-hand side and CPU on the right. And you see that sort of green line is our read IOPs, or sorry, write IOPs, and the orange line is CPU. So I start this test going. I get 21,000 TPS, pretty impressive. But I'm like, wow, I'm doing more than 5,000 write IOPs. But I'm only doing read transactions. What's going on? Well, when we did that data load with Postgres, it loads data into the blocks, but it doesn't clear everything. So it doesn't know that the transactions are done. So as you're randomly touching them, it's writing them out. So this has an impact on your performance. So I stopped the test. And you can see in the middle box there, I run a vacuum. So I'm doing lots of writing. It's basically writing out all those blocks. Now they're nice and clean. So then we start the test again. Notice we no longer have that high level of write IOPs anymore. And we're getting 45,000 transactions per second, because we're not having to rewrite those blocks as we're reading them. So that's one of those interesting things with Postgres that a lot of people don't understand, is that there are these kind of things that happen with vacuum, and you have to really understand what's going on with your system.  And really, you want to tune these to how much change you have in your data and the size of things. We go with a slightly modified set of defaults. But in general, it's pretty normal. But depending on your workload, you may want to change those. And then the other one is you need to make sure vacuum is doing enough work each time it runs to get through your whole database. Because if you don't, you're going to get a transaction ID wraparound. Now how many people know what that is? How many people have had that happen to their database? Yeah, that is not a fun experience. You're basically down. You're in single user mode until you get this fixed. So if vacuum doesn't get through your tables when you run out of transaction IDs and you need to start reusing them, it'll basically go into this state. And we've had a number of customers that have this has happened to, even when auto vacuum is turned on because of locking or other issues. I think one person was telling us that they had a customer that, on premise one, where they were killing the auto vacuum with a script and wondering why they ran out of transaction IDs. So you control this with max workers. That's how many processes you have doing vacuum at a time. Nap time is every time it finishes with a table, it basically is going to nap. So if you set that to a day, you're not going to get a lot of vacuuming done, right? Because it's going to just skip along. And then the nice thing about vacuum is it's got these cost modifiers in it so that you can do it gently, depending on, you know, for some form of gently, where every time it has to vacuum a row or block, I should say, it has to read that into memory if it's not. And if it's in memory, it's cheaper, right? So there's a cost difference.  So let's say all of your blocks are on disk and you're going at vacuum too hard and you're causing a lot of IO for your system, you could increase the delay, right? And then it's going to sleep a little more between those or, you know, lower the limit. So you've got a lot of kind of variables to work with. But the thing you need to do is you need to be querying your database to make sure that you're not going to get transaction ID wraparound and that you're vacuuming at sort of the correct rate. And there's a number of talks and I think Gabriel has one and a few other people have one that sort of talk about this and I know Josh has blogged about it a number of times and, you know, how to look at sort of where you are on that. One of the other questions we get is, you know, how do I get data in and out? Well, one of the nice ways since 935 that we support is the Postgres FDW, which comes, you know, with the base engine. So instead of just connecting to a regular, another Postgres instance of RDS or an on-premise one, I thought, well, you know, Redshift is an older but still a Postgres front end. It's an eight version, I think eight three, something is their front end. So I thought, well, what will be interesting is just to connect to it and just show that as an example. So now you need to be the RDS super user if you're going to do this, which is the sort of equivalent to super user to be able to create the extension but you can grant that to whoever you want. You do the create extension and then you create a server and as you can see in the server name here, it's redshift.amazonaws.com and they've got their five, four, three, nine port which always messes me up because I'm always like putting in three, two and go, oh, I can't connect. And then you create your user, you create your table and you can see I'm doing a count out of that table and I can select from it and what's really cool is I can insert into it. So you can actually use, you know, this to combine multiple databases you could have your analytics and let's say redshift and you could then select and have people pulling data from redshift as well from your actual RDS Postgres instance. Now again, because it's an older version there may be some incompatibilities. I haven't tested the full scope of everything that's possible but it does point out an interesting way to get data in and out of something like RDS. One of the other changes that we made in the 935 and you know, continues with 936 and 941 is the support of the section replication role parameter. So if you could imagine you had two databases where you'd kind of wired up triggers on tables so they would insert into the opposite tables and that's sort of what I'm depicting here, right? So if you have that and you did an insert into one of them what's gonna happen is that insert's gonna hit that trigger it's gonna insert into the table and it's gonna flow around to the other table and then it's gonna hit the trigger and it's gonna come back and this loop would just continue basically forever, right? Or if you had a primary key on that you'd get a violation, right? So the session replication role allows you to build replication models around triggers to say, you know, don't fire in these certain conditions, right? So it's used by a lot of the replication tools that have been built to, you know, support this. So with the addition of this we now support some of those tools and I'm not an expert by any means on any of these tools but I just set up Ricardo as an example to show that this could be done and Ricardo requires PL-PURL-U, the unsafe one. So we don't support that directly in RDS but you can install this on an on-premise or EC2 instance and configure it and you can talk to an RDS instance and by doing that you can replicate in or out or both ways if you really want to between Ricardo and RDS. So this is another possible way to migrate data in or just refresh data to on-premise. What's really interesting is if you wanted to do multi-master between RDS which I don't necessarily advise you could do it through Ricardo because it's basically reaching out and talking and this does work but it's not a recommended thing but it just shows you the possibilities of the session replication role and some of the tool chain that use that. So that's now possible to do and like I said we haven't looked at all the tools so I can't speak to them but if you do use one of these tools and it's not working how you want with RDS we'd love to hear about it so that we can go figure out if there's things that we can change to make sure it's more compatible with that tooling. So let's move on to talk about sort of scale and availability with RDS and the things we've changed. So the big thing that we released in 9.35 was support for read replicas and I mean these are basically standard Postgres replication. So what we had before that was we had what we call multi-AZ so this is synchronous replication with very fast failover and because it's synchronous you're not going to lose data when you failover. You don't have to make that choice of should I failover and lose a couple minutes of data or should I wait for my machine to come back up? So this is what we really recommend as kind of the primary method for availability but there are some cases where you can use read replicas to enhance your availability. So we use asynchronous Postgres replication and streaming mode to be able to support read replicas. By default you can get five up to five per instance but that's just a limit that we have set if you need more we can do more we've had some customers ask for that. So let's say you have a blogging application you can basically split that application into two sides. One where the people writing the blogs need to write and read the things that they wrote and then there's just all the people reading your blog right? They don't need to change anything per se. So if you split those into two then you get sorry a little disconcerting the you get you can split the eventual consistent stuff and you can see that that's in the purple and that can be directed to your read replicas. Meanwhile the blog writers can go to the primary right? So let's say that instance fails that machine fails that you're running on. What we're gonna do is we're gonna promote that secondary to be a primary and we're gonna fail over to it and this process takes about a minute in total including DNS propagation and everything. So while that's happening if you have read replicas right your blog is still up from a read perspective so you can still be having read availability that's even higher than your write availability right? And then once we finish failing over like I said takes a minute then you're back and you can start writing again right? So there's ways to sort of use read replicas to enhance the availability of your system if you can kind of do these kind of splits between reading and writing. This is very, very common in my sequel and we see a lot of this in RDS with that and I think obviously read replicas are still newer for Postgres so I don't know if it's quite as common. The other thing you can do is if you need to upgrade your primary for some reason or do some maintenance again you have read availability and because you can have multiples of the read replicas you can take them down one at a time and do changes and still have a whole fleet of them up. So that's very useful. Now the other thing of course that's usable for a read replicas is scaling. Essentially being able to scale your reload. So if you have your application again talking to your primary again we recommend having multi-AZ across multiple availability zones but if you put your application web servers let's say in each of the availability zones here and I just call them fake AZ1, AZ2 and AZ3 they're all pointing back to the primary. Latency's pretty good but it could be better for some of your reads if you put read replicas in each of those zones and then you can scale up and you get slightly better latency for your reads because you're not having to go between data centers. It's still probably like a millisecond but it does matter. So this again is a very common practice that we see. And one of the questions I get from a lot of people is like how do I think about scaling up with read replicas? So I have split my application. How much scale can I get? And I use this kind of this little diagram to show that. So let's say we had a box and it's completely utilized doing 90% read and 10% writes. Well, let's say we go shard that out and we put, no it's not shard that out we make read replicas and we split all the read traffic out to those four replicas. So we haven't scaled up yet. All we've done is moved the data around, right? So what's not necessarily intuitive to people is the writes still have to go to all the replicas, right? Because you're transferring the data there it has to be applied. So that workload doesn't go away. Now I mean you save some on the parsing and the running of stuff but it's still an overhead especially from an IO perspective. But you can move all the reads. So now you can see instead of having 90% reads on the primary we only have 10%. We probably still have a few that need to have consistent reads. So now we can scale up, right? You know we can add two X scale and we see that the writes again they go to everywhere. So in this example we're getting at least three X scale because we're only doing 10% of writes. So this graph basically tries to illustrate that you know if you have very high write rates so the yellow there 50% it's not gonna scale well with read replicas, right? You're gonna use 32 nodes to get two X scaling probably not a very cost effective solution for your company. Your boss probably wouldn't be very happy with you if you suggested this. And the 10% one that I show you gets up to 32 nodes can scale up to eight X. But most people actually even have higher read rates like in the 99, 99.9% and can get tremendous scale. We've seen people scale up quite high on this. And I actually as a test I ran with 15 R38X cells that's our biggest box 256 gig of RAM lots of processors. I ran 15 of those replicas off of one master no problem kept up fine was doing I think like 30,000 TPS on writes worked perfectly fine no problems. And the replication delay was in the tens of milliseconds to get to the replicas. So very nice model, love the Postgres replication very solid. Now one of the things that's a little different about RDS because when people think about promoting sort of a read replica there's what happens to the rest of my stuff when I do that, right? So we're back to that same model where we have our primary and our secondary we're doing multi AZ and we have a bunch of read replicas. So that replica in AZ three let's say I wanted to make it a reporting one just cause I'm like, okay I need a box and I just want to use it for a while but I want to make changes. So maybe I carve that off. So I just say promote in our RDS APIs and it becomes a new primary. And at that point your application can start doing writes to it and whatever it wants. If you want to make this multi AZ you can now you can have read replicas off of it but notice how nothing changed in the picture from before, right? It's all exactly the same. So none of your other replicas follow it or do anything like that. It just stays exactly as you are. So that's a little different than some methods that people use. So there's a bunch of questions that we get with read replicas about how to configure them how to think about them. And one of the first things that we talk to customers about is wall keep segments. So we're using streaming replication the primary is just feeding the wall information as it gets it over to the replica, right? So very little lag works great. But what happens when you have like a little bit of a network blip and that streaming replication is gone for a little bit, right? You start to accumulate logs, you know that haven't been replicated on the primary. And this will basically occur until you hit the wall keep segment limit, right? So if you set it to 99 at some point it's gonna start removing things. But as part of RDS we do backup and recovery for you and we're loading the logs every five minutes up into S3 or quicker as you generate them. So that guy's being copied up. So even if it's going to be removed on the primary it's not a problem, we have it. So we can feed that to the replica and we can do archive log based apply to catch the replica backup and keep it current even if streaming's broken. And so this is if you have a very large replica this is good because you don't wanna have to recreate it. And then once it catches up it's just start streaming again once the network's working again. So if you do have those kind of interruptions. So wall keep segments is not super critical because we do do this pull from S3. But obviously when you're applying from a log versus streaming there's a little more delay, right? Because it's sort of a little more herky jerky. So you wanna set wall keep segments to a reasonable size so you don't have a problem with this happening very often. But on the other hand if you make it huge then you're wasting a lot of space on your instance that you otherwise don't need. So there's a balance there. So one of the other interesting ones is thinking about how sort of vacuum effects running queries on your secondaries, right? On your read replicas. So in this case I have a table T1. It's got a row in there called A for the primary key and it's got some data called foo on one of the columns. So on the replica I start a query that says select star from T1. No one's ever done one without a where clause, right? You never had any of your users do that. So it's a huge table. It's gonna run for hours or something. Meanwhile, your application updates that table and sets the new value for that primary key. So in Postgres we end up with a new row in the database, right, and the old one is still sitting there. It's all great. This thing's gonna get replicated down to the replica but it's not gonna be seen by the new query because it was already running. So it's all good, the normal MVCC stuff. But someone runs a vacuum or auto vacuum runs on the primary, right? Well guess what's gonna happen? It's vacuuming out that old record. Not a problem, but that gets replicated. And what you get is this. You get your session killed. So it's gonna terminate it when there's a conflict with recovery because essentially what's happened is you've lost the snapshot data that you were using for your session, right? So this is pretty critical depending on how you wanna use your replicas. I mean if you all have very small fast transactions, you know read and selects it's not gonna be much of a problem but it can be. So PostRisk gives you four, count them, four different ways to work with this. So there's the vacuum deferred cleanup age and there's two delays and one feedback and I'll go through each of those. So the vacuum deferred cleanup age, it's set on the primary, the default is zero and that's what we set it to in RDS and it's in the number of transactions you'd like to delay by and we'll see why that's important.  You hit transaction six. Now let's say on your replica it's reading and it needs transaction four, right? Well that's fine, it should be available except for if you vacuum, right? All of those old records are gonna get removed because no one needs them on the primary. So you could set vacuum deferred cleanup age in this example to two, which means keep two transactions. Now this is of course across the entire cluster so it's not like per table as I'm showing this example but I'm just trying to make it simple and this would allow it to not vacuum out those last couple records. So this is obviously gonna have an impact on how many old things are sitting around in your database and it looks like it works pretty well but the problem is you gotta set it in a number of transactions and how do you figure out, like you'd have to kind of be like my right transaction rate on my primary is X and that turns into X number of minutes onto my secondary so you could do that but it's pretty complex and it's gonna be a little difficult to understand when people come and say my query got broken why was that and you're like, well because we had too many transactions going on in the primary, right? So what we see used is a little more is the max standby archive delay and streaming delay and basically these go with the various types of apply. So we use streaming replication mostly so the default is 30 seconds and that's what it's set to and that's what we run with and so you can think about it this way when a conflicting thing comes in you're gonna delay that replication apply really for 30 seconds, right? And this happens on streaming but remember how we said we could do catch up via logs so you also wanna think about setting the archive delay, right? Because if we do have to go to archive you wouldn't wanna have your queries again die there, right? And you'd wanna have those matching values on those because otherwise you're gonna get kind of weird behavior depending on which mode you're in, right? And because it might switch back and forth that wouldn't be good. So this one works pretty well but again is a trade-off, right? Because what happens when you set this to an hour? Well, your queries are gonna complete but someone wanting current data isn't gonna get it, right?  So new sessions coming in are gonna be like, man. The other funny thing is not all your sessions will see the max delay, right? When you think about this, session one comes in it's doing something, this thing comes along that's gonna break its snapshot, it starts counting for 30 seconds but another session comes in 15 seconds after that. Well guess what, it doesn't reset that timer, right? So 15 seconds later, bam, you hit the 30, it's gonna apply that and it's gonna remove that data and your second session's gonna fail, right? So essentially you would have only seen, like on that session, 15 seconds of delay, right? So this one's a little tricky in that as well. So if you only have one long running thing or all of your things are more or less the same time then it's probably fine but it can be a little tricky to set. The other one that's kind of really useful is hot standby feedback and so we get the same picture again and the way to think about this is like it's on the replica and it's off by default in our configuration as well as just in postgres in general. It gives basically a backwards channel from the read replica to the primary to send information about what's going on. And if you think about it this way, let's say you have a select on the replica. Now if that select would have been running on the primary the vacuum would not remove rows that were needed for that select, right? That's how postgres works, wonderful, right? It's how it has that consistent view. Well, this feedback basically tells the primary that same information to make it almost think in a virtual manner that you're running that query on the primary, right? So it's gonna do the exact same thing that it would on the primary, it would block vacuum. So if vacuum starts up, it's gonna say, oh no, there's a session using that data, I'm not gonna do anything, right? Your query will finish. Now again, there's a catch on hot standby feedback because it's part of sort of the streaming replication protocol it has to be able to talk to the primary, right? If the network breaks, well, guess what? Auto vacuum is not getting any feedback that there's sessions going on and it will start vacuuming. And because we do apply of logs, that vacuum will go through the logs and will get be applied onto the read replica and it would break your session, right? So what's usually recommended is a combination of both hot standby feedback and some of the delay settings in combination to really meet whatever needs you have for replicas. So that one's just, you've gotta kind of understand what your application's doing and then sort of tailor it. But hopefully that kind of helps with understanding that. When you get a conflict, you can actually see these. There's a PGSTAT database conflicts table. In this case, you can see I had a snapshot conflict where basically I had to remove the snapshot that my session needed and I got that error and this is what happens in this table. So, it's pretty reasonable. We also support, obviously with replication, you can look at the PGSTAT replication view and you can actually see if you're in streaming mode, what modes you're in when you're on your replicas. So that's nice and handy. We do provide a replication leg metric in CloudWatch. I should put a graph in here of that that just shows where you are behind. One of the interesting things about Postgres is the typical way to measure replication leg is to look at the last transaction that was applied. So if there's no transactions going on, you'll get a lag up to the last point. We do a little heartbeat one every five minutes. So you might see five minute replication lag and then as soon as you start doing transactions, you won't see any, right? And typically you'll see less than a second in most of our situations. So the other thing that we added in 935 was support for PGSTAT statements. How many people use this extension or shared libraries I should say? Okay, good. So you have to go in, we don't load it by default because we have some very small instances and we don't wanna necessarily add extra load to every customer but you can go into your parameter groups in RDS and you can add the PGSTAT statements to your shared preload libraries and then you can create the extension if you want and you get that good data that shows you exactly what's going on. A number of our customers sort of collect this data and stick it somewhere so they can see it on a time basis so that's super useful. Other things that we did in 935 and 936, we added support and again this goes on for 9.4. We added support for PLV8, the V8 JavaScript engine. We had a couple customers that said, you know, we really need the test parser extension. We had left it off because we read the documentation and we went, I don't know what this is just like but they'd used it for something and they said they needed it so it's part of the, contrib, we threw it in. We had a new post GIS version that we added support for grid shift files and then as I said, now shared preload libraries we can add things to that. Currently we only have the one which is PGSTAT statements but in the future if we find other ones that we think are useful or if the customers tell us we'll be able to add those. So that's sort of the deep postgres stuff that we've done in the last little while but one of the other interesting ones that I wanted to talk about was sort of some of the other exciting stuff that's sort of coming out of AWS and is in RDS which is really burst mode computing which I think is quite novel because it's definitely something different than most people run on premise. And we currently have two offerings in that. One is around IO capabilities. We have what we call GP2 or general purpose 2 EBS volumes. So they're SSD based like our provision IOPS but instead of saying I want 1000 IOPS you get three IOPS per gigabyte. So if you buy 100 gigabytes in a volume you get 300 IOPS guaranteed. Now if you don't use those 300 IOPS they basically build up and then you can use them later and you can burst up to at least 3000 or higher on RDS because we use multiple volumes on very large instances so you can get actually even higher bursts and that's really useful for variable workloads and I'll show you some graphs on that. And on the other side from a compute basis we now have the T2 line and we have micros small and mediums that do the same thing. They have burst.  Now the nice thing about the T2 is that we actually have CloudWatch metrics for these so you can actually see hey, how much credit do I have left? How am I burning it? And we're pushing the EBS folks to do the same thing with the GP2 stuff because we've had a lot of customers kind of go I don't really know where I am on my usage and obviously we want them to be able to see that. So how does this look in practice? So I ran a benchmark, read only workload and you can see that I have CPU and in the orange and the blue is read IOPS so I started running and I got 6,000 transactions per second and then we get throttled down here so we get down to about 3,800 TPS and that's because we used up all of our GP2 credits so we were doing a lot of IO and you can see the kind of dramatic drop and then we run for quite a while and then we drop again down to 2,200 TPS because we've exhausted the T2 credits so you can see it's pretty in this kind of test it's pretty obvious what's going on but this is actually how you can see it in for the T2s, the credits you can see that nice orange line as it goes down it's the credit balance, right? And it'll decrease over time as you use up your credits and when it hits a zero you get throttled to your minimum and you can see the drop in my performance of how much CPU I was using, right? So why is this important? Well, it's all about price performance so I went and looked at sort of burst mode versus some of our classic models and some of our provisioned IO systems that our customers use today. Just did a nice simple test, 20 gig of data, read workload, transactions per second on the vertical axis, and hours on the bottom. So I started with one of our classic instances which was an M1 medium and 200 gig of standard storage so this is the older EBS magnetic storage. Comes in at just about 58 cents an hour and it does about 1,200 TPS not bad, not bad, very consistent, right? So let's go and pick a newer machine, an M3 medium. So same amount of memory and this is a fairly heavy, I mean, because we don't have a lot of RAM so it does a lot of reads. We did 200 gigabytes of storage again but this time we did provisioned IOs and those cost more but you can see that we almost doubled our performance, right? And we saved money because we needed a lot more resources to run this, right? And we basically paid for them so little more efficient CPU and a lot more IO. I thought, okay, this is good but I also noticed the M3 medium was tapped out on CPU so I thought well what happens if I go with like an M3 large which has twice as much memory, twice as much CPU? Well I only got a slight improvement because guess what, I had the same amount of IOPS and this test is very IO centric, right? I mean, and it's specific to this one but even in this, notice that we went from about 2,500 to 3,000 transactions per second, right? So we got a good, that's a decent bump, right? And we only went up in 10 cents an hour so it might be worth your while but so this is all good but now let's talk about the burst mode ones. So we went and got an M3 or a T2 medium, same amount of RAM, right? But we went with 200 gigs of GP2 storage. Now you'll see right off the bat we're getting amazing performance, three and a half thousand TPS, right? But it only lasts for two hours because essentially we ran out of GP2 credits and we get throttled. And you can, you know, cause we're only got 200 gigs, we got 600 IOPS so it's not really a fair comparison, right? 600 IOPS versus something that had 2,000 but look at the cost, you know? 10 cents an hour, 11 cents an hour. So if you only needed it for a couple hours a day this is a great value and save yourself a huge amount of money but I thought, hey, why don't we just throw a little more space at this to get a few more IOPS and see how that works? So I take a T2 medium again but I do a terabyte of GP2. So look at that result, 6,000 transactions per second, right? Because now we have 3,000 IOPS guaranteed plus we can burst, right? So we've got more baseline than I had before and it's 23 and basically 23 and a half cents an hour. So even when we're throttled right at the end now at this point we've hit the T2 limit so we're throttled on both CPU and IOPS, right? But we're still getting almost the same amount as the M3 medium, right? And this is 17, 18 hours later and it's half, you know, about right around a little more than half the cost. So it shows the economics of these models and I think you're gonna continue to see a lot more of this kind of work from us in the future and so if you can think about when you design your applications and your database to kind of take advantage of these things you can really save a lot of money. Now GP2 in general I would just, you know it's a great solution for tests and for sort of, you know, medium level of IOPS because the cost model is so much better. So I recommend it to all my customers, you know depending on their workload. So with that I'll wrap up and I'll take questions on anything and you know, happy to answer on RDS in general or Postgres, you know, RDS. The question was are we supporting replication slots not currently, it's something that we're looking at. It does change a little bit of, you know how many logs end up on the host and so we're trying to understand how to do that in a way that will be understandable for our customers so that they know, you know that there's a lot of space possibly in use if they create slots. So that's something we're definitely looking into. And the question was have I been able to tune auto vacuum settings so that you don't have to do vacuum foals? Well, we work with customers but every customer is sort of unique for us. So I mean, I can't specifically say that we've kind of gotten to a point where we could do that like I can't say I could do that for everyone. I mean, most of our customers are pretty happy with how their vacuum stuff works but you know, it's really case by case. Right, so the question is what is their priority for doing cross region read replicas with Postgres? Yes, our customers have asked us a lot for that. And it's definitely, you know for me what we have with my SQL is what I consider sort of a minimally viable product you know, from an overall perspective and so that's kind of the goal for all of our engines is to have sort of the same functionality including things like cross region because we get a lot of customers asking about it for DR purposes which I assume is what you're looking for. Yeah, so it's definitely something that we wanna do. We could talk more offline on sort of when that might happen. Do we have any statistics on adoption on 9.4 kind of versus 9.3? Well, we only launched last week so it's quite early but we did see a lot of crates. We were seeing more crates I would say on 9.4 on those days than we were on 9.3 but obviously there's just a lot of enthusiasm around testing and checking it out so I think it's still early to know but typically what we see in our RDS engines is we don't make the new major version the default for a little while because we wanna make sure everything's working really well with it but once we do most people just then pick that as the default so then the customers tend to move up pretty rapidly. It's one of the things that we sort of encourage is to move along with both minor and major versions when possible. Other questions? Oh, yes? You mean on the wall keep segments? No, I mean it's the Postgres Maximum, so. Which is something that really helps. Yeah, no, I mean you could fill up your entire three terabytes I think with wall files if you wanted to. I've set it to in the tens of thousands easily and had no problem, so. Yes, yeah, so we use one shared storage because it's kinda too hard to explain to people like oh hey you only have this much space for wall or so that way it's pretty understandable. Okay, other questions, bye. Yes and no. EBS does not currently support as many IOPS as we do so we're gonna continue to do some of that. We're working with them to try to get to closure on how we do things soon but yeah at some point we will probably be more following their model so that's why we don't advertise, we only say you can burst above 3,000 because at some point it probably will change to being their model. Right, the question is do we really have tools around kind of figuring out what your optimal sizing is? Not really direct ones, I mean we have pricing calculators that obviously you can kind of throw what you might use in and we provide CloudWatch metrics to look at what are you doing for IOPS, how much memory you're using, how much CPU but we don't have anything that integrates that all together to give you kind of a hey you could be running on a smaller instance or maybe you might need a larger one. It's definitely something I think that over time we'll look into kind of trying to give more advice but that's where we are today. Sure, so the question is what kind of model do we use for promotion? So just to clarify, our multi-easy is not using a Postgres replication, it's synchronous and we do use a Quorum model to basically do that. Pretty standard model. For when people promote read replicas that's just an asynchronous promotion so there's no kind of Quorum or anything done. If you tell us to promote it, we'll promote it and good luck. I mean that's the thing with asynchronous replication. It's why we offer the feature because many people, we have gaming customers and other customers who are like, I don't mind losing a few minutes of data and they like that model but for many of our customers that say I run a transactional system, I can't afford to lose anything. We believe synchronous replication with sort of that model is the right way to go. Anyone else? Yes? Yeah. Yeah, I know, I was like, I was waiting for someone to ask it. Yes, saw that. Yeah, yes, we do plan on supporting major version upgrade. Again, it's one of the core features that we believe in. We didn't want to wait on 9.4 before having that done but because a lot of people just want to test right now but as it turns out a lot of people want to really upgrade their database right now. So it's definitely something that we're working on. Sorry? No, that is what, sorry. Our major version upgrade will be using PG Upgrade, sorry not to be clear, yes, it'll be the normal PG Upgrade process with just the automation of it and making sure it all works perfectly. Other questions? See any more hands? Okay, well thank you very much for coming. We have a booth downstairs, I'll be around for the rest of the week. If you have questions, any of us, you can see us in the black t-shirts. We're happy to answer any questions about RDS, AWS, any of the kind of stuff. We know a lot about EC2 and EBS and some of the other things, not everything about AWS but happy to answer any of those questions as well. And we have shirts down at the booth so if you want a shirt, come get it.


If you've been with the packet hacking village and the speaker workshop before, as I said, this is the fifth anniversary of a packet hacker of the speaker workshops at the packet hacking village. And I think you've spoken at L5. I know you were out of a lot of missing wands. Like an old friend of ours. And you've given quite a few talks. And so it feels for me, it just feels like for me, not only the annual tradition to come here to DEF CON and the packet hacking village, but also to give an introduction to both Chet Hosmer and Mike Grego. Thanks, Ming. Thanks for having us. I'm going to let Chet kick things off. Great. Thanks, Mike. Hey, welcome, everybody. We're going to talk about covert TCP with the twist today, give you a little history about this, which is really kind of interesting because we've come 20 years from the history that I'm going to talk about, and we're no better off than we were 20 years ago. So hopefully we can work together to correct that problem. Can you guys hear me okay in the back? Everything good? Okay. No? How about that? Yeah, that better? All right, good. So anyway, welcome. Mike and I have done this talk together, similar talks together for quite some time, so hopefully you guys will enjoy this. Just a quick introduction. Mike will do one for him as well. We've been working on the area of secondography and data hiring I have for over 20 years now, before anybody even knew what it was. And was working with the Air Force Research Lab back in the mid-1990s. And there were two groups at the Air Force Research Lab. There was the Offensive Information Warfare Branch and the Defense of Information Warfare Branch. I worked on the defensive side, and we first started looking at secondography and data hiding. My first course of action was to go steal the best guy that was on the offensive side and have him come and work for us on the defensive side. And really start to look at how we're going to actually detect, disrupt this kind of activity that's out there. And since then, Mike and I wrote the book Data Hiding a couple years ago together. Mike kicked it off and brought me into that so we could actually share the book, and it's been really, really popular out there as well. And since then, I've written several, and Mike has written several as well. So I also teach in the cybersecurity graduate program at Utica College, and I teach in the digital forensic graduate program at Champlain College. So if any of those folks, and sometimes there's folks in the audience from my classes, if you are welcome from that perspective. Let me turn it over to Mike to have him introduce himself, and then we'll kick this off. Awesome, thanks, Chet. So my background, while not nearly as many years as Chet, been focusing on Stego as well for quite some time, I think about 17 or 18 years, was really inspired by it kind of organically and soon after had actually met Chet. And we've shared research and ideas ever since then and collaborate quite a bit. Currently work for a company now, 802 Secure, we focus on IoT security. And so a lot of the current testing involves an IoT lab where we do a lot of analysis, reverse engineering, and some of that will be incorporated in what we're covering today. Furthermore, along the Stego front, just out of a talk that we did in Track 4 on deep neural networks for social Stego, so leveraging social media for covert communications, as well as ML for automating that from an offensive ethical standpoint. And then something else, having spoken at DEFCON 12, released a tool called Steg Spy, whereby the idea behind it was to kind of take a little bit of a different approach to Steg Analysis in identifying hidden content by saying, well, with all the really cool things that even Chet and his company were doing at the time, is there a way you can go about fingerprinting the actual applications that are being used to perform this Stegrenography? And having mapped that and fingerprinted that, understanding what kind of fingerprints or remnants that they would leave behind in the image and look more for images that had evidence of a tool being used to hide the data rather than looking for the hidden data itself. All right, so we can go ahead and jump into it. I think Chet has the first part. Great. Thanks, Mike. So back in 1997, when we were starting to really get into looking at Stegrenography and how it was going to affect, I read this article written by Craig Rowland in first Monday, the Hacker publication. It was published on May 5, 1997, so single to my own 1997. I can't forget that date. There's a link here in the brief that you can go get the actual article that Craig wrote and I want to kind of walk you through what he talked about then. Is Craig in the room, by the way? Did he come? No. Too bad. Anyway, I met him once a long time ago, a really interesting guy. So anyway, the concept was it really intrigued me because we were looking at Stegrenography from the perspective of hiding information images, audio files, videos, that kind of thing. And what Craig did was he kind of broke the ground in looking at this from a covert communication perspective and actually being able to hide information in protocol streams. And we're going to talk about that today. So that's kind of just the kickoff of this. So just kind of a quick definition that came out of the 1985 Orange Book from the Department of Defense to define what covert communication actually is. It's any communication channel that can be exploited by a process to transfer information in a manner that violates this security policy. So basically the concept is I want to be able to leak information out of the organization. I want to be able to communicate with people without anybody knowing. So again, just to find the difference between encryption and Stegrenography, encryption is all about keeping information private so that only those that hold the keys can actually decipher the information unless you have a lot of time in big computers, right? The purpose behind Stegrenography is to hide the mere existence of the message, right? So a very different concept of what we're trying to do in Stegrenography versus what we're doing in encryption, right? Kind of much more dangerous from that perspective and you can kind of see the idea behind that. So in Craig's original article, he basically had three different methods that he actually defined on how we could do this using the TCPIP protocol. How could we actually hide information in these protocols? And as I mentioned at the outset of this, most data structures, most protocols, most file structures have this limitation where there is no concept of being able to prevent the data structure or the protocol from carrying illicit information. It's not part of the thought process of those folks that are building that. Last year Mike and I spoke about the MP3 ID3 header and the vast amount of information that I can hide in that and transfer data and we showed how you could actually post music on popular websites and actually have that information included in the download. So I could covertly communicate that way. So this has kind of come a long way but let's talk about the three different methods that Craig had defined. The first couple are fairly simple but kind of interesting and they still stick today. I have a lot of friends that work for defense contractors, big banks and that kind of stuff and I do this to them all the time to see if their defenses can detect any of this and they never can. So the first thing is this is what the TCPIP packet header looks like and by the way, everything I'm going to talk about works with IPv4 and IPv6 so there's no difference. The same vulnerability exists. We didn't fix any of these problems when we went to the later version of IP. So we're going to talk about the identification field within the IP header. We're going to talk about the sequence numbers and the acknowledgement numbers that exist in the TCP packet for structure. Mike's going to talk a lot about wireless and a bunch of other stuff as well but I'm going to focus just on what Craig talked about originally so you can kind of see how we can actually use that same technique today in order to leak information. So the basic concept is pretty simple. We want to be able to use fields within the TCPIP header to convey information and since these are standard protocols that you have to obey, these fields must be there. This information has to be part of the header in order to be able to send this out whether it's in the IP header or whether it's in the TCP header. So basically looking at this information, we're looking for areas within those headers where we could actually hide information within those. So let's take a look at that. The first one we're going to look at is the identification field. So the identification field in the IP header is specifically used so that if, let's say that I'm transmitting a packet, an IP packet, and it has to be broken down, right, it has to actually be fragmented so that packet would have to be fragmented. So the ID field is there so that if the packet becomes fragmented we can put the pieces of the packet back together. The ID field is used to basically provide a number so we know the sequence of the packet as if it was broken down. Today this rarely happens, especially with smaller packets. They're typically not fragmented so we can actually use that field as is to put any information in it that we want. So if I'm the sender of this particular packet, I can replace the identification field with data. For example, I might put a value, it's a 16-bit number so I could put a value in there and then you could have a, Mike could have a constant that we're going to divide by so I can basically put a number divided by a constant and then Mike will get the letter that I'm going to transfer in that particular field. So that's the first example. Really simple, just using the identification field in order to be able to do that. So as an example, this is what it would look like. Basically I'm going to send this identification field that is here, the number you can see is 17-152 is what I put in there. If I have a constant of 256, it gives me the number 67. So converting 67 from ASCII to a letter is the letter C. So by just basically putting that value in the identification field I've transferred the letter C to Mike. He's going to receive the message. Real simple, you've got to do a lot of these in order to be able to pass the message. If we want to do this with a twist, I could use a code book. I could have standard messages that I want to communicate and basically it could just be an index into that standard message. So now I can from within an organization I can basically push out information by just encoding information in that single field that I want to do it. So this is the simplest form that Craig talked about. So let's move to the next one. This is the manipulation of the initial sequence number. This is one that probably has been most talked about. I actually think the third method is really more interesting. But what this is is that whenever we establish a TCP connection we have to basically provide an initial sequence number. This kind of goes back into history with the Robert Morris attack where the initial sequence number wasn't random. So in this case the initiator of the communication which typically is the client is going to define this initial sequence number so that the server can respond with an acknowledgement of that sequence number and now the sequence numbers will increment by one for every packet. But since we get to choose what that sequence number is the same as we could choose whatever we wanted to put in the identification field we can actually use that value in order to be able to convey information in a similar manner that we did. Now remember this is a bigger field so this is a 32-bit number so we could actually transmit a byte, a word, a double word, a quad word of information with every time we establish a connection to a server. So that's how this one works. So basically the way this works if I just take an example packet this is what that TCP header would look like if I extract the information from the header we can see what the initial sequence number looks like down here. Again it's a very large number 32 bits in length that we can encode anything we want into that initial sequence number. So if I take that to the next level and I extract that sequence number and again in this case I've chosen a divisor something that I want to divide that number by to actually convey the information. Now this looks really simple because once you've figured this out you can see what the message is but clearly what we do is change the divisor for every sequence number that we send. So that you don't know what it's going to be Mike has the codebook so he knows what it's going to be in this case it'll convert that to the letter H by dividing those two numbers. So all we have to do ahead of time is exchange the fact that I want to be able to communicate this and all we have to do is have what the divisor or the method is to communicate. So that's pretty cool. Very similar to the identification field we do this. Do we defend against this today? No, nobody looks at the initial sequence number in order to be able to do this. The third method is really interesting because you can actually hide the identity of the sender and of the receiver. So let's talk about the third method which I think is the most interesting one that Craig defined. And I just did an experiment with this last week to see if I could actually do this off popular servers and it works really good. So method three the concept is that I want to expand on method two and actually on method one a little bit and what we're going to do is we're going to have the I want these two clients to be able to communicate with each other. And I want them to be able to do this fairly anonymously so I don't know who the sender or the receiver is of the packets that we're going to exchange. So this is a little bit more difficult to do and we're going to use this method called the bounce method. So what we're going to do is we're going to send a request to a server basically that same packet that we talked about in the previous example. But I'm going to spoof the IP address of the sender and I'm going to have the IP address be of where I really want the response to go. Okay? So I've got these two people that want to communicate. I've got the sender and the receiver and the sender knows the IP address of the receiver. So when it basically sends the packet to the first server it's going to bounce off that server and it's going to go to the second client because I've spoofed that IP address. The second client's not going to respond to that but he has the initial sequence number because it comes as the acknowledgement from the server comes in that packet and since it does he has that piece of data. So now you might say well that's kind of interesting so let's take a look at how this actually works. So basically what happens is client one sends the send packet to let's say a server out there on the internet someplace with the encoded information in the initial sequence number but because it spoofed the IP address of the sender the response or the acknowledgement is actually going to go to client number two just as I kind of talked about including the encoded initial sequence number which again using the method that I talked about earlier you can just divide that and get the actual data out of that. You can use multiple methods in making that happen. So now we've actually sent information from client one to client two using that method. Now we've sort of masked who the sender and the receiver is but you might say well wait a minute the server is going to know where that request came from and where the response went to. Not going to know the sender but it's going to know where the response is supposed to go so it could track that. Well you don't have to use the same server right? I could pick a thousand servers out on the internet and send each one a single send packet and have the response go to the client. So now I've got these thousand servers out there that are going to respond to this and this happens all the time the packet's going to be dropped but now I can bounce this off thousands of servers so that now I can't even identify who the recipient is because the recipient is different for each of the servers that we have. Does that make sense? Everybody kind of understand it? So basically the bounce method really kind of extends this attack to the level where now I can really hide the identity of the sender which I could always do even if it was only to one server but now I can hide or mask the identity of the receiver of the message because it's going to come from a thousand different servers if I want to send a message that's of the link thousand bytes I've just got to pick a thousand different servers to do this. And I did this with Google and Amazon last week just to see if they would actually recognize the fact that I bounced this send packet off and it received it on its other client and it works perfectly, nobody cares. Right, so 20 years and a few months later the methods that Craig had defined really work today. We haven't improved the design of our protocols. We haven't secured either IP or TCP in order to be able to prevent this from happening.  We'll talk about some more sophisticated pieces of this as we go forward in the talk. I'm going to talk about a couple and Mike will talk about some as well. But to just kind of set the stage that we've gone 20 years and this has been a very public information about the methods that Craig had defined 20 years ago and we're better off because we tend to be chasing the shiny object. We don't actually focus on fundamentals of our protocols, of our data structures, of our file structures in order to prevent this kind of leakage of information. We tend to be focused on things that are kind of outside. What is the latest attack that we have? What is the thing that's going on? And we don't go back and look at the fundamentals that we're trying to protect against in order to do this. So just, that's my soapbox for the day. So I'm going to turn this back over to Mike who's going to talk about UDP and IoT and how those didn't get any better either. Even though we were able to create these brand new protocols and brand new techniques in order to communicate with these devices, we actually may be worse off, right? Great, thanks Chet. So as of late I've been doing a lot of testing and modeling in an IoT lab I have at home and when I came up with this idea for this talk I said gee, you know, I think that could be a good launch pad and make a nice addition that would be very complimentary to what you're going to, what he was speaking about in terms of TCP and reveal this from a UDP standpoint. In terms of IoT we've seen numerous articles about the inherent lack of fundamental security across IoT both within the devices as well as how they communicate and to that end a lot of these devices support UPMP or more specifically SSDP and this allows you to have an app that discovers a new device on the network maybe even configure it even allow devices to speak to one another machine to machine if you will and how a lot of this communication is performed is believed to be maintained locally on the network through use of multicast broadcasted packets on the local network so I said UPMP's been around for a long time as well Microsoft been around for numerous years yet again we're probably not any better off with UPMP than we were when Microsoft first released it and now it's been adopted as part of IoT and in addition to that it's been extended into IPv6 with all the backward compatibilities as well but in terms of what I like to call stupid simple discovery protocol or really simple service discovery protocol being a component of UPMP you have a variety of packets that are leveraged in terms of identifying these devices on the network or those devices advertising themselves on the network so everybody can find one another M-search for example is a discovery packet can be sent by an app, can be sent by a device to go out there and discover the IoT devices and then furthermore there's another type of packet called a notify packet typically devices who have employed this SSDP type capability will wake up from time to time and typically on a regular frequency advertise themselves peeling the onion back a little bit further what are the characteristics of these packets well there is what's called a ULA OPT field this is for unique local addresses and site-routable but specifically the OPT value within that carries a URL which is an optional field I believe hence the OPT label within the packets but in addition is used within both notify and M-search messages the second thing is that supported not only in IPv4 but IPv6 for backward compatibility so I said to what extent could this be used for communicating covertly is this a free form field is this a field that is used by the vendor of the IoT device to basically define characteristics about the actual IoT device so we've got the IP address serial numbers other things that are embedded within these types of packets but in addition to that this OPT field usually carries a standard URL that really points back to the schema or more importantly the information about UPNP so in the spec there's kind of limited information about it but I said in playing around with all these different fields this was almost like a free form field you could put a different URL in here and it would point to command and control furthermore it doesn't even have to be a URL so you can go in here and put any kind of text any type of content clear text or ciphered within this field so that's exactly what I did to see to what extent this would be communicated across the network and successfully received by the actual intended recipient or furthermore if it's broadcasted to all the devices on the network and it worked completely successfully whatsoever and certainly could be a method of that which you could communicate covertly using UDP in what would arguably maybe be a different technique but similar to what you're trying to accomplish with TCP so it also makes for a great dead drop so you could just if you're communicating this covertly and it does harbor a URL that could point to something benign like a WordPress site content as well and this just gives you kind of a a zoomed in view of this actual field that comprises this actual packet that's part of SSDP so again designed to incorporate a URL of your choice typically used by the vendor who creates that particular IoT device but as I mentioned doesn't have to be a URL put any kind of content you want in this field to whatever extent you want you don't have these limitations of that yet but nonetheless you can certainly hide a variety of content in here if you wanted to further obfuscate that you could of course go ahead and cipher that as well and well although this is designed for sort of the local network what we've seen in terms of IoT with Mirai and even other variants that predate Mirai and their infections of IoT devices is that this coming in from the internet would potentially target your or your router with your firewall and furthermore would those packets be allowed into the internal network and thus with this type of thing could those packets escape out or have a communication with the actual router itself so theoretically the router itself could actually pass this on to the internet into an intended recipient and in addition a lot of these fields could be exploited in that manner so in the diagram here we've got a two-way communication between two different IoT devices or an individual to an IoT device and vice versa you could send this hidden content in what appears to be normal discovery protocol information leveraging SSDP that's actually harboring hidden content within the actual packet probably undetected by any of the local IDS MPS or other features you may have for your security products on the actual network and then receive this on the other end to actually reveal the hidden message the other area that I did a lot of testing with and this is actually covered in our book has to do with Wi-Fi and layer 2 packets so in terms of wireless there's a variety of beacon packets probe response probe requests a variety of different types of packets that are used within that one of those within the beacon packet is something called an information element field kind of used in exchange but this field sometimes is leveraged by the vendor so in a beacon packet you've got a number of fields here including the SSID and the BSID but you additionally have this information element field and that information element field sometimes is used by the vendor and they may say hey I'm an Aruba, I'm a Motorola, I'm a Cisco but many times this field actually goes unused and it can hold up to 256 bytes of actual data so if I wanted to leverage this to hide information that's a decent amount of information for actually hiding text and as Chet mentioned earlier if you send multiple packets which I did with this with my DDWRT you can actually have a massive message or content that's split across these beacon packets such that they could be put together on the receiving end to reveal the hidden content or the message so I went ahead and did that and hid a message within the IE field it totally was accepted by the protocol, it was transmitted I sent it from my DDWRT device and monitored this and also determined that a laptop I had was receiving these packets and that you could communicate this back and forth I was using Scapi Scapi, yep, automated with Python, yep, so and so what would be the application of this? If I modified a beacon packet and inserted a message into the IE field even if I used something fundamental like Air Replay to replay a PCAP file or I automated it in Scapi I could communicate covert messages over enemy lines for example and although people in the neighboring airspace would just see a whole bunch of access points all send in beacon packets how would you distinguish that out of, you know, 10, 50, 100 access points seen in the local airspace that one of these is communicating covertly and therefore over you know enemy lines or to a different building the other side of the coffee shop for that matter provides a really covert way of doing that versus actually using a wireless ad hoc network for example in which a Russian spy was actually caught using that technique and as I mentioned I prototyped this and configured it within Scapi so that you could run this using Python and essentially I called this Wi-Fi stego stuffing the reason I called it that was back a number of years back Microsoft had done a bunch of research around beacon stuffing I believe the intent behind it was if you walked into a store and you had your Wi-Fi enabled on your smartphone they would detect when you came into the store because they would be looking for probe requests for example from your device to connect to networks that you had previously used but in gathering that information they could actually distinguish that if you had their store app on the device that I see your device you have the app associated to this particular MAC address let's say for your Wi-Fi I know you're in the store I'm going to respond with a beacon packet stuffed with a coupon that's going to flag it in the app and give you a one time use coupon was sort of like what they were looking to do at that time yeah exactly I didn't want to mention any by name but yes exactly and there's others too so it's kind of exploiting that method in a different way but more for covert communications essentially so in terms of beacon stuffing I said I'll call it Wi-Fi stego stuffing instead and with that I'll turn it back over to chat thanks Mike so let's take this to kind of a different level a number of years ago we started to take a look at how we would hide data within streaming protocols and especially things like audio streams VoIP streams that kind of stuff to see if we could actually use that as a method to be able to leak information out and to be absolutely honest it's really a pain in the neck and part of the reason is that those packets that are part of either the VoIP stream or a media stream they're encoded they use different encoding methods there's a very small amount of information in every packet right it's not large it's very brief it can be lost because it's unreliable right and if you lose a couple packets no big deal you can still hear the audio just fine and that kind of thing so it didn't seem at least the folks that we were talking to didn't believe that it was a viable method to be able to use in order to be able to communicate information using those streaming protocols by hiding information by modifying the payload section of those packets okay it sort of makes sense just kind of a complicated thing to actually do well I wasn't done I was certain there was a way to actually do this to do this effectively without you know I wanted to do this a lot more simply than I was talking about so obviously one of the problems with covert communication over these channels is there are just many different protocols we talked about TCP and UDP but then we've got RTP and we've got Notella we got VoIP we've got RTSP and so on and so forth there's all of these different protocols that are available to communicate streaming information so streaming protocols are ubiquitous and most organizations don't prevent people from being able to use these protocols to stream music over their networks and understandably their employees want to be able to listen to Pandora while they're at work right so the sheer volume of the streaming packets flowing in and out of an organization even if it's allowed even if it's VoIP that's being used can be enormous so how do you actually monitor this other than encryption and how does that affect again due to several factors that I've kind of mentioned the complexity of the encoding method the large number of packets the relatively small size of each packet because you want these packets to be really small at least the content portion so if you do lose some information it won't affect the actual experience of the folks that are listening to the streaming data and obviously today from a leakage perspective if you look at data leak protection technology that's out there it almost exclusively focuses on email and web traffic and those kinds of things it doesn't focus on streaming data that's coming from media so let's talk about how Bob and Alice might pull this off and again why it's complicated so when Alice wants to communicate with Bob and they want to basically carry out this streaming protocol we have these packets that are going to go across and the packets have a header as I mentioned that's pretty standardized it is standardized and then they have the payload so again most of the focus up to this point has been on the payload portion of how to encode information in the payload to be able to communicate it over there's a couple of problems besides the fact that it's complicated to include things within this encoding method it's going to mess up the sound of the media if I actually encode other information within those the payload portion it's going to potentially have some impact on what you hear does that make sense I actually don't really want to do that so basically the way this works in normal situations is each of the packets contain a portion of a message that we want to communicate between Bob and Alice so the question is how do we actually pull this off without getting into all the messy stuff that I talked about and kind of extending onto the core concepts behind what Craig Rowland was talking about within this environment so let's take a look at this in a little bit different way and assuming that I have control of both ends of this interface I could extract a packet out and what I want to be able to do is instead of letting all the packets flow I want to pull one out and just delay it so remember on the other end these are going to come in in a sequence and have to be reassembled remember this is fake technology and all of these protocols it's not like TCP so basically they're going to come in potentially out of order there's going to be packets missing so I'm going to intentionally pull out one of the packets and not have it go through and what we're going to do let me pull this all up so I can see it because it's a little bit out of sequence so the concept is if I extract this packet out of the stream and then I reinsert it later what's going to happen when it gets since it's out of sequence and it's quite delayed what's going to happen with the normal music player on the other end it's going to throw the packet away and it's going to throw the packet away because in the stream of things it's already compensated for the fact that the packet is missing so it's not going to have any impact on what we hear but I can put anything I want now in that payload because I know it's not going to actually play music so I can use that entire payload for whatever content that I want to transfer over that medium so now what I do is I take the message that I want and I start to break it up into packets that I'm delaying and actually sending them over in that way so all I do is end up with packets that are delayed so on the other end all Bob has to do is extract the delayed packets so if Bob is watching the network he has to do is basically say wait but this packet is way delayed I'm going to pull that out and I know it has content in it that was meant for me and it's not going to affect wherever that stream was going to because it's going to have thrown that packet away already and not going to use it in order to play the music or communicate the voice so it will have no impact on what the user was so I can be man in the middle here in order to be able to pull these packets out insert what I want and have them be sent across the wire in this fashion so the advantages of this approach again remember there's a lot of naysayers out saying that we couldn't use this kind of protocol this streaming protocol is in order to convey any meaningful information using stego the advantages are the volume of streaming traffic is enormous most of it is not checked some of the disadvantages of this approach it's a little more complex to pull off right so we had to write some significant python scripts on both end in order to be able to do this but it is easy to do once you actually build that so that normal data loss can corrupt or lose the message content however as we talked about before when you look at things that are in the IP header right they're still going to be there so we can use things like the identification field within the IP header in order to be able to re-sync or provide error correction to the packets that may be lost so even if we do lose packets we can actually recover from that by using other fields within the IP header that are not really used that much or are not necessary in order to carry this out so this is an example where taking Craig's original work and looking at first of all we haven't solved that problem either but taking it to the next level with all of the different protocols and formats and data structures that we have we can actually exploit this today to leak information out and we're not checking it unless you're a very sophisticated organization looking at this you're not going to be detecting this within your environment and I'm going to turn it back over to Mike yeah so that was actually the last slide right okay cool so a few other things I was going to mention last year when we presented we had explored mp3s and the id3 headers and the type of content that could be hidden in that how you could identify that from a forensic perspective and one of the interesting things we actually stumbled upon during that testing was in modifying an mp3 and hiding some content in it started to upload it to various music services and one of those was a streaming music service a very popular one a very popular one yes and in doing so had accidentally discovered that if you had say like a musician type of account that you can anybody I believe can actually create you can upload your music to create a band or a channel right that people could bring up on their streaming music player or app for that matter and listen to it playing back and what was kind of interesting was in doing this found that you could upload that music and it would be playing it would be streaming really for anybody just like in a posting an image to social media or as Chet mentioned over the network but only the intended recipient would know it was out there so bottom line if you were the intended recipient you would go to that band you would listen to the streaming music and within Chrome on a PC you can actually download that mp3 and that mp3 would contain that covert message and we actually tested that and actually worked so just something else to think about too so in terms of like additional research I'm doing more research on some of the other protocols aside from SSDP UPMP to see what other fields can be used for hiding content and communicating that and I don't know Chet I know you're doing additional research I don't know if you wanted to comment on anything else Mike and I presented this morning in SkyTalks and we talked briefly about some of the other fields within these protocols that are vulnerable we talked about time to live we talked about some other fields within those protocols but right now what we're doing is looking at a variety of protocols that are available with IOT devices looking at how those could be exploited and those special protocols that are being developed there in order to be able to do that our real reason for presenting this however is for us to really think about how we actually do this better right we tend to take this for granted and we don't consider the data hiding elements of whether it's a protocol which we're talking about today whether it's a data structure for a particular file format for example for things on the multimedia side that we discovered was the ability to hide information within an mp4 file using a program called tcsteg and that actually includes a truecrypt volume right within the movie so now I can actually play the movie it plays perfectly but if I bring that up in truecrypt I can actually extract the hidden content and actually mount it as a volume so all of these data structures have these traditional vulnerabilities built into them because I guess just nobody cares right but so I'm trying to kind of encourage you guys to care so in the graduate programs that I teach in we kind of are looking at ways that we could actually secure these protocols and secure these file structures such that they can't be manipulated as Mike mentioned when we talked about the id3 header of mp3 many of you may know but the id3 header has images associated with it whoever built this structure for the id3 were certainly audio files that were interested in including all kinds of information in the id3 header of an mp3 which is great it's artist information it's information about the publisher it's information about the tracks and how they're put together in images of the band and the singers and all of that kind of information is part of the header but that means that there are images contained within the mp3 right so you've seen this in your car right you play a song within the car and the image of the artist pops up on your dash screen you can put as many of those as you want in the id3 header there's no limit to that so now I can actually put 20 or 30 images in there I can put the lyrics of the song I can synchronize the lyrics in there but I can also hide whatever I want in those images that are there so again the general concept that we're talking about is the fact that these protocols and these data structures have no security in them to prevent the hiding of information so that's kind of what we're trying to encourage you to think about and I guess with that we'd love to hear questions, comments and I guess you should come up to the mic if you're going to do that right? I mean, yeah let's see if I can paraphrase it correctly like other than a Russian spy what would be the what like a practical use case for this either offensively or defensively yeah so there's a variety of use cases we actually covered this just an hour ago we did a presentation you were there on doing this on social media there's a lot of different use cases so aside from a Russian spy whether it be a dead drop or covert communications it could be used for data exfiltration within a corporation or an enterprise or an organization and in addition to that we've seen with things like hammer toss and others where it can be an operation shady rat for that matter where it can be used for pointing to a specific URL where an image resides and within that image is hidden like updated CNC information such as a different URL address to allow the malware to update itself by occasionally going to what appears to be a very benign location like WordPress downloading an image that is just of a cat and again looking very benign but again contained within all of this infrastructure for updating the CNC and leveraging that to further like exfiltrate information the flip side of this also is while we look for areas in which we can hide this information in a lot of metadata fields as Chet alluded to with MP3s and the ID3 headers we explored that last year and there was a plethora of different ways in which you could hide information and then furthermore in the talk we just did hey I've got an MP3 I'm going to post that to social media and leave that as a dead drop for somebody else and rather than actually change anything for the audio or the MP3 itself but rather the JPEG within the MP3 that displays on your MP3 player hide the data there instead or append data to the end of the file a plethora of different use cases are other groups, agencies, good or bad using this for covert communications I guess a lot of that is left to spy movies but there is certainly a variety of use cases I don't know did I miss anything Chet you wanted to add? So currently while there are no security around the abuse of these protocols can an organization use its firewall to monitor the egress of these protocols and drop these packets? Yeah I'll let Chet take that one first Yeah there's two things that you can do one is you can jam the protocols so I talked earlier today at the SkyTalks talking about embedding information in the time to live field the upper bits that are never used so if you're there if you have a way to store in forward packets you can basically jam those channels by basically overwriting the data if information is leaking out as part of MP3's I can re-encode it if it's a JPEG I can re-encode the JPEG I mean obviously this is what Facebook and those do when you post an image to Facebook it eliminates the geotag and the header information of the JPEG but it also actually re-encodes the image so if there was something hidden in it it's supposed to wipe it out now those algorithms are not perfect and we figured out ways around that but nonetheless at least it's an attempt so one of the most effective ways is to jam those protocols if you think that's happening the other thing to do is to be able to monitor those behaviors we talked about the initial sequence numbers we talked about some of those aspects and you can if you're looking for that carefully you can actually have a sensor that would look at that specific information and then be able to detect anomalies within the behavior of those protocols that we would not normally see for example information hidden in the identification field in initial sequence numbers that don't appear to be random so those kinds of things we can detect but the question is is anybody looking from that perspective yeah the only thing I would add and I think Chuck kind of hit on this is you know in modeling out the network and knowing what's normal leveraging that to determine what's anomalistic that's going on right but that's very difficult to do but attainable in today's world right so so we've spent a lot of time talking about how this this has been an inherent 20 year probably 30 year problem we've been doing it a long time we're all so do we fix the problem or do we continue to let it go and if we fix it is it at the vendor level is it at the protocol federal regulation level yeah so I think there's a number of different viewpoints I'll hit on one or two and then I know Chet's very passionate about this one too what we just presented a little while ago in track four here at DEF CON was leveraging social media for COVID communications but further automating that or testing it in an ethical way but from an offensive standpoint and automating it using machine learning the intent behind that was to say while we run around trying to find ways in which we can detect this stuff with the technology we have today we can emulate it from an offensive standpoint and leverage ML and neural nets to actually discover a whole bunch of new ways that humans haven't figured out but the machine can and the result being that here are another 2,000 methods that to our knowledge nobody has used but are ways in which the data could be hidden in what we demonstrated across social media and then leverage that intelligence for better defense the only thing that I would add is that we're trying to inspire encourage the standards bodies to really take a look at this so we keep trying to bring this awareness of this problem and a lot of the big organizations are actually looking at this McAfee actually just released a bunch of information about the concern for secondography which we've been talking to them about for 10 years and now they're actually seeing this as a real problem so once the standards bodies realize that the protocols I mean the data structures that are coming out that our people are using are vulnerable to this hopefully they'll actually start to take steps that will actually redefine those protocols in a way that we can eliminate this I mean part of it starts at even Mike and I both being authors I mean it's like DRM doesn't exist it's just worthless right correct and they're abiding by standards that have these fundamental flaws in them so they can hide behind that right exactly that's okay that was rhetorical yeah any other questions yes yeah no that's a great question and it kind of goes to a talk that we did a couple years ago about actually being able to spoof the MAC address and changing those so obviously it's going to affect the ARP table but at the time that the SIN comes in right it's just going to be responded to and acknowledged unless that particular IP address has been blocked and if you're not abusive if I only use that server one time to basically spoof the IP address it's not going to look like a denial of service attack because I've only sent one packet right to that particular server so the only way to do this is to distribute the effect over thousands or millions of devices that I want to bounce packet off of in Craig's original this is not my design this was Craig's um original article that still stands today I can use that to basically hide the identity of the sender and the receiver by using that method you know judiciously it's like denial of service right one of the things that you want to do with denial of service I want to have millions of IP addresses that have compromised right and have packets sent from all of those IP addresses that I've compromised not from one location right so even if I shut those off I'm still going to have millions that I have to deal with and we obviously deal with that a different way from a denial of service perspective we handle that differently today but nonetheless the issue is exactly what you said it's like nobody's going to basically defend against this because it's a non-issue to that one server getting that one packet from that client that never gets responded to right right right after you've done that exactly right correct correct that's the defense right those firewalls at the egress points will actually rewrite those if they do the same thing with time to live whatever if they see that wrong they'll actually fix it right but they're not implemented very often and so that's our wrap for Chetan Chetan Mike Chetan Mike welcome back always a pleasure