So our next talk is by Dr. Joey Shaw. He is an associate professor at the University of Tennessee at Chattanooga in teaching botany classes. He's a research associate at Britt, the Botanical Research Institute of Texas, and also the vice president of the Association of Southeastern Biologists. And I know that he is also me deep in organizing the wildflower pilgrimage at the Great Smoky Mountains right now. So we really appreciate him taking the time to update us on the Kentucky Tennessee Plant Atlas project. So, Joey. Hi everybody. Can you hear me okay and see my presentation? It's up? Yep. Good, good. Okay. Yeah, so the Tennessee Kentucky Plant Atlas. I guess before I get started, I don't know if I'm visible in any of your little windows there, and you might be wondering, you know, why somebody from way down in Tennessee, about as far away as you can be in Chattanooga, you know, given this talk to this group, I just want to show that I have credentials here. All right. So thank you, Heidi. I have my mug from a couple of years ago. Anyway, let's get started. So before I begin, I want to stress that herbarium specimens make atlases scientific, right? Because they provide records that can be verified. Obviously pictures or expert claims are great, but they're not scientific in terms of being able to verify these things or make changes to earlier observational data. And so really, herbarium specimens are foundational to our knowledge of plant species distributions. So I'm going to begin with a brief history of the Tennessee Plant Atlas, and then I'll move into Kentucky and combine those two things later on. So the Tennessee Plant Atlas was really first initiated in the 1950s by doctors Sharpe and Shanks at the University of Tennessee at Knoxville. And these workers combed through the University of Tennessee at Knoxville Herbarium, and they created data sheets on the county distribution of each of Tennessee's known plant species at the time. In the 1970s, the effort was joined by Dr. Chester from Austin Peay and Dr. Crawl at Vanderbilt, and also Dr. Wofford, who would eventually become the curator of the Knoxville Herbarium. The first editions of the Tennessee Plant Atlas were published about 30 years later, in 1993 for turretophytes through monocots and then 1997 for dicots. In the late 1990s, though, the UT Herbarium Tennessee Plant Atlas was really one of the first in the US to go online. And those of you who are, you know, nearly as old as I am, remember that this was really an incredible resource for almost all of the Southern Appalachians, because it was the only thing really available. Dr. Wofford and Victor Ma, who at the time was sort of a database manager in the Herbarium, they were the primary collaborators. And really what they did was they began by taking the dot maps from the spiral bound atlases and converting them to image files. And so these are just simply image files with counties colored orange and each that image file showing county distribution for a particular species was then hosted on a species page of the Atlas. So the Atlas grew through the early 2000s to eventually contain somewhere between zero and three or maybe four habit pictures of most species. And today it hosts around 11,000 images of Tennessee's vascular plant species. Now, remember that number 11,000 just sort of store that in your head for a little while. It turns out that that's about three images per species, including the image showing the species distribution. Now, four things I think are of note about the UT Herbarium Atlas. And one is that it's based on, you know, currently it's based on 25-year-old technology. And that is as Dr. Wofford learns of a new county record, he may relay that to Victor or now Margaret Oliver, who's in Victor's old position. And they'll simply modify the JPEG, JPEG dot map and re-upload it to the website to sort of re-show the new distribution. The second thing is that in the the 90s, late 90s, about half of the Memphis Herbarium, about 20,000 specimens were transferred to Knoxville. And that really kind of increased the, you know, the Knoxville Herbarium Plant Atlas to include some species from West Tennessee that it really was underrepresented by. And then the third thing I think is perhaps the most important, and it has to do with the breadth of the collaboration narrowing as soon as really, really as soon as the Atlas went online. With Dr. Kroll's retirement, the Vanderbilt University Herbarium was moved to the Botanical Research Institute of Texas. And Tennessee botanists no longer had access to all of those specimens. So as species circumscriptions changed through scientific processes, the 40,000 or so Vanderbilt specimens from Tennessee species couldn't be referenced. And so what does that mean? Well, it means that the UTK Atlas information from Vanderbilt specimens really hasn't been updated since the early 1990s. The other thing I wanted to note is that specimens at Austin P were also no longer represented on the Atlas unless they were significant. And so early on in the early construction of the Atlas, Austin P specimens were included. But about this time in the late 90s, that collaboration sort of stopped. And the fourth thing is that no system was established to include data from other Tennessee Herbarium or to continually add new data to the Tennessee Plant Atlas unless they were new specimen data coming into Knoxville itself. So in essence, the Tennessee Herbarium Atlas became really kind of became a UTK only Atlas after about the year 2000. It ceased being collaborative. It was nearly only reflective of the holdings at Knoxville, plus historical references with old circumscriptions from Vanderbilt, Austin P, and Memphis. And that's reflected, that's also reflected on the website in terms of design and credentials, right? That is the other Herbarium that contributed early on are not really listed. And the colors being orange and white really sort of make you assume that it's UTK only. And so it sort of leads out a lot of these historic and even contemporary contributors. Well, what I think is really important is that the UT Herbarium is really only one of 13 Herbaria in Tennessee. And it only holds about 34% of Tennessee's total plant data that are supported by Herbarium specimens. So if you think about it, this incredibly, highly used important resource for Tennessee, and especially through the early 2000s for a lot of the Southern Appalachians, it was really only built off of a small percentage of our potential knowledge of the state. So if we think about a history of Tennessee's Herbaria, the history is not nearly as complicated as you'll see with Kentucky. But we had 13 Herbaria in the state. One Vanderbilt moved to Texas in the late 90s, and half of Memphis was transferred to Knoxville in the late 90s. The history of the Kentucky plant atlas is maybe even more complicated. And like Tennessee, it sort of suffered from a lack of statewide collaboration, kind of historically. One difference I think I see in thinking about this stuff through the years is that in Tennessee, most workers simply deferred to Knoxville and understood or maybe didn't, that the data were largely based on specimens at Knoxville. However, in Kentucky, other workers didn't really defer to the flagship. And there were competing projects and sort of different information and even sort of information quality in different atlases and different works. The Campbell and Medley Atlas is probably the most used. And to my knowledge, it never went through peer review. Today, it's present in two forms. You can find it online in a PDF form, as you see here in the left, or in a searchable Esri format created by Judy Redden, and that's on the right. And so the platform created by Judy is pretty easy to navigate, but it's also fairly simple. If we think about the history of Kentucky herbaria and summarize the herbaria in Kentucky, there were nine herbaria, and you can see these listed in the table there with the numbers of specimens that they hold. Today, however, there are five. And really what happened was Eastern Kentucky, oh, there we go. I'm sorry, Berea College moved to Eastern Kentucky. And you can see some arrows in there to sort of show these movements, right? So Berea moved to Eastern Kentucky. Center College vascular plants also moved to Eastern Kentucky herbarium. The University of Louisville moved to Western Kentucky, and then both of those together moved to Austin Pee just a couple of years ago. And so today there are five herbaria, but there's been a lot of moving around and including moving a lot of specimens across state lines. So I think Kentucky and Tennessee both sort of suffered the loss of specimens from the state. However, Kentucky specimens really are just sort of right across the state line, whereas unfortunately, the Vanderbilt collection is now 1,100 miles away in Texas. But it was really, I think it's 2014 that is sort of the start of a new age. And that is because it marks our landing of this National Science Foundation funding that funded this CERNEC. And CERNEC, the award was called the Key to the Cabinet's Building and Sustaining a Research Database for Global Biodiversity Hotspot. And the primary goal of this massive collaborative was to digitize about 4.7 million southeastern held herbarium specimens from 130 herbaria. Now Brad Rufel was the state lead for Kentucky and I was the state lead for Tennessee. And we did just that. We digitized from 2014 to 2019. We digitized nearly all of the herbarium specimens in our respective states with the help of a lot of students and a lot of colleagues in the state. Now, as we saw the end of that digitization effort, I in 2016 and Brad in 2017, Brad and Tara, sought funding to build new atlases in our respective states based off of, and here's an important point, based off of all of the herbarium data in our states and using this new technology and crediting all contributors. And so the Tennessee-Kentucky plant atlas includes specimen data from all Tennessee and Kentucky herbaria regardless of where they are currently housed. And a combined atlas for our two states really makes a lot of sense for a couple reasons. And I'll show you a primary reason here on this table. The first thing I want to do is orient you to this table. And so here I'm showing you the 13 herbaria of Tennessee and the five herbaria of Kentucky. This column represents the total number of vascular plant specimens held by each of these herbaria. And both states combined, it's nearly a million herbarium specimens. That's a million data points, right? In the next two columns, what I'm showing you are, these are the number of Tennessee-collected specimens, or right, the number of Tennessee-collected specimens that are held in either Tennessee herbaria or Kentucky herbaria. And this column represents the number of Kentucky-collected herbarium specimens held in either Tennessee or Kentucky herbaria. So if we're thinking about building a Tennessee-Kentucky plant atlas, we really don't care about total vascular plant specimens. That is all of these plant specimens that are coming from other states, right? I mean, they're incredibly important, just not for this project. So the first thing I want to show you is if we add up all of the Tennessee and Kentucky herbarium specimens that are housed in all of the herbaria in our two states, it's over a half a million specimens, 546,000 specimens that are Tennessee or Kentucky-collected. Tennessee herbaria hold 337,000 specimens that were collected within the state of Tennessee. And they also hold 78,000 specimens collected within the state of Kentucky. So in other words, for our combined atlas, Tennessee herbaria hold 416,000 specimens collected in either Tennessee or Kentucky. Kentucky herbaria hold 6,066,000 specimens collected in Tennessee, and 123,124,000 specimens collected in Kentucky. So we combine those two in Kentucky herbaria hold about 130,000 specimens that are collected in our two states. So Tennessee herbaria hold about 76% of the data, and Kentucky herbaria hold about 24% of the data. There are 344,000 Tennessee-collected specimens, and Kentucky herbaria hold 6,600 of those, or about 1.9% of the data supporting sort of the Tennessee side of the atlas. On the flip side of that, though, there are 202,000 Kentucky-collected specimens, and Tennessee herbaria hold 78,000 specimens, or about 40% of the data for the Kentucky side of the atlas. So Tennessee data, or I'm sorry, Tennessee herbaria hold nearly 40% of Kentucky plant atlas data, and sort of adds significantly to the combined plant atlas. A second reason that a combined plant atlas makes sense is that there are so many shared eco-regions between our two states, and I was actually going to add a third slide I'm sort of wishing I did at this point. I honestly I didn't do it because I didn't feel like I had time to look up pictures, but I also I kind of wanted to put the personal relationships. There are so many, you know, personal relationships between Tennessee and Kentucky botanists. So let's get to the Tennessee Kentucky plant atlas. We actually launched this in July 2020, and here's the landing page or the homepage of the atlas, and the atlas covers nearly 4,000 species and lesser taxa, and nomenclature largely follows weekly. There are over 401,000 digital herbarium specimens that support the species distributions shown on this atlas, and we also have over 20,000 live plant images like habit shots, and that means that currently the Tennessee Kentucky plant atlas houses about 420,000 images of plants and live habits or herbarium specimen images for users to to get at, and that's incredible when you think about and compare it to the 11,000 images of the UT herbarium atlas and how heavily used that thing was for so many years. But the Tennessee Kentucky plant atlas is much more advanced than anything else in Tennessee or Kentucky. So I want to just orient you to this if you haven't used it. There's a quick species search window where you can just sort of type in a name and hit go, but down here there's a quick or advanced search, and the advanced search is what really what I want to talk about, and I'll get to that in just a minute. This window here will allow you to browse that the plant atlas by county so you can just sort of hover over all of these counties maybe you hit McCreary County here and you click on it and it'll generate a list of plant species names known from McCreary County. This third window here is really pretty cool because it allows us to search or browse any Tennessee or Kentucky herbarium itself. So you you can select a herbarium that you want to search leaving out all of the other herbaria. So if we do a quick search for adiantum pedatum for example we could type that in here or we could type it in here and hit search and it'll take you to the species page with an opening table of information on that species. Over here you could see is a window for photos and a button to click and you can see more photos and it'll take you to lots of photos of those species many of which people here in this meeting today have taken. I've tried to focus my pictures on important characteristics that also align to keys to the species. Scrolling down a little bit further on on this adiantum pedatum species page will take you to the Atlas map and we have over the the Atlas map we have these level three and level four ecoregions. So you can see the distribution of the species in our two states. If we scroll down a little bit further there are actual links to the herbarium specimens themselves on CERNEC that support each of the the counties being turned on. If we look at the herbarium search feature we search within it so you'll click this button then the next page will open up. You could do a search for example to look at the Austin P. herbarium for specimens that were collected in McCreary County Kentucky and you hit search there and it'll then show the specimens that are held at Austin P for that state and county. But it's this advanced search feature that I think is perhaps the most novel and it's what I've spent the most time working on. Before we get to that I just want to let you know the bulk of my work on the Atlas it goes into database building. So if you think about it really what I have is a massive spreadsheet of nearly 4,000 species times 150 different morphological and ecological and ecological characters. So I've got an Excel spreadsheet of about 600,000 cells of data and you can see there are a lot of gaps in here. I've got a lot of work to do. It's about halfway complete at this time but it's still I think complete enough for us to sort of start using to some degree. So some of the characteristics that I have in the advanced search feature are characteristics that are related to conservation for example. Is it native? What's the known C value? Does it have a state status? There are ecological characters like what is the sexual phenology and you might click on that and say I'm interested in things that flower in April. What is the elevation? What are the moisture requirements? What habitat, soil and geology, um region of origin and so on. There are general plant characteristics. Is it woody or herbaceous? What's the growth habit? What's the duration evergreen or deciduous? Does it have milky sap? Is it armed? Are stipules present or absent? What are the leaf characteristics? How dissected are the leaves? If it's a compound leaf what are the number of leaflets and so on? So there are all these leaf characteristics. There are reproductive characteristics, pharyngeum arrangement and ferns and allies, inflorescence types, whether hypanthea or present or absent or sepals or petals fused. What's the ovary position and so on? And so and then the last thing you can do is you can filter by county. So if you really wanted to do a big complex search you could look for something for example if you're pretty sure it's native, it's flowering in June, it's likely from North America, it's an herb, it's photosynthetic, it's got dehiscent fruits, it's deciduous with white as the base flower color and it's known from Marion County, Tennessee. You can select all of those characteristics and hit search and it'll it'll give you a list of all of the species that fit that profile. So then you might think well maybe this is premula media, maybe that's what I'm looking for and you click that and it'll take you to the species page to show you the distribution or the atlas page for premula media along with a lot of photos, right? Habit shots and close-ups. I'm adding more and more every single year, I'm adding more and more sort of dissections to these things and so on. And so from the time we launched the atlas back in June 2020, this is the Google Analytics for the atlas. You can see we started using it and then we updated it and added this advanced search feature and it's being used more and more. And so with that I'll finish by thanking both the Tennessee Native Plant Society, the Kentucky Native Plant Society and the Office of Kentucky Nature Preserves for funding to get us going and this is, I'm not fooling myself by and I fully understand this is sort of a lifetime kind of commitment to build this thing and to keep it going and I'm happy to do that if it stays useful to folks. With that I'm happy to answer questions if there's time. Thank you Joey. We do have one question from Rachel, how can the public contribute to this atlas? So that's a great question and you know I've gotten a lot of emails over the last couple of years people saying, hey I've got Greenbrier in my backyard but you don't have the county lit up for Greenbrier. One of the things that we're relegated to for now is to making sure that there are herbarium specimens that support us turning on these counties. So herbarium specimens make the data verifiable and so usually my response email is you know if you want to send the herbarium specimen to me or if you want to collect a little bit of plant material and tell me where you got it and the date you collected it on you can put it in a ziplock bag and send it to me. If you want it to go in a Kentucky herbarium rather than sending it down here to Chattanooga that's okay too. Send it to the herbarium of your choice and the curators there will turn your collected material into a specimen and eventually will get the data up. I can bribe you folks a little bit by saying send it to me and I'll the turnaround will be a little bit faster because I'll be I kind of control the thing and I can push it right up there. Awesome thanks Joey. I also just kind of wanted to mention I know that this goes a little bit beyond the scope of the Kentucky Tennessee plant atlas but I know that our working group is looking at integrating other observational data through iNaturalist in some form in the future so if you're not a plant collector or herbarium person and you just want to put a photo on iNat that data can be useful to populate plant distributions as well. Absolutely. Thanks.


Police in Amos State on Sunday foiled an attack by armed men suspected to be members of IPOP. In a statement, police spokesperson Michael Abatam said the militia wing of the IPOP came in their numbers from the rear of the Oboa Division, which is an ungoverned forest. The three petrol bombs and IED that fell on the transit camp and the vehicle shooting sporadically. Abatam further narrated that the police operatives of the division and some of the command's tactical teams responded swiftly.


All right, hello, everyone. Welcome to our talk on Fernet tokens. Right now, the only thing standing between you and lunch today, so it should be a good talk. We want to do this talk because Fernet tokens have been around for a while, and people have been successfully ignoring them, especially operators. But we're here to tell you the day of Fernet tokens is coming. It's going to become default. And so when you watch this talk today, you need to plan for how you're going to switch to them or how new clouds you build are going to use them. Just a brief intro and what we're going to cover today, starting with the basics. What is a token? Then we'll dive into why was Fernet created, why do we need to do format. We'll cover the importance of non-persistence in your Keystone tokens. Then we'll cover rotation, key rotation in two scenarios, just on a single system and also in cluster. And then we'll dive into some operational bits about Fernet tokens, starting with just a brief overview of OpenSec at Time Warner Cable, the prerequisites we needed when we switched to Fernet tokens, the migration path to Fernet tokens, and then just some things we picked up while operating for Net Tokens. With me here today, who might be able to introduce earlier, is Lance Bragstad from Rackspace and Dolph Matthews from Rackspace. And Dolph's going to start by diving into some details on Fernet tokens. Awesome. So what does a token start there? So if you interact with OpenStack in any means whatsoever, if you use Horizon, if you use OpenStack Client, you use Keystone Client, Nova Client, anything, you're always going to start by authenticating with Keystone. Keystone's going to give you back a token. And then the client or yourself or Curl or whatever, every interaction with OpenStack beyond that point is going to involve that token. And it kind of looks like this. This would be an example request to Nova. So why do we need another token format? Let's start with some history. We can go back through kind of how we got to Fernet. And hopefully that'll help explain. So we started with UUID Tokens way back in Diablo. UUID Tokens are basically just big random numbers. We represent them as strings. And Keystone writes them to a token table generally. They have to be persisted somewhere. Basically the way they work is when you hit Nova with your request, Nova calls back to Keystone over HTTP and says, hey, Keystone, is this a valid token or not? Keystone goes, looks up in the database, is this a valid token? And if so, Keystone remembers the identity and authorization context that belonged to that token, returns that to Nova, and your call proceeds. So PKI Tokens. We perceived that online validation process had to not be scalable. And so we implemented PKI Tokens. PKI Tokens are gigantic signed responses that Keystone would otherwise be returning to Nova. But instead, we are basically signing it using a PKI cert, base 64 encoding the whole thing, and passing them around over HTTP and headers. That turned out to be a problem. This is about as small as PKI Tokens get. This one has a very minimal service catalog in it. And we found that as soon as you start adding a lot of services to your OpenSec deployment, these get too large for HTTP implementations. So you start having to recompile Apache and stuff as these blow past 8K. They get really, really big very, very quickly. So we tried to address that. We implemented PKI Z Tokens. As you can tell, this one's way smaller. So they're almost exactly the same as PKI Tokens, except we introduced a layer of compression in the pipeline. And so they actually are a little bit smaller if you have a very large catalog. Catalogs are pretty compressible, and so it kind of works out. But these are still far too big to be practical. You can't copy and paste them around in your CLI because it'll flip past your screen, things like that. And so that brings us to Frenet. We knew we wanted something small. PKI was simply too big. But we're also pursuing non-persistence. Non-persistence is a super valuable feature of Frenet that I'll get into in a second. We also had another problem with PKI that's just kind of a hurdle to making it the default token format in Keystone. And it requires a little bit of setup out of the box. And really, the setup that Keystone offers is not really production grade. And so we kind of fixed that with Frenet. Frenet does require some setup, which we will also get into. But you can use it in production. It's ready to go. And like UUID tokens and unlike PKI tokens, Frenet tokens must be validated online. So we kind of gave up on that front. This is a Frenet token. It's also base 64 encoded. But unlike PKI tokens, there's not a gigantic JSON blob in there. So we'll dive into the details of that in a minute. So what does non-persistence matter? Again, this is really important. So if you've operated Keystone at scale before, or even small DevStack cluster for a long period of time, you've probably learned very quickly that your token table gets very, very large. And Keystone grows slower and slower and slower. And so for a long time in Keystone's history, it was just sort of like a documented thing that, hey, you need to go purge your token table periodically. Eventually, we implemented a command called token flush to actually help you do that. And you can run it on a cron job or whatever. So if you have 24 hours worth of tokens in your token table, it's still big, still kind of slow. And if you want a longer expiration window than your token table, it just gets bigger and bigger and bigger. So if you're actually using Frenet, then your token table is empty. And we're not just hiding Frenet tokens in memcache or persisting them somewhere else. They actually do not need to be persisted at all. So when you authenticate with Keystone, there's actually no write operations anywhere. We just generate a token and send it back to the user. And so the impact of that, when you authenticate, again, there is no write to a back end. So we can authenticate very quickly.  As a result of not doing all those writes, and if you're using UID or PKI, most of the write operations to your database are going to be token creates. And so if you're doing large multi-region deployments where you want all of your tokens to work on all of your other data centers, that's a lot of data to be shuffling around with Glare or whatever. So also because we're not persisting them, there's actually no replication lag. So one of the use cases we're trying to tackle was we have data centers in, say, Sydney and Washington DC. And we actually have race conditions where you can authenticate and say Sydney and get a token back and immediately try to go use it in Washington DC. We're actually like racing our replication process, and you might beat it. In which case you get a four one, and you're like, well, I just authenticated. Wait a couple seconds or a minute or whatever till that replication catches up and then your token works. And that's not a great user experience. Again, because your token database is empty, you don't have to run that cron job anymore, or you can, but it's essentially a no up. Fernet has some downsides though, and that's that token validation is suddenly kind of difficult. We have to do a lot more work to validate Fernet tokens. With PKI tokens, all we had to do was check the signature, decode it, return it, you're done. With EUID tokens, all we had to do was a single read from the database, good to go, return the response. With Fernet, we have to do a whole bunch of reads and rebuild that entire authentication and authorization context on every token validation. And token revocation is also hard. With EUID tokens, we can just delete them from the database and your tokens revoked, there's no record of it anymore. With PKI, that was a little bit more of a challenge, but it's essentially the same process. We deleted or market is deleted. Your entire cluster gets made aware of that and kind of you're done. With Fernet, we had a long process of implementing token revocation events, and we won't get into that today, I don't think, but it's a more interesting problem than we've ever had before. So what's inside of Fernet token? It's kind of one of the first questions we get asked when we talk about, you know, they're not persistent. Here's all the advantages they have. So what makes them tick? So again, every token represents a user's identity and a scope of authorization. And so that generally comes in the form of a user ID and a project ID. We tack on an expiration and we message back the whole thing together for Fernet. Message pack is a lot like JSON, but it's basically a binary format. So we call that our payload, and then we actually AES encrypt the payload, append another timestamp, the Fernet version, take an HMAC of the whole thing to sign it, and that's our decoded token. We just base 64 and code it and we're done. That's our final token. So if you actually want to dive more into the details of that process, we didn't actually write the spec for this. This is actually a spec originally implemented and proposed by Heroku, and we just actually took advantage of a library and defined our own payloads for it and kind of wedged it into OpenStack, made it work. But the HMAC and the AES bits of that are interesting because they both require keys. So what is a Fernet key? Fernet keys are a combination of a 128-bit key that we use for SHA-256 HMACs and a 128-bit key that we use for a symmetric key for AES encryption, and so that string on the bottom is actually a Fernet key, and they're actually just base 64 and code it as well. So we store those in files on disk, and because we're actually using the Fernet keys to generate ciphertext that we're then passing back to users constantly and you can request as many tokens as you want, they're eventually vulnerable to being broken, so you have to rotate your keys periodically, and I will let Lance explain how key rotation works. So now that we understand a little bit more about what's inside of a Fernet token, let's talk about key rotation, we're gonna break it into three different parts. First, we're gonna talk about how to configure keystones so that you can issue your Fernet tokens, but also make sure that our config allows us to do successful key rotation. Second, we're gonna go through some commonly asked questions that we get around key rotation, and then third, we're gonna go through a few different examples. So starting with config, the only thing that you really need to set in order to issue Fernet tokens is your token provider, which is pretty straightforward, but there's two other things that you need to know and you need to be aware of before you start issuing Fernet tokens, and that's your key repository and your max active keys count. So your key repository is just a location on disk, but it changes depending on where you deploy keystone and how you deploy it. Your max active keys count is the total number of keys that you can have in your repository at a given time, and this is gonna be very dependent on your token expiration, as well as how often you plan on rotating your keys, which brings us to our next biggest question, and that's how often should we be rotating our keys? This is gonna be dependent on your deployment security model, but you can start by asking yourself a question, that's how fast would you expect somebody to crack a 128-bit AES encryption key, as well as a 128-bit SHA-256 HMAC signing key? Your goal as an operator is to make sure that you rotate more frequently than whatever that is. So once we know what that timeframe looks like for us, the next thing we need to know is our token lifespan, and then using those two things, we can actually step through and figure out the number of keys that we need. So these are the three things that you need to know before you can actually start implementing a key rotation strategy. So here's an example. Let's say we want to do rotations hourly, and we're going to have a token expiration time of one day. We need to have a max active key count of 25 keys. The reason why is if we go below 25, we're gonna be removing keys that have been used to encrypt tokens that are still valid, which is poor user experience. You can set that value higher than 25, but you're just gonna have unused keys in your repository, which is fine, it's more or less a cleanup operation later. Here's another example using token expiration times of one week, and doing daily rotations, you're gonna have to have a max active key count of at least eight. And another example using token expiration times of 90 days, and doing monthly rotation, you're gonna have to have a max active key count of at least five. So some other questions that we get around key rotation. How many max active keys should you have? We set the default to three, and we suggest three or more, because if you go below three, you're prone to removing a key that's been used to encrypt a token that's still valid. Another thing that we get all the time is when I deploy a cluster of keystone nodes do I have to initiate that rotation simultaneously across my cluster? And the way that the key repository is set up and how the keys work together, it allows you to do a rotation on a single keystone node, inspect that everything works, make sure that tokens issued from that node are still validable and usable across the rest of your cluster, before syncing that repository out to the other nodes in the deployment. So as an operator, that's kind of nice, right? Because you can do your rotation, make sure everything still works, and then commit that across the rest of the nodes in your deployment, which kind of brings us to the third question, and that is, why not just put the keys in the database? Because then technically, if you have a replication solution already set up, you can leverage that to do your cluster sync, right? But then we'll start looking at the problem and we'd be putting keys in that are not encrypted, so we have to encrypt the keys that we use for encryption and so on and so forth. So keystone provides a couple of different tools to help you manage your key repository, and these are packaged with our keystone managed command. The first one is Frenet setup, and this is going to bootstrap your key repository location that you specified in your config with a couple of different keys so that you can actually start using Frenet tokens.  The second is Frenet rotate. Frenet rotate is the actual thing that goes in and will give you a new key to start encrypting with. This is something that you should be doing consistently over the life of your deployment, and we have to keep in mind that Frenet kind of gets its strength from your rotation policy, so the more often you rotate or having like a dedicated rotation schedule is very important. So let's start looking at an example, and this is going to be examining a key repository on a single keystone node. So at this point, we're going to be using a max active key count of four, and we have initiated our Frenet setup command. So when we have done this, we're going to be left with two keys in our repository, and please note the colors on the slides, this is actually going to denote the actual contents of the key, and then the state or the type of the key is to the right of that, and every key in your repository can be one of three different types. There's a staged key, and the staged key is always going to be named zero, it's always going to be the next primary key in the repository, and there's only ever one of them. The other kind of key you can have is a primary key. Primary keys are used for token generation as well as token validation, so they can be used to encrypt and decrypt. They're also the key with the highest index. There's only ever one in the repository at a given time. The third type of key that you're not seeing yet is a secondary key. Secondary keys are used to only decrypt. They've been the primary for some given amount of time previously, and there can be multiple of them in the repository at once. So at this point, if we're to ask Keystone for a Frenet token, we'd be getting back one that's been encrypted with the green key. So we're going to move on to our rotation. We're going to take our maroon key, we're going to promote it to be the new primary key, and then we're going to introduce a new staged key. So at this point, if we were to take the token that we had gotten on the previous slide, pass it back to Keystone and ask Keystone to validate it, Keystone still has access to the key that was used to encrypt it, so that validation can still happen. But if we were to re-authenticate against Keystone, we would be getting back a token that's been encrypted with the maroon key, which is now the new primary. So this same cycle goes through on our next rotation as well, and we're going to take our yellow key, promote it to be primary. We've demoted our maroon key, and we introduce a new staged key. But at this point, we're also at our max active key count. So on our next rotation, we're actually going to do another step, and that's going to be removing the oldest secondary key from the repository, which is going to look like that. So if we were to take the token that we had gotten on the very first slide of this example and pass it back to Keystone, Keystone cannot validate it because it no longer has the key that was used to encrypt it. So this is where setting your max active key value is very important with your rotation schedule because you have the ability to remove keys that could be encrypting tokens that are still valid. And then your max active key count keeps those keys in check as you keep doing subsequent rotations. So that's what rotation looks like on a single Keystone node if we were just to inspect a single repository. But what happens when you need to do rotations across the cluster, right? You have a pile of Keystone nodes and they all need to be able to validate each other's tokens. So we were solving for two different things in this case. The first was that as a client, I want to be able to validate my token against two different Keystone nodes without any sort of replication race. The second was that rotations are something that should happen frequently. So they shouldn't be causing any sort of downtime. For this example, we're going to use the same kind of color scheme that we did previously. And then we're also gonna change our max active keys value to three. But we're also gonna be working with three different nodes. The nodes on the right side of the slide actually depict the actual step that we're in. The nodes on the bottom of the slide highlight some facts about the repositories as we go through each rotation. So at this point, we're not ready to start issuing for net tokens yet because nodes two and node three don't actually have a repository to start encrypting keys with. So we've initiated our for net setup on node one and our next step is going to be syncing that rotation to the other nodes in the deployment. So we go ahead and push that to node two and to node three. And at this point, everybody has the same primary and secondary keys. So any token that you get can be validated across any other node in the deployment. And we're gonna go initiate our next rotation that's gonna be initiated from node one. So we're gonna take our maroon key, we're gonna promote it to be the new primary, and then we're gonna introduce a new staged key. So at this point, we've done a rotation. Our cluster is out of sync though too. And that's fine, technically everything still works because node one is gonna be issuing tokens that have been encrypted with a maroon key. Node two and node three have that key and that is the staged key. So they can technically validate tokens issued from node one. On the other side of the coin, we also have node two and node three that are still issuing tokens with the green key. And node one still has the green key around as a secondary key, which is only used for decryption. So that validation can still happen. So everything is still working. We can validate it, nothing's on fire, nothing's broken. And then we can go ahead and we can sync that out to the rest of the nodes in the deployment. So we're gonna do another rotation, but we're gonna initiate it from node one. But we're also at our max active key count. So we now remove the green key. So at this point, regardless of where we get the token or where we decide to validate a token, any tokens that have been encrypted with a green key should hopefully be invalid due to expiration. But the same pattern repeats. So at this point everything still works. We can still validate node one tokens against node two and node three. And we can validate node two tokens and node three tokens against node one. So we're gonna go ahead and sync our cluster out to the other nodes in the deployment. And everybody's back up to the same state. So your rotations are gonna go through that cycle where you initiate a rotation on a single node and then you go ahead and you push that out to the other nodes in the deployment. So this is actually what Keystone does. This is our implementation. And you can see that our colors have now been used with integers. So in Keystone we actually use integers for names. So if you remember the stage key is always a zero key and the primary key is always the key with the highest index. And then we go ahead and remove the old keys that are secondary. So what happens when you over rotate? And what we mean by over rotating is that you've taken and you've done a rotation twice without doing some sort of sync. So how does that affect your cluster and how does that affect token validation? So we're gonna move back to the same example. We're gonna initiate our rotation from node one and we're going to take our maroon key promoted to be primary and then we're gonna introduce a new stage key. So everything works at this point like we had seen in the previous example. So our next step should be doing a sync to the other nodes. But let's say we go ahead and initiate another rotation from node one. So we've taken our yellow key, it's new to primary. We have a new stage key, we're gonna remove the green key. At this point, depending on where we get a token and where we validate a token, that is going to be a problem because node two and node three can no longer actually validate tokens that have been issued by node one. Likewise, node one can no longer validate tokens issued from node two and node three because it's already cleaned up the green key. So we got kind of a mess here. But this is just going to show what happens if you break that cycle of rotation and syncing. And this is how we kind of suggest to do key rotation upstream and just some facts about it. And we used all of this with the Furnet setup and the Furnet rotate commands. You don't have to use that. It's just tooling that we provide with Keystone out of the box. We do have operators that use other things and Matt can actually talk to that because he's kind of broken that mold a little bit. All right, thank you. Okay, I'm gonna just give a brief overview of OpenStack, a Time Warner cable and then talk about how the Furnet switch went for us and how we're doing things like key rotation. So first, we run OpenStack in multiple regions. But for Keystone specifically, we have a region spanning MySQL Galera cluster that way users have the same credentials in both regions. Versions is an interesting story for us. Right now, we're running stable Liberty Keystone in a Docker container. We run mixed versions of other services. For example, Horizons off of Master. Most everything else is Liberty. But we don't tend to keep everything in sync because we're in using containers so we have a little more freedom there. And we've been using Furnet tokens in production since July of 2015, so about nine or so months. So before we started, what were the prerequisites for Furnet tokens? And some of these we knew, they were documented and some of these we kind of discovered along the way. The first one, very clear, is you need Keystone, Kilo or Neewer. That's when they were introduced. However, for stability and performance reasons, we recommend you use at least Liberty in probably Metaka now. Also, between Kilo and Liberty, Furnet tokens underwent a slight format change where some padding was removed and that change is disruptive. So if you're going to do Furnet, just move to Liberty first and save yourself the hassle. Okay, what about Horizon? Horizon needs Django OpenStack off of at least Kilo or Neewer. What about everything else? So all the other services use Keystone Middleware to talk to Keystone to validate tokens passed into them. They also are holding their own service token which is how they talk to Keystone. And as it turns out, if you have an old copy of Keystone Middleware pre-Kilo, the services don't realize that the token they're holding is no longer valid and let us keep trying to use it until it expires. So this can be disruptive. So recommend you upgrade all your services to at least Kilo before you do this or you have to restart services.  Cashing, this is not technically a prerequisite. You don't have to do it but Keystone does a lot more work to validate a Furnet token than it did for UUID token. And so the very first thing we did after we released out with Furnet is we turned caching on. On this, this saves Keystone from having to do a ton of work. Okay, so what was our migration path here? The first question people would ask me is like, why do this in July of 2015? Dolph's already alluded to most of these problems. We had some weird token replication issues between data centers.  They would get a token in the West region, try to use it in the East region, it wouldn't work. It would fail like a certain percentage of the time. So the solution was to go to all our check scripts and put sleep calls in them and hopefully those were long enough to get the token replicated and it was just kind of a horrible hack.  Also, we tried to kind of like to be running the latest greatest thing. That was kind of the latest greatest thing back in 2015 and kind of gets us ahead of the game for when Furnet becomes default, hopefully in Newton or Alcada. Okay, how do we orchestrate the upgrade? We don't do anything in our cloud without automation and for this work, we chose Ansible to drive Puppet and Puppet was gonna do the things it's good at which is changing config options in Keystone and restarting services. Ansible is gonna do the stuff it's good at which is handling the inner node dependencies. Also, I should say, we knew this was gonna cause a brief outage and so we wanted to sort of minimize the number of outages and also we just like doing things the hard way so we did Furnet token switch, upgrade Keystone version and switch from a vent led to Apache all at once. And so this set of playbooks I have a link to here for is designed with that in mind, all three of those things. And so it includes things like database backups shutting down part of the cluster, anticipation of database migrations. If you're just switching token formats, the playbook is a good start but it will be totally overkill. You'll be deleting stuff. Okay, what was our outage? So in production, our API outage was 16 seconds. However, almost all of that was due to things like Puppet having to upgrade packages, run database migrations, get Apache configured, get Apache restarted and then at the end, like I said, we had to bounce all the other services so that they would get a valid token. If you are just switching token formats and not doing anything else and you're on Liberty, it should be very brief, just a couple seconds probably. Token rotation, let's dive into this because we don't do it the way that upstream does. Okay, step one, we don't use Keystone Manage for net setup, we don't use Keystone Manage for net rotate. The reason why is because if you watched his slide, he's kind of implying, Lance is kind of implying that one node there is like a master node and all the rotations happen on the master node and then all the keys move over and that kind of violates how we view our cloud where all Keystone nodes are just throw away, we don't have any special code. We don't like to run any special code on one node versus another. If Keystone node one died and we didn't rebuild it for a month, we want to not have to handle that situation with key rotation. So what do we do instead? We actually store our keys in encrypted EAML and have Puppet drop them on the box just into the Etsy for net keys folder and that happens whenever we do a deploy. So this one's interesting. So we actually wrote tooling since we're not running net rotate, we wrote a Python script that would basically take that encrypted EAML, check it out, do a key rotation, re-encrypt it and then post it up to Garrett. And so you might ask yourself as my team often does why am I reviewing an encrypted version of a Furnet key? I can't read these to make no sense. The only point of this for us is to prevent over rotation. We thought about having this just run automatically every week but there's certain weeks we don't do deploys like this week where most of my team is here and if we have something rotating every week and checking in and we miss a deploy, we've now done two rotations and it's gonna break all the outstanding keys. Lance went into this but to reiterate, the rotations do not have to be in sync. When we do deploys, we do one region first and then the other and within those regions we actually serialize Keystone nodes so anywhere along the way when we've done a rotation our keys are about to sync. In addition, we'll stop the deploy after one region and do a bunch of validation and if we find that the region is kinda screwed up we won't do the other region and in fact we had a deploy where we waited two days between doing East and West. So for two days, East and West had different sets of Furnet keys and doesn't matter because they're all using keys that are already on the box, the staged and the old primary. How often do we rotate? So all the factors were explained before but a big primary factor for us is how often we do deploys and that's weekly and so therefore we rotate keys every three to four weeks generally. Okay, so since July, what have we learned here by operating the Furnet tokens? First, already said but empty token table which was great. Our Keystone database is super small now. There's hardly anything in Keystone database without tokens, it's just users, some assignments and some projects and it's pretty simple.  Finally, without having so much database traffic especially and not having to replicate tokens across regions, the replication lag vanished. Just in general the load of the Keystone replication load decreased greatly we're only syncing users and projects and stuff and there's just not that much going on. Okay, performance, this is probably a big one. So Furnet tokens are slower to validate than UUID tokens. They're faster to get but they're slower to validate and token validation is important for your cloud performance. Every API call you make to OpenStack requires a token to be validated and I just scanned through some logs a few weeks ago and it looks like for every token we create we validate about 10 tokens. So you're doing a lot of token validations. So what we highly recommend is you just turn on Memcache. There's config options for it in Keystone. You're basically Keystone seeing the same tokens over and over and over again. Get those cached in Memcache, save Keystone all the work of having to re-validate it. Also if this is important to you we've been benchmarking our cloud in terms of token create and validate performance and kind of watching that, seeing what tweaks we could make. So we recommend you do that if it's a concern. And the other good news is LanceDolph and the rest of the Keystone community is gonna be spending some time on token performance and Keystone performance in general in the Newton cycle and hopefully most of those patches should be back portables, especially if they're for Furnet tokens at least back to Mataka. So things should get faster. Okay. I wanna just leave this slide up as we do the closing remarks. This is basically just a list of where you can get more info. It covers key rotation. It covers like what's in a key and just some of my experiences and some of their experiences with running Furnet tokens. But hopefully you guys will learn something today about Furnet tokens and how they're created and you've least thought about some of the things you're gonna have to consider when you switch your cloud over to using them. And we have some time for questions. I have two questions. The first is in all your examples, you have max keys set to essentially n plus one of your rotation schedules. There a reason you couldn't do n plus two to work around the over rotation problem? Yeah, absolutely. That's an option. You can always set that higher. You don't have to set it to the absolute minimum in case you do accidentally do an over rotation. It kind of gives you a little bit of a buffer there. So setting it to n plus two, that's totally acceptable. The other question was, you mentioned key synchronization a lot, but not the mechanism for doing so. Could you describe a couple of ways that you all do that? So the OpenStack Ansible project has an interesting way of doing that that kind of touches on the point that Matt made about not having a single keystone node be the master and that's just done through or syncing between the different nodes and syncing that repository. So it's really kind of whatever tooling makes sense for your deployment. If you already have something that maybe syncs keys between, or like SSH keys or whatever, between different nodes, you can build on top of that and leverage that same kind of tooling for syncing your cluster with your front keys. But there's a few different ways of doing it. It just kind of depends on what you use. Maybe back in. I was just saying there's no official upstream way to do it. So that's why we just kind of put it into our existing deployment process that used to Puppet and EAML. But everyone's cloud may be different there. All right, thanks. How is Keystone protecting the keys that are stored on disk? We totally leave that up to the deployer. So disk is a fairly safe place to put them. Put them in a RAM disk or something if you're really worried about things being on disk. But we punt, totally. So first, thanks for the talk, it was really useful. I didn't forget, you mentioned that to determine the frequency of the rotation, you need to analyze what you expect, like what time you expect for people to break the keys. But since it's like fixed size, I expect that upstream can have a recommendation for that. Do you have a recommendation for the frequency? All of your examples were like every half and every week and every month. And so it's totally different. So you're asking what the recommended rotation frequency is? Yeah. Totally up to you. Well, that's broken, I think. There should be an upstream recommendation about that because people don't know how to calculate that.  But it'll be hard to get upstream Keystone, I think, to make a recommendation on that. Because it's just gonna depend on how many tokens you're putting out and your security. I mean, I understand, but my point is that if Furnit is default, then that must be a recommendation because people don't know how to choose that. So. Oh, sure. So we are working to make Furnit the default. It's been a long road to make that happen. But based on the other variables we have in Keystone, we currently have a default expiration. We currently have a default max active keys, which just kind of exercises the full rotation cycle. And so from there, you have a minimum value of how frequently you need to rotate, or sorry, maximum rotation frequency. So, could start with that number, I guess, but it's just a recommendation. Yeah, I mean, I'm just saying there should be an upstream recommendation because most people will not know how to come to a conclusion for that. I believe there is a Keystone work session either today or tomorrow, where Furnit defaults getting covered. So it would probably be good feedback to come and bring to that. Sounds good. I think so. Again, thanks for all the talk, it was pretty good. That's it. Thanks, everybody. I think we're done. Oh, one more. Informally, I keep hearing statements that Furnit is not quite ready. There's some missing elements to it for production use. Can you speak to that? Well, I think Matt would disagree with you. The road to getting it default is probably the biggest blocker. Tempest makes some assumptions about, like, what it can do with tokens. And so it's been a matter of, like, changing Tempest's assumptions or increasing the timestamp precision and Furnit to avoid race conditions just in a very aggressive testing suite. So that's about it. Yeah, I just said we haven't had any major issues with it. We noticed this a little bit, this performance change and validation when we implemented it, but getting rid of all the database replication stuff was a win for us. We're also using it in Rackspace, public and private cloud, essentially, moving forward. Definitely production ready for us. One of the things in the release notes was that performance verification is indeed very slow, probably a couple of times than issuing. So how or where would you try to mitigate that? Or when would you see so many revocations? Because usually you'd expect them just to expire. Revocating is not something that happens all that often. You're asking about token validation performance? No, revocation. Replication of keys? Yeah, so keys or tokens? Tokens. So token revocation is something I'm concerned about because the performance in Keystone as that table grows gets really horrible and in fact we like to just kind of keep an eye on it. The only time I know that we really revoke tokens at our company is when my testers here are testing whether or not it works and when people sign out of Horizon. So we typically only have like 5 or 10 in the table unless the test suite is just run. So I haven't been super worried about it performance-wise, but it definitely hurts. And I have a blog post on it that you could probably go find off my website. Thanks. I think we're out of time, but we can talk to someone. We can just talk to you up here if you want.