Hey, welcome. This is the introduction to Wikimedia Cloud Services staff. I'm Maldivis, this is Brian and Chase, and they work with me on the Cloud Services team at the Wikimedia Foundation. And we work on infrastructure and platforms to service things for the Wikimedia movement projects. And it's built by us and Andrew, who's on our team, and a lot more, while I get a little worse from the community. So what is Cloud Services? Basically, we are a whole thinking platform, which means that if you have software that has anything to do with the Wikimedia movement, we have computing resources for the university. On top of that, we provide a lot more abstractions to make it easy. We have a web service, if you have an API that you want to run somewhere, you have a little bot, you have a visualization that you don't know where to go. You write a query and you want to just run it against our public databases. We have tools and services that help abstract all those things, and that's kind of what the second thing there talks about. And then all of these platforms and services that we offer, we offer technical and community support. There's a documentation, talking about IRC, people helping them, tracking bugs and responding on a calculator, which is the platform that we use for tracking feature requests and bugs, and responding there. That's all the technical and community support stuff that we do. One thing that I want to add is that this is super big definitely, you can ask questions at any time, so be fine, and you can stop me if I'm using any words that are charging at you and ask. There's no questions that are not meant to be asked except why you overcloud me. Yeah, so these are all the products that we offer and started Cloud VBS, which is the conditional infrastructure that we have. It is an OpenStack-based hosting platform and OpenStack basically is this big piece of cloud software that helps you cluster a bunch of different servers together and offer virtual machines, which are lightweight computers or servers that you log into and install and run things on. So that's the underlying infrastructure on top of which we run all of the other stuff there. Toolboard is the next thing in which we run tools, and tools could be the things that I talked about before, which are a web application that you wrote, a bot, or a job that you want to run every two hours, every minute. Anything like that, you just don't want to handle the underlying server, don't want to install stuff, you just put it in, you write tool, you put it on, you post it on Toolforge and we manage that stuff. Quarry and Paws are things that actually run on top of either VBS or Toolforge, and both of them are GUI graphical interface tools that have quarry servers that run SQL queries on the web, the single web interface. And it runs it against the wiki replicas stuff that's over there, which is the public replicas of our production database servers, all the media wiki databases are there. And Paws is kind of similar to quarry in that it's also a graphical interface, but it's Jupyter Logo. How many of you know about Jupyter Logo? It's basically you can write Python in a graphical interface, so you just maybe can show it later at the end.  And you can also access our databases and stuff through it. The wiki replicas, ToolsDB and DOMS are all data services that we offer. They're kind of one layer beneath everything else. And we have, I think, more detailed architecture-ish diagram later that you can cover in more detail above. And ToolsDB allows people to store data in a MySQL data store. So MySQL Database is a service. And DOMS is all of the public media DOMS that you can get from DOMS or wiki media.org that we can allow you to access through any of the services that you use in our platform. There's some cool stats around there. Just to show off how much of stuff is used in that these are all the OpenStack projects down here on the left. To the team projects in 750 instances, those are the CloudBBS projects. And Toolforge hosts about 1,413 tools. And there are 1,700 maintainers. And these stats I think are ground last month. So those are from a little earlier. And 24.67% is the number of edits that bots and services that run on our infrastructure may be to wiki media wikis. And that's the amount of 3.8 billion API requests, similar by bots and services that run on our infrastructure to the wikis. You've maybe not heard of the word Cloud Services before, but maybe you've already used it in some way or the other. So this tool here, MonoMetal, it is one of the wiki loves MonoMens projects with a bunch of other projects that I'm not naming. And it basically allows you to explore MonoMens by location. And it's cool that it's hosted on the Toolforge.com. This is wikiSource.export. If you use wikiSource, they have a tool that runs there that lets you explore wikiSource's book to an EPUB or any other mobile reader format. And that runs on toolstool. This is outreach dashboard. And I think this is its own little VPS project and all of the program events for the outreach folks that's having here. Here it's just a wikiPia offline project. All of their, they generate the ZIN files that basically get used for their time site like apps and all the things that they have for making the offline wikiPia available. All the generation from the dumps happens on our end. And I think it is a tool for the end of the offline world. And this is the service that I covered before. It's Quarry. And you can see here, it's just, you write as you well in a graphic reader page. And it spits out the results as a table right beneath this, which it is not covered in the screenshot. You may have heard of any of these tools. A lot of other things that you may have heard of today like OARS, which is Aaron's AI project. All of those things were once hosted on labs or still like our... VPS projects will provide like staging environments for those projects even though even if they have graduated on to production. Yeah, and Brian is going to cover a bit more about how the team performed and what the industry is. Yeah, I'll ramble for a while. So, you heard, audio would have to put a dollar in this storage art for saving labs. So, this environment that we're talking about cloud services was until April of this year called labs. And the thing that we're calling tool forage now was until April of this year called tool labs. You may start to see that we had a little naming problem. The team was also called labs. So the team was called labs, the environment was called labs. One of the things running it was called tool labs. And nobody knew what the word labs really meant other than it sounds either depending on what kind of person you are. It either sounds sciencey and cool or like mad experiments. And over time, you know, things started out kind of in the mad experiment phase. So all the way back in 2005, Wikimedia Germany, Wikimedia Deutschland set up an environment that they called tool server. And this was a place for hosting tools, which whenever I say it, see the big air quotes around it. Tools is kind of a generic term in the movement for any piece of software that does anything that was replaying anybody and runs anywhere. But specifically, the tool server was set up to help people who were running some of the early bots and some of the early web services that were helping augment what MediaWiki can do. We all know that there's limitations in MediaWiki. There's only so many things that you can get done inside it. But we have more complex workflows, especially social workflows that happen on the Wikis. And people found they needed to build extra things to help make lists of horrible things that needed to be cleaned up or make lists of good things that needed to be promoted or look around for gaps in coverage in some topical areas. So that's kind of where tools started. And Wikimedia Deutschland folks took care of that environment for a long time. From 2005 all the way up there into 2013. That was a volunteer run project that was funded in various summary ways. Jump forward a bit. We jumped forward a bit to 2011. The Wikimedia Foundation, Ryan Lane, thank you. Too many Ryan's. Ryan Lane was an operations engineer at the Wikimedia Foundation. And he was into this software project called OpenStack that allowed you to take a pile of computers and turn them into this virtual cloud thing. Kind of like Amazon AWS hosting or Rackspace or a whole lot of other providers that are available in the world today. So he came in and pitched to Mark, Bersma and Eric Muller that we should set up a cloud environment at the foundation. And that this environment should specifically be used to set up a replica of production that we could allow volunteer system administrators into. And give them full-room privileges and let them mess around and help build the puppet manifests. The software that we use to automate software employees to help build that project and system out. And get elevated rights that we would have difficulty giving them on the actual production cluster. But since this was kind of a side testing environment and hopefully would be easy to recreate multiple projects that did almost the same thing. And let people tweak things in different ways. We should build that out. So in 2011 they started building that, started with just Ryan in the beginning. And then pretty soon after that, Andrew Bogot, who is part of the current cloud services team was hired to assist Ryan. And then things went on and it kept getting better and better. And in 2013, the tool server folks were having some growing pains issues. Their hardware was getting kind of old and they were having some issues keeping their maintainer community vibrant. Keeping the volunteers that were keeping everything under the hood running. Keeping them from getting burned out and active and supported. And so they started talking to the foundation probably actually in 2012. They started talking to the foundation, but in 2013 we finally did something about it. Started talking to the foundation about the next generation of the tool server. And could it have more resources? Could it have some dedicated support? Were there other some other things that you could do? And so what happened in 2013 was that we realized that we had this labs environment. And it was up and it was running this OpenStack Cloudy. And it was being useful. And so we decided to found a project inside our OpenStack environment to become the replacement for tool server. And Corrin, Mark Andre, was hired around that time to be part of this building out the new tool labs thing. A whole lot of cool stuff happened that around 2014-ish, I think 2014-ish, Yuvie Panda came in from another team. He was working on the Android team at the time building the Android mobile. That project kind of got to its conclusion at that point. And he came over and became part of the tool labs group. Inside the larger labs team, check building things out. And we kept adding more resources. In 2016, Chase became a dedicated manager of the team. And it kind of broke out as not just a couple of free people in the TechOps team, but actually became an official team inside TechOps with Chase as the HR manager. And also in 2016, I convinced my boss at the time, Toby Negrant, to let me jump ship out of my role as a manager of the reading infrastructure team and move over into the community tech team specifically so that I can work with the tool labs developer community and start to try to be kind of a developer liaison and a resource to build new things for them. And then in 2017, in late 2016, Chase and I were talking at a conference about a lot of the challenges that he was seeing from inside the team and I was seeing as somebody who was trying to help promote the tool labs forward. Yeah, I can't even see the wrong names there, that's awesome. So we started talking about the idea of taking this team that he was leading inside the technical operations group and this team of one that I had formed for myself inside community tech and merging them together and pitching to solve several problems. And one of the problems that we really hit hard on in the internal pitches that we gave to the foundation was this naming problem, that labs and tool labs didn't mean anything to anybody. We actually have a page on wiki tech called labs labs labs that he started years ago that lists the many ways that the term labs was overloaded and that when somebody just said labs without qualifying it as witch labs, confusion ensued. So that was part of what we pitched. Another part of what we pitched was some of those numbers that were back on the slide that that Lu showed you and they're up on the poster over there behind Chase that what the real impact of these volunteer developed software projects was for the overall movement, right? When you see that number, what is it? 24.6%, 24.6%. So that was over a 90 day window across all projects, all 813 wikis. A quarter of the edits were made by software and volunteers ran inside what at that time was Chase's environment. There's some footnote numbers there and I don't, you know, citation needed. I actually do have the citations. I can dig them up. But 50% of the edits, a little bit over 50% of the edits to wiki data during that same time window were made from labs and tool labs.  The number was somewhere around 15%. I don't have the exact number on the top of my head, but the number was somewhere around 15% of the edits on English with P and were made by lots of them. So that led to this pitch inside the foundation and April 2017 we got the green light that we were a real team and our job now is, you know, now we're called cloud services instead of the labs team. We're actually called the wiki media cloud services team and we're in charge of the rebranded projects. So the thing that was called labs that meant OpenStack, labs that meant the team or labs that meant something else. They called labs that meant OpenStack. We're now calling cloud VPS and we picked that word, the VPS word because we hoped that it means something to people who've never heard it before. It made everything to everybody. It's not a universal term, right? But it is something that means things in the world of software hosting. I'm probably way too low on this. Virtual private server. And the rebranding of tool labs to tool forge. Basically once we started chasing the word labs we want to get rid of it everywhere because if we leave it anywhere, then we leave it and it never goes away. So tool labs turns into tool forge. And then we started actually giving names to some other things that hadn't had names before. Like wiki replicas have been called labs DB or the replicas or the slaves or the database or whatever. Tools DB was called tools DB when people thought of it as a thing in and of itself, but it often got logged in with the wiki replicas as just our database and not that it was special and distinct in that it provided different services. And we've come to the point where we're taking on some new responsibilities. So the dumps has historically been managed by Ariel and the ops team. And they do a really great job of focusing on producing the dumps, but never really were given many resources to worry about getting the dumps out to the world once they were produced. And so we're taking over that getting them out to the world side of it. We're starting to build the servers right now. Just a couple of weeks ago building new servers. New servers with bigger hard drives that are faster to hold the dumps and we're going to be taking over the HP interface and the NFS interfaces and the R-sync interfaces and all of that. It gets enough ramboos. We got architecture over here. Chase would love to talk to you about architecture over here. Sure. I'll cue you or anything. Sure. This is a logical review of things. The tool for the environment which is sort of the friendly environment that you can run something like that. You don't want to be responsible for a whole virtual machine. Say you want to run a web service and get a project and add virtual machines to have to update, maintain, you have to pay attention to this where we email you the idea that we're going to be rebooting them or we're updating some portion on that. And there is an overhead involved. So if you just want to do that, hopefully Toolforge is the place where you can do that. We have, I think, 1,200 or 1,300 of those tools. Our most of the web services are at least more fairly low-service top heavy which makes sense because a lot of it's data availability or visualization or research output or just web interfaces that some specific initiative needed some place to part. So within this project we have Open Grid Engine which is a distributed, scandaling environment. Essentially if you have some number of computers and you want a way to launch a job to run a web service or go crunch numbers you need to find a way to spread it out around a pool. And Grid Engine has a project to do that and it knows how many things are running where and what nodes are visiting when nodes are not visiting and it knows how to find the place for your thing that is most likely to be successful based on resources. Kubernetes can perform very similar functions but for more modern it has the ability to constrain the resources that a job can actually use so you want to run something that you say, Grid Engine looks and sees work that exists and shoots you out there but if it goes haywire it uses five gates and affects other people there's not a lot of room for exploits. Kubernetes has the ability to put a ceiling on resource usage but then also track how many resources are being used like how much compute time is being used by someone's web servers for a job and those are numbers that we've sort of longed for over the last few years. People really want to know understandably how much bandwidth it is and how many hits does a certain website get, how much CPU, how much memory. And Kubernetes uses some of that for free along with the fact that it's just a more modern container-based platform. Containers a big word but essentially it means that you can wrap up your tool in this little package and ship it out and you don't have to concern yourself with what everyone else is doing. So this all runs as a project but the project itself isn't special other than that it's really big. I don't know what percentage of VPS's tools but I think it's 125 virtual machines and we have 715 or 18% or something like that. But we have 212 other projects that people are running and some portion of that, I don't know, 20% or 30% are internal initiatives, right? People want to rewrite a new Kubernetes control application that we want to use in production but the majority are volunteer run or at least volunteer and staff collaborative projects. There's tons of things happening in the mass realm. I don't know all of them but they're busy. Query is the sort of dumps my SQL exploration tool where you can see what other people have run and kind of cobble together queries from examples you've found and it's a way so that what's in front of you is a black screen with a cursor. UV when he wrote it has a very accessibility mindset and one of his tag lines is that you don't have to pay the command line tax which is having all the context it takes to get up to the very last inch of what we actually wanted to do which was just to run one query to find out one little thing, right? So in theory this jumps you to 90% because everybody has a web browser. And then tools which is kind of the meta there. But other than sort of we started viewing these as products for ourselves and so we can communicate outward. So we call this the platforms of service which is a fairly industry standard term. This is infrastructure as a service which is fairly industry standard. So this is VPS but that's actually the old term that's been used since the 80s. And then data services is sort of the category of things that fuel these farmers so to speak. A lot of the reason that people come to tools or people start a project is to get easy access and to get free computer resources to look at the data that's available. So the looking records are essentially sanitized copies of metadata from all public wings and people need really crazy number crunching looking at editor retention and all kinds of things. The super popular stats page that's been around for a million years and we're putting tools of work into recreating that has all been done forever by volunteers using open resources. Dumps, everyone knows about. The site bar there is essentially we have dumps for the world and then we've been making copies of that to present to this environment and we're just collapsing all of it because it makes sense it's a combo. And then shared storage is sort of the underpinnings of the entire tool portion of our environment but then we also have to like share the scratch base if you need some place to park and carry away sort of temporary data and stuff like that. One question. What's the difference between replica dumps here and dumps here? Okay, so if you go to dumpstabicmedia.org there's a web engine you can download a bunch of stat files essentially and the archives there's all kinds of stuff like that. So those are files that you could download and unpack and kind of dig through. We offer those in our environment as well via NFS so like instead of so they're pretty large storage wide storage-wise so not everyone has to have their own copy of these static flat files, right? The wiki-rate because our real-time live that are constantly receiving updates from production as things are happening. So like one... I don't want to use a bad use case but I can't think of it better but basically let's just say you want to watch every edit as it happens you know for some patrol reason this is not something that you should do that but many moons ago that was really seem to have been very common because it's easy to understand that as changes are made to the database and they are replicated that you can consume them and then take action, right? Now we want you to use RCStream or whatever you are using. What is it? EventStream. EventStream, please use EventStream. What is it? So the difference is one is a built and placed completely static file that doesn't change I think we generate dumps twice monthly so like every few weeks we have a job that runs that takes everything and kind of combines it and drops the file and we do have people who unpack those and dig through for statistics or to build specific visual interfaces to historical data the use cases for labs do tend to be more dynamic like if you want to build query query has the ability to run ad hoc queries you know and you want to do that on some kind of structured query language back into format so it makes sense that just hits my seat with other words. Does that make sense? Yeah, thank you. I think the other one major difference that maybe was buried in what Chase said but maybe was buried is that the Wiki replicas do not contain the actual Wiki text of the pages so it's all metadata about all the edits and things that happen but there's no actual Wiki text there the dumps contain not only the current revision but all the historic revision Wiki text so like the dump XML file for Barack Obama is many many megabytes long and if you page through every revision of every history of Barack Obama it's all collected in one check on this file then you can go. Yeah, that was a lot of historical One neat thing about the dumps kind of revitalization is right now I believe we have three historical iterations of dumps available to our users whereas as we combine platforms we're hoping that people can access just everything that we have which will be set in the moment and there's also a lot of data on dumps that isn't accessible from this environment that will be nearly preferred We wanted to use Quori to analyze content because we wouldn't be using Barack Obama because that's just metadata Yeah, so Quori today cannot actually access page content so you can't do things like show me all the pages that have template X You can do things if there's metadata about it so you can do things like show me all the pages that use image Y because there's a special metadata tracking table that keeps track of what image is used on what content page and there's also special tracking tables for what pages link to each other so you can ask the questions like that but you can't ask it like find me all the pages that include the phrase two more tabs somewhere in the content Quori is just in front of Wikipedia because you don't have to have SSH access to log into a server and run the same query you just log in somewhere or you can detect for meta credentials and then you just run the query but the database is the same the reason for no content as I understand it is essentially resourcing that I don't know if it's that Yeah, it's essentially that like to re-host all of the content for all of the ways is this sort of attractable problem where we can offer you all the metadata and you can use it to make cat calls to production which there are a lot of tools that do that like figure out what you want based on kind of syncing through the pile of the rebel and then just being told I can get a couple of these when I need them Sure Alright So, multiple zones per client DPS and this is a big one this was a big part of our pitch to sort of leverage more resources and look for more resources Right now the media foundation has a number of data centers, right? and we run through testing where one of the data centers were to disappear we can move to another one All of this, the stuff that makes not quite 25% of all the edits and those are disproportionately valuable edits because a lot of that is copy patrolled and those are patrolled that drown the rest of the 75% of the edit 5% all of that is currently physically run out of one single data center and that's very problematic at least as far as robustness is concerned and the main problem is that the underpinnings platform that opens stacks up it's sort of operating in a legacy mode that the main projects have moved away from and we never had the extra time or the time to kind of put in the work to climb that mountain where we were trying to water it So when we talk about can we fail the wikis over if the primary data center disappears? Well you can, but you're going to use all of the tooling that people have spent for a decade working on to actually make the ecosystem viable So zone is just a designator for particular components of the environment where we can sort of have virtual machines that are run out of physical places so like run out of east coast run out of central US and that allows us to hopefully be robust beyond geographic events and not like giant stormings or hurricanes or whatever the thing is that you definitely want all you don't want all of your animals in that Modern and robust networking is a big part of that and essentially the problem that's the reason why we can't have things that exist outside of this one location is the networking model is old as we move away from it And I'm going to let Madhu talk about the dumps and the polygons I'm sure I also want to give a pause and ask if there were any questions before we learn about all of the stuff that we are about to do in the future I'm wondering when you say that we help people post tools tools that are going to add value to the big media ecosystem How do you decide that a tool is going to add value who decides that and whether that process will also be affected by the network Yeah that's a pretty good question It's an awesome question So the process of deciding right now today is that there is a tool within Toolforge called the Admin Tool and everyone who is a maintainer of the Admin Tool has the rights to grant new people into the overall product and there is a web-based workflow for that that is managed through toolsadmin.wikimedia.org So that's where you go to apply for tools membership and when you do there is basically an echo like notification that goes out to all the admins and as to how we decide basically if you can say anything that makes sense about what you're going to do we'll let you in it's a really low barrier so ideally it would be a transparent barrier like ideally it would be the wiki way and we'd let people in and we'd let them try to do something and if it turned out to be a bad thing we'd undo it just like we undo a bad thing It turns out in our software tooling it's a little harder to undo bad things so there's just this very low level like some other human has to click a button that says okay I thought about this for 5 seconds and sure they're in I think in the amount of time that I've been helping judge these applications maybe 7 of them haven't been denied and in that same amount of time probably 250, 300, 400 maybe even have been approved mostly the things that we deny end up being people who say I need to use your computers so that I can build some more profit project that has absolutely nothing to do with Wikimedia or you wanted to play closer software or you wanted to play closer software or a couple of times it's basically just been somebody who really didn't make anything in the fly and so we left a note and said please tell us more and then after a month that they don't respond then I just close it as decline and figure if they come back tomorrow they can open up a new one in SafeLine That's mostly tools but we also do I mean there's two levels you get an account and you can say I need my own project which is just an allocation of quota to go do things or you can say I just want to join tool the tool for a project because you know I'm simply the one specific thing and applying for a project basically requires who are you what in turn do you want and almost always the main reason we're asking why you want to do it is because someone else might already be doing it and we're going to point you their way or you may not quite understand what you're asking and what you've described as humanly possible or technologically possible or maybe you would be a bad actor if you were in the system like you're close or soft or some kind of poor property we're essentially a niche hosting environment with a lot of big problems one of the questions that came up when we were pitching the branding and using the cloud term and sort of consolidating on industry standard terms is how do you better out bad actors how do you make sure someone doesn't run a business off of this platform you know what I mean like how do you make sure someone is an active bad face and the answer that we gave Ben and Ben is well we pretty much have a lightweight contextual barrier to entry we understand what people are doing on the platform we don't you know we're not looking at our shoulder after that but for the most part the users of the platform themselves have way more knowledge inside the context into what's happening I mean most of the time when someone is doing something like using external resources that lead user IPs it's other tool for volunteers who are probably editors probably on wiki active and they're the ones who see the issue and call it out so I think in general sorry this is kind of long I think in general myself and Andrew there was a time when it was just he and I and we had a discussion and we really don't feel like we're the arbiters of what's valuable to the projects you know what I mean like in some ways you could be standing in that role but that's not our thing our thing is more like technical reality like are you describing a thing which you could actually do here and can you describe it to me in a way to which I understand sounds good to me so it's a very low barrier to entry sort of litmus, smell test slash you know actual feasibility we covered quite a few times but trying to change basically it's just that combine all of the services use cases because right now we sell like we have our speakers we have people who download it on the web and then people who access it through our environment and we just want to combine all of the external service use cases and post it on our site rather than currently it's handle by operation folks so all the data generation will be handle by them and then we're going to handle serving the data to all the different consumers of data that works post-performance initial improvements I am not really the person to talk about it you may who is no longer in but is still active you're working on post people and also for you and I just know how to log into your tool sometimes I will restart it but he is working on some of the online post-off and is saying that it will get a lot better so watch out for improvements there I don't know what else I think Brian is working on a lot of cover stuff documentation, workflows contractor community involvement do you want to cover some of that? sure a lot of the things that we're looking to approve from the foundation side is to provide better support for our communities of volunteer developers we've been doing an okay job for several years of making sure that there are computers and that the computers are up and that they're running but we kind of want to go beyond that and try to help help new people learn how to use the things better help people who have built things better promote the things that they've built to other developers and to people on wiki and to people off wiki we want to do a better job of simplifying things of following those things that UV was really kind of a groundbreaking champion for inside the foundation about simplifying things where they can be simplified and streamlining the processes that volunteers have to go through the thing there that I kind of I talk about internally quite a bit is that there's a very large difference between Greg sitting over here in the corner and Greg's release engineer team is a very very heavy user of cloudvps services the Jenkins build testing system runs on cloudvps the beta cluster that's our near real time continuous integration environment runs on cloudvps several other initiatives that they work on right now there's a really big difference between when Greg's team who's primarily paying software engineers and employ the wiki media foundation the time that they have to invest in learning how the tooling works and taking care of things and keeping them running versus the time of our average tool developer who maybe figures out how to take hours of a Saturday afternoon one weekend a month to work on making a cool thing that makes it easier to do something, some workflow on wikis it's kind of our job I think as providers of that platform to work at making it more and more often that when they sit down to spend those four hours that they get to spend three hours and fifty minutes of it doing something cool and only ten minutes of it doing something boring and repetitive and without some care and feeding of that it quickly becomes the other way that every time you sit down to touch the thing it takes you three hours to figure out what am I supposed to do and then you're only left with an hour to do the cool thing that you came to make everything better so that's a lot of that the better workflows or account creation for new people hiring a tech support contractor and even like consolidation of platforms is all towards that that kind of idea about just get as much of the mess out of the way as we can or take it on ourselves so that the volunteers who are spending those very precious hours can spend them doing good things that's a good idea alright I'll get it to the end of the point hi man we got that aha wiki replicas upgrade so we didn't really talk about this much yet but this is a big thing and our brilliant dba team has been working for how many months have you been working on this? yeah yeah so in these real time replicas of the wikis that we have one of the historical problems is that media wiki's database structure itself is a little unique and making copies of it making redacted copies of it so when we copy it into the cloud of data services environment we can't just take a complete binary copy of what's running in production and put it out where all our volunteers can access it because there's data inside there that's special protected data probably a lot of you that work on the wikis know something about check user and that check user means scary and there's something about IP addresses and privacy so we have to take that kind of data that's very privileged data that people aren't normally allowed to see and we have to remove it from the database copies that we allow people to get to and this process historically has made it difficult and error prone to keep up to date with changes sometimes when data is deleted or marked as deleted in production sometimes when things get added in production they don't get added properly in the replicas kind of because of all the kinds of mechanisms so long story long the dba team thought about this a lot and tried to think about ways to make it better and they came up with a whole bunch of changes in the tooling pipeline that they use and the actual way that the data servers are set up and deployed and maintained to make it more robust and easier to keep track of and then once they had that set up they had a many many month process of refilling those databases with all the copies of the production data run through the new pipeline to clean out the things that we aren't allowed to show you because the privacy invasive somehow and just last week we finished we got all the data pushed in and now we have them keeping up to date with new data changes as they come in so we have a new a new set of servers that we hope are well they're bigger faster hardware so that's a good start but we hope that they're also more reliable copies of what's in production particularly to be deltas between what you can see on wiki and what you can see in the replica and so now what we need to do is we need to get some people who are existing users in tool labs and labs of these databases of the existing wiki to switch over and start using these new wiki replicant databases so that we can kind of break them in and make sure that the performance characteristics that we hope are there are actually there and shake out any problems that we might have with changes that we did in the permissions model and things like that that you're not seeing things that you expect to see hopefully we won't find any where you're seeing things you don't expect to see but we also need to look for that and then so we're kind of putting out a call to existing users and we're going to start doing this with a little blog post in a couple of weeks and the mailing list push lets the whole wiki media craziness guys guys down give people to use them and then hopefully within a couple of months we'll be able to switch over and make these the real copies that everybody is using all the time and solve some kind of long standing problems that never do 22 yeah so the existing setup is interesting in that so you want to connect to a specific wiki you have some DNS magic to make that easy for at least a few people and you can find a specific wiki but then you run your query or whatever there are protection measures in place that take up disproportionate amount of resources but all of those users are kind of lumped in and you've got people doing they want to be long term analytics and you've got people running web services and so part of the idea of the switchover is to hopefully separate use cases to be more friendly to all so that a web service isn't tripping over long running analytics queries and less personal so these two surface profiles are part of the transition and we're hoping that people will be able to point to the right place and that on the back of the battle process we'll only see if it was for the users so a minor trivia and you know it's mostly come out of the hardware and scaling up horizontally and all of that but these are actually at least at the time the most expensive surface that we should have in the first place so partially because the existing setup just gets hampered you know like we need a lot of to a lot of users to not constantly above the average so we owe the DVAs pretty big than a grant the entire Spanish DVAT I've been working on this for six months every database administrator in Spain has been downgrading in this project so it's pretty cool the last slide this is one of the ways that you can contribute or participate I want to note that we try to be extremely beginner friendly on our IHC channel we're subject to the core of conduct on this channel and to just get started with any of our projects we have documentation in the Wikitec page there it links to a bunch of different things as to whether you want to get started yes or to forge and it's somewhat organized now even inside things that need to get better we have two mailing lists all of our announcements about maintenance things like that more discussions which will both get renamed to club announcements but they're not yet renamed so here we go we have a blog where it's mostly just technical updates and we also have some things that go on to that we also have our blog and it's a fabricated blog which is some reason does not have a pretty URL but you know it's view file and we track all our bugs in future requests and so if you want to file a bug or a request then you can just go on to fabricated I link to the thing but it's the cloud services then it's just the cloud services and then it shows up when we're going somewhere and Brian will try it


So this event is hosted by the Kentucky Native Plant Society. We are a non-profit statewide organization that focuses on preservation, protection, and conservation of native plants and natural communities. So I'm just going to go right into Dr. Allen Weekly's introduction because we want to hear from Allen. So Dr. Allen Weekly, he is my, personally one of my favorite botanists in the country, in the world actually, prominent southeastern botanists. He's a plant taxonomist, community ecologist, and conservationist specializing in the southeastern United States flora. So I think what's really interesting about Allen is that he is so many different things. You know, he's a heritage botanist, taxonomist, a professor, you know, works with so many different groups across the state to push some of these conservation issues. And on top of all that, he's created an awesome flora of the southeast, which also includes the whole entire state of Kentucky. So I'm looking forward to Allen maybe letting us know how much of Kentucky really is in the southeast as a longstanding argument that we are below the Ohio River. So we, so Allen is the author, I said, of the flora of the southeastern United States. So if you haven't seen that, definitely check that out. Various floras of different states like the flora of Virginia, and wildflower books like wildflowers of the Atlantic southeast. He's released an app called FloraQuest, which I use frequently. It's much easier to use an app in the field than carry around large books. So that's been a great resource. He's done so many studies and mentored so many different people over the years. He's very active with the Flora of North American Project and the United States National Vegetation Classification. And he serves as an advisor to North Carolina Heritage Program and the North Carolina Plant Conservation Program. So very similar to a lot of the folks here today. He's also the co-founder of the Carolina Vegetation Survey, which we heard about from Devin's talk, starting to use that CDS methodology. And as a trustee and board member of the public and private conservation granting agencies, he has helped oversee $400 million of land conservation grants in the southeastern United States. So this is huge. Really land acquisition is kind of the ultimate conservation goal is, you know, protection of land. And then all of the other conservation strategies come after that. So without further ado, I'm really excited to hear from Dr. Alan Weekly. Well, thanks, Tara. I'm a little embarrassed by that introduction. I'm blushing, if you can tell. But anyway, it's great to be here with you all. And it's been great to hear the talks this morning and to also look around at the set of people who are attending and see so many friends and acquaintances and colleagues that I've worked with and so forth. So I will jump on in, share screen. Do you see a marshalia? Okay, great. Okay, so I'm gonna, my talk is sort of roughly divided into a couple of sections. And the first I'm gonna talk primarily about kind of the evolution of the southeastern flora. And that has important implications for us and understanding also how to go about conservation. And I'm going to talk about tools for identifying plants in the southeast. And sort of woven all through is going to be a basic conservation theme, which in my personal philosophy is the fundamental reason why I do all this. Conservation is really the foundation and the bottom line. So the vascular plants of the southeastern United States, there are about 6,900 native species, about 3,000 alien species, fortunately only a small percentage of those are invasive and problematic. And interestingly also a few hundred where we don't really know whether they're native or alien. So that's kind of an interesting set. But when we look at the southeastern United States, we see really high species diversity. This is a map from the bone app project showing areas of high species diversity. You can see these sort of scattered areas around the southeast, including parts of Kentucky, and also of course, California, the Sierra Nevadas and the coastal areas. When we jump up a taxonomic rank to the genus level, the map looks quite similar. But when we jump up one further rank to the family level, the map changes pretty substantially. And we see that the diversity at this higher taxonomic level is really heavily weighted towards the southeastern United States. And why is that? It's a really interesting thing that California beats us on species diversity, but we beat California on family diversity. What's going on there? So it's related to ancient lineages and resilience through deep time. If we look at plant narrow endemics, California, the Appalachians, parts of the coastal plain. If we look at tree diversity, you always hear the stuff about the Smokies having such great tree diversity, but nobody talks about the Red Hills of Alabama. It's just not fair. If we look at the group of plants called the Paleo-Dicots, that's like magnolias and water lilies and cerurus, lizard's tail, things like that, the southeast is also really, really rich. And this also has to do with its ancient status as a relic area holding temperate biodiversity, a major center of temperate biodiversity. Native grasses, you think, oh, that's going to be out in the Great Plains where all the diversity is, but no, the southeast is also rich, including parts of Kentucky. And I hope, I know it's the Native Plant Society, but I hope you'll let me slip a few animals in here as well. If we look at amphibian diversity or reptile diversity or fish diversity, we also see that the southeast is where it's at. And karst, as you all well know in Kentucky, karst diversity, the southeast is also really rich in diversity of underground organisms, cave and underground aquifer organisms. So when you sort of put this all together as nature serve has done into a map of biodiversity importance, the southeast is really a highlighted area. But I also want you to notice that it's not evenly distributed around the southeast. It's very patchy and every one of these little patches has its own distinct richness of biodiversity that's different from the other patches. So the Edwards Plateau or the Florida Panhandle or the Kentucky Bluegrass or the Southern Blue Ridge, all these have their own independent biodiversity. So how has the southeast developed and retained this biodiversity? I guess I'm going to suggest that it is edific diversity, that is geology and soils, times climate diversity, particularly rainfall and temperature, times fire regime diversity, and then times time, both a really deep time of hundreds of millions of years and also more recent time of the time since the last Ice Age. And so maybe sometimes we think of the southeast as kind of mostly looking like this, kind of boring really. Lots of different trees, not a lot of herbs, it's a nice forest to take a hike through, but sort of where is that biodiversity? Well, a lot of the biodiversity in the south is in the small patch communities, the specialized communities that harbor a lot of these relics, such as rock houses in the upper left or longleaf pine and other fire-maintained pine land communities across the southeast, or rock crop communities such as the granite flat rock in Georgia there on the lower left, or high elevation summit communities like Grandfather Mountain in North Carolina. And so we can kind of describe the underlying diversity that supports that biodiversity, that has created that biodiversity in the old-fashioned earth, air, fire, water kind of paradigm. But we also have to add in kind of the crazy clock of time that has sort of altered and shifted all this in really critical ways. So it's not that the southeast is static, the change in the southeast is both, it's a, how do we put it? It's not just a bug, it's also a feature. And so if we look at the diversity of physiographic provinces in the southeast, we have a lot of physiographic provinces. If we look at geology such as this map of Alabama, we have tremendous diversity in the coastal plain, but then also in Piedmont, Blue Ridge, and the sedimentary rock provinces that predominate in Kentucky. And we have a fair bit of elevational diversity, not like out in the west, but a fair bit, enough to matter. And we have a bit of precipitation diversity with some of some temperate rainforest areas in parts of the southeast. And we also have diversity of fire regimes.  But it doesn't mean that every bit of that landscape got that there were fire refuge areas that were sheltered from fire and so forth. So high fire frequency across the southeastern region basically is another driver of diversity because it means that there's a diversity of fire regimes in areas that have high fire regime in their most exposed sorts of settings like dry ridge tops in the Cumberlands and so forth. So it's also, I think, important for us to remember that the southeastern United States has been available for the occupancy of land plants since there were land plants constantly. It has never been under ice or entirely under water, et cetera. So it has been a refugium from ice ages from periods of high sea level and so forth. And this is an artist's rendition of Archeanthus. And I think many of you all will be able to recognize what this might be related to. It's actually a close relative of Tulip Tree. This is from 115 million years ago in the southeastern United States. So it's fun to be able to time travel back. And people always talk about inventing a time machine, but it's really disappointing that nobody has actually gotten around to doing it. I can think of about, I don't know, a thousand times in places that I wish I could take that time machine back to. So a reminder of Archeanthus' modern derivative. But more recently, of course, we've also gone through massive climate changes, the Pleistocene Ice Age and so forth, which shifted things around. And so I just want to emphasize that the climate has never been static. The southeast has gone through shifts. Those do result in extinctions, but they also result in evolution of new species. And so Kentucky 15,000 years ago maybe looked like this. So we have this permutational kind of combinatorial diversity of each of these different kind of factors and so forth. It means that each place on the earth is somewhat unique or is unique.  And she'd look around and say, oh, well, I must be in X County. That's how specific the biodiversity around the region is. So I want to go through a couple of kind of examples of these diverse areas. So the diversity in the southeast is local in space. Any field biologist for heritage programs and so forth who's gone about looking for for rare species will tell you they're not just sort of randomly scattered across the landscape. They're in these little small patches. And you walk through a lot of pretty boring woods sometimes to get to the small patch that has all the rare species. So the raisins and the chocolate chips in the cookie, not that the cookies bad itself. But diversity is also local in time. And this is I think a lot harder for us to think about the the important factor of time. And so the combination of those gives us this kind of crazy quilt of diversity in the southeastern United States. And I guess the one of the one of the messages I want to get across is this is sort of exemplified by this slide. So the red areas are really high areas of endemism, the orange kind of next highest and to the two shades of yellow. And Kentucky is in here and it has a good bit of that endemism and so forth. And as we have gone through climate changes in the past where cold and moisture climates have pushed down from the south, such as during places to see inglation, or when warm and dry conditions have pushed in from the west, or when more tropical conditions have pushed in from the south, they move in and then they move back out. The climate changes again back to sort of our more standard climate of where we are on the earth. But when those climates change, the species change too, they migrate. So we get cold boreal species that move south, or we get dry adapted grasses moving in from the great plains in the west, or we get tropical species moving up from the west indies in Mexico. And when the conditions change back, do all those species just dutifully trot home? No, they get scraped off in odd little micro habitats and so forth. And when they get scraped off in those places, if they stay there long enough, they evolve into new species. So the change is both a threat to biodiversity, but it's also a generator of biodiversity. So when we see these patterns, these patterns often reflect these relictual areas where biodiversity has been sort of scraped off and left behind and then has evolved further. So trillium discolor along the Savannah River, I think it grows pretty much exactly where it grew at the height of place to see inglation. It's mostly right along the river in the river valley. And there might have been spruce fir forest around it on the uplands, but there wasn't spruce fir forest where it was growing right along the Savannah River. So that relictual habitat retained that species. And then most species, the ones that aren't dispersed by ants, which is not a real good strategy for speed. Most species then were able to move back out across the landscape. So I'm going to give you a couple of examples. One is high elevation Rocky Summit habitats in Virginia, North Carolina, Tennessee, Georgia, and some of the endemic species we have there, like GM radiate of the mountain avans. Well, this is the kind of habitat it grows in. And this is what Tara would be doing if she was in North Carolina. But this is the distribution of the species. It grows only on about 15 different mountaintops. And it's sibling to two species, GM peckii. It's up in the White Mountains of New Hampshire and in Nova Scotia. And GM calfifolium in British Columbia, Alaska, out the allusions to Sakaline and Northern Japan. This is actually a picture of GM calfifolium and Hokkaido in Japan. And so these are sibling species, close relatives, but GM radiate, I'm having been left behind here in the Southern Appalachians has evolved into its own species. Tricophorum cespitosum, the deer hair, likewise growing almost always with GM radiatum, but it hasn't speciated. It's related to common populations all through Canada. So a very common species that is just disjunct down to the Southern Appalachians. Houstonia montana. Now this one occurs is even rare than the other two and occurs on even fewer mountaintops has this sort of bushy habit and brighter pink flowers than its sibling Houstonia perperea, a common southeastern species. So when we look at the some of the rare species in these high elevation outcrops, they have this strong signal of both new endemics and old endemics and unspeciated disjuncts like the tricophorum that are recruited from very old circumborial lineages kind of things that you generally that you would find in in the Arctic of Canada or Sweden or Russia or England, Rotodendron, Sibaldia, GM, etc. There's some really strong Asian affinities and that's related to the connection of North America, the closer connection of North America to Asia than to Europe, but there are also young species of warm temperate or even tropical kind of lineages that are have been long separated from Eurasia, Houstonia, Kalmia, etc. So it's not surprising the boreal aspect when you think that not that many tens of thousands of years ago, the high mountains of North Carolina and Virginia had tundra and had many of these northern species. So a second example, serpentine barons. These occur in western North Carolina and in a situation where topographic situation and elevation situation where you would expect to have cove forest or montane oak forests with white oak and northern red oak and so forth what you have instead is this. It's kind of cosmically weird. So the reason is the underlying substrate, the odd serpentine chemistry of the rock and the soils derived from the rock. And so in these serpentine barons we have these disjuncts of western and northern grasses like tufted hairgrass, slender wheatgrass, elemus trachea callus, spiroblous heteroleapus, prairie drop seed, disjuncts, mulenbergia glomerata, more northeastern poasulchuensis, a more northeastern species, genitianopsis crinida. Parnassia grandifolia, very weird scattered distribution around the southeast. This is currently under study and may actually turn out to be several species. There's some preliminary evidence that this is actually more than one species and will need to be broken up, split apart into several species. But we also have endemics like symbiotricum rianan, the buck creek aster that occurs only at a single serpentine baron. And at that same site, packera serpenticula just named in 2014. And now we're working on three additional endemic species to this one serpentine baron, a hexa stylus with very tightly closed mouth of the flower, clonal growth, not very much the variegation of the leaves with these long style extensions that are different than hexa stylus erypholia variety rupii or hexa stylus erypholia variety erypholia. And it also has a distinctive molecular signature, these two samples from buck creek showing some amino acid positions that differ from any of the other samples of that complex. So in these serpentine barons, we have a lot of western taxa that moved in in the past during hot and dry periods, got orphaned and left behind because of the odd geology at the buck creek serpentine baron. And some of them have evolved into neo endemics. And others are remain as just disjunct populations. Okay, closer to home for those of you, the Kentuckians among you. So these Cumberland plateau rock houses have also a very interesting flora. This one, Mononuria Cumberlandensis Cumberland sandwort, very narrow endemic here in the big south fork area, Tennessee and Kentucky. It's sister to the broadly Appalachian Mononuria glabra. By the way, for a species named only a couple of decades ago, this one has been entered into the genus frequent flyer program. It's been earning a lot of mileage. Having been originally named an air and area moved to Minnewartia moved to Mononuria and is about to be moved to porcidia. So yeah, frequent flyer miles. Stenanthium diffusum, narrow endemic also in the big south fork area, sister to the more widespread but sort of oddly scattered rock stenanthium griminium. The Silenis, Tony Romano mentioned one of these. So the rock house catch fly Silenia rotundifolia mostly restricted to these rock house kind of communities, not entirely so. So sister to Silenia regia and Silenia virginica. So to somewhat more widespread and common relatives, well the regia not really more common. And then Hucroparva flora, this gets into sort of new taxonomic research and also using traditional and new methods like molecular techniques. So Ryan Folk studied this a few years ago and what had been considered to be a single species widespread from North Carolina out to the Ozarks. He split into three species, one with two varieties and in Kentucky you all have Hucroparva flora, variety parva flora and also Hucro-Mazuriensis. So Mazuriensis here is with the triangle Southern Indiana and Illinois and this area of Kentucky. And then we have Hucroparva flora, variety parva flora more widespread through the Cumberland Plateau area but then disjunct over into North Carolina, a separate variety endemic to the Piedmont and a separate species out in the Ozarks. Geography is destiny. When plants or animals get stranded somewhere they change and they evolve and become new species. Another critter, sorry I can't resist letting them in as well, but I want to make a particular point about this. If we look at the distribution of Hucroparva flora, Cumberland Plateau here and then disjunct over into the Blue Ridge escarpment along the North Carolina, South Carolina, Georgia tri-corner and the distribution of the green salamander Aeneides Aeneas, Cumberland Plateau disjunct over into that same area. It should give us pause, it should make us think a little bit. Why would a salamander and a saxophrage or a crag jangle, an alum root, have the same distribution?  And so the green salamander is actually in the process of being split up into multiple species. This paper came out just a couple of years ago last year, 2019, which separated the hickory nut Gorge salamander. The botanists among us will appreciate the name Aeneides caria ensis as a new endemic and it is this branch in the molecular tree and there are three other major clades, a southern Cumberland, a northern Cumberland and an escarpment clade. And so as the authors wrote, Aeneides Aeneas has been petitioned for listing under the Endangered Species Act and our study highlights the need for conservation management of this complex. Our formal recognition of the extent of genetic and evolutionary diversification of the complex is a critical step in establishing conservation strategies. In these rock houses we also have things like Hupersia paraffila, sibling to the Circumboreal Hupersia Cilego complex. And then we have the Filmy ferns.  Why do we have tropical ferns in Kentucky? There's something wrong here. So Vandenbashi Abashiana, Didama Glossum, Peterzii, doesn't make it to Kentucky. Hymenophilum tailoriii, somebody needs to find that. A lot of these Filmy ferns are mainly found by bryologists because they they look a lot like liverworts and mosses and grow in the same kinds of habitats. And crepidomines and trichatum grotto felt. So these Filmy ferns, these three, four species of Filmy fern are in very different lineages of Filmy fern that have been separate for over a hundred million years. Now we don't really know when they arrived in the Southern Appalachians and when they speciated there, but they appear to be old tropical relics.  East Asia, Eastern North America and Northwestern North America. That's a pattern that we see over and over again. So the Lamshade spiders. So a recent molecular study of Lamshade spiders found what they described as fractal genetic structuring. Essentially every rock outcrop had its own genetically distinct population of Lamshade spiders because they can't move from one rock outcrop to another. And so they've been there on each individual rock outcrop isolated and evolving on their own. So they recommended that there will be some splitting in the Lamshade spiders as well. Hypokylus, Pococci, and all will get split up. So these rock houses, they have these ancient tropical lineages of ferns. They have northern cryophils, cold adapted things, spiders and club mosses and all, warm temperate recent derivatives. And I think all this is because they are so buffered from extremes. They basically have this kind of near constant climate where they don't really freeze. They don't really get very hot. They stay constantly moist and the moisture further buffers the temperature. So I think all this has implications for how we think about conservation. I won't read this whole quote, but Alan Graham in talking about past ecosystem dynamics, he said, what we are actually additionally doing is setting aside way stations. We're, even if biodiversity changes in a particular place, those places that have held on to biodiversity in the past basically show a resilience and a hope for the future and holding into biodiversity currently in our sixth extinction. So we have areas like the Florida Panhandle sticky spot. So here we have the Apalachicola River bluffs with things like taxis and taurea and sort of ancient circumborial lineages. We have the Apalachicola National Forest savannas with all these tropical and subtropical related taxa that have been written there. We have this area in Washington and Bay counties with all these are different habitats, moist bluffs, saturated savannas. These are mainly sand hills, really dry habitats and also seasonally flooded lime sink ponds. And yet they're all jammed in together here. So this suggests that this part of the Panhandle of Florida is this ancient relictual area that has been able to hold on to lots of different kinds of biodiversity through time and through change. If we want to bet where we're going to be able to conserve biodiversity into the future, it helps to pay attention to where biodiversity has been conserved in the past. So I'm at an age where I get mailings from my 401k and so forth, and they always have this sort of phrase, past performance is not an indicator of future results. Just because this stock or this mutual fund gained 20% last year doesn't mean it's going to gain 20% next year. But I'd like to suggest that in biodiversity conservation, past performance is an indicator of future results because it shows where the landscape has a kind of resilience because of its ability to withstand climate change and retain and generate new biodiversity. And I'll close this section of the talk with this quote from Charles Darwin, there is a grandeur in this view of life with its several powers having been originally breathed into a few forms or into one and that whilst this planet has gone cycling on according to the fixed law of gravity from so simple a beginning endless forms most beautiful and most wonderful have been and are being evolved. So I'm going to move into the second section of my talk, which is really about kind of flora and describing new diversity and so forth. And I'm going to start with the idea that you can't conserve what you don't know about or if you manage to conserve what you don't know about, you did it by accident and we can't really, we don't really want to leave conservation to accident. And so taxonomic science is a foundation for conservation and we've been able to persuade funders at UNC of that to fund taxonomic science because of its importance for conservation. And so in the talks this morning we talked about species as conservation targets and also communities and that sort of fine filter course filter dual approach I think gives us the best opportunity to identify the most important conservation areas and work to conserve them. So species are drivers of conservation action where where species are and so forth. And I think it's important in conservation that we avoid both type one or type two errors. Many of you probably know that language, some of you maybe don't, but but a type one versus a type two error and the type one error, do I remember which is which? Anyway, it's if we think something is really rare and we try to do conservation there but it turns out that it wasn't a good species, we sort of wasted resources on it. Or if we didn't know that something existed, we hadn't described it, we didn't know that there was a rare species there or we hadn't done the heritage style inventory to find that occurrence and it gets destroyed without us even knowing about it that's a lost opportunity. So you know sort of wasted effort versus lost opportunity with conservation resources being as precious as they are, as limited as they are, we can't afford either. And so a lot of what I do is oriented towards trying to help us not make either of those kinds of errors. So Jenkins in 2015 based on vertebrate animals and on a few tree species basically identified these priority areas where a lot of conservation was not, a lot of biodiversity was not conserved. And you see the same theme of the west coast but then very broadly the southeast and all. So if you look at plant species that have been named over the last decades in the southeastern United States, about 500 species have been named since 1970. And that's about 10% of the native flora. And when I tell people this they're often kind of shocked it's like oh gosh you know we've been doing taxonomy in the eastern United States for centuries. Haven't we gotten it all figured out aren't we asymptoting towards the end don't we know everything there is to know. And the sort of both frustrating and exciting thing is no we don't there's a lot of new stuff out there there's a lot of stuff that hasn't been described. And most of the new species that are getting described are narrow endemics they're in heritage lingo G1 G2 G3 species they're the highly most highly imperiled highly imperiled and and imperiled species the species that are most at risk of extinction. And if we look across the southeast as a whole at the G1 G3 species across the region something like 50% of the of the these most highly imperiled species have been named in the last 50 years. So you know we got white oak right and we got other things named a couple hundred years ago but we're still finding a lot of these new things and getting them described. And so no we're not going towards being done in fact the number of new species being named by decade is sloping up and part of that is due to new techniques like molecular systematics but mainly it's due to biologists getting out and getting to these specialized habitats and looking with the recognition that there might be a new species there. So you can't find what you don't believe is there so part of it is believing. So there was mentioned earlier I think by Tara about trillium pustulum and and so forth the trillium pustulum complex is in the process of being split up each of these colors will be its own species most likely some of the work is still being done Trillium Georgianum here just a couple of remnant sites remaining in North Georgia is has been named some of these others are in the process of being named Ozarkanum and Texanum in the West have been named before. Tulip trees something as familiar as tulip trees Carl Federer my grad student did molecular sampling of this we think there will be three taxa probably one at new species down in Florida and then two varieties named for the rest of the southeast a new highly imperiled blue curl tricostominesophilum we just got described last year it occurs on on the main capes in northern South Carolina Cape Hatteras Cape Fear Cape Lookout Cape Fear and Cape Romaine and in Kentucky there's been a steady stream of new species being named through time this is about the last 40 years of of new species name that occur in Kentucky and we're not done I'm going to probably move past this slide quickly before anybody complains that their favorite new species is and on here or whatever but lots of new things coming up that are being worked on and will get described and once they are described many of these will be narrow endemics and they will shape the conservation agenda in Kentucky and beyond but I don't want to focus just on naming new species there's also the what I call taxonomic resurrections and changes in status and this is one that was mentioned earlier and I threw this slide in while I was listening so Baptisia aberrans the the aberrant southern Baptist blue wild indigo has been lumped often into Baptisia australis or lumped into Baptisia minor and as such it doesn't have that high a conservation status when you look at its G rank but when you recognize that it is actually its own separate species its G rank becomes you know indicates a much more highly imperiled status and a higher conservation priority for Kentuckians and other states that are looking at this species so now I want to talk about tools for identifying plants for understanding the flora and you know I said earlier you can't conserve what you don't know exists but I guess I also want to emphasize that you can't conserve what you can't identify and scientific papers describing new species you know the ones that pertain to the southeastern United States I think in the bibliography of my flora there's about 6500 references so that isn't very practical people can't go to 6500 scientific papers many of which are paywalled by the scientific publishers anyway so so what you need is flora's field guides apps etc things to help you identify plants so um this is a picture of me when I was five and some of you all will recognize the plant that I'm playing as a trumpet it's a highly hallucinogenic and toxic plant my parents knew a lot about plants and they really shouldn't probably have been letting me do this but they did some people might say that it explains a lot but but I grew up in a family that that spent a lot of time outdoors and we always had Peterson field guides and I first started identifying plants while I was taught them by my grandmother and my parents and my aunt and all but I first started identifying them using a Peterson field guide and then by the time I was in my teens I started using the flora of West Virginia I grew up in Virginia but I spent a lot of time without electricity on some land that my aunt owned up in Winchester County Frederick County near Winchester Virginia and she had a copy of the flora of West Virginia that the original edition of it are the original printings and so flora's and field guides and apps nowadays apps are where sort of the rubber meets the road in terms of of enabling people to access information about the plants around them you don't access it by going to the herbarium you don't access it by reading those six thousand scientific papers you access it through some that synthesizes that information and so you know late in my teens I started working with the flora of the carolinas at the time it was sort of brand new and you know the flora of North America is 30 volumes that's not very practical increasingly there are apps and wonderful wildflower guides like this one for Georgia done by Linda Chaffin so when we wrote the flora of Virginia it was the first new flora for the state in 250 years here's the old flora of Virginia and here's the new one and John Clayton came and visited when we when we had our our party for the new flora but what I found is that what everyone really wants is a tricorder they don't want a flora they want they want a magical device that they can point at a plant and it'll tell them what it is and you know we're sort of working on that and and there've been there were various mentions of iNaturalist I'll just talk I won't talk a lot about iNaturalist but I'll mention it but iNaturalist with its artificial intelligence machine learning approach to identifying plants you know is sort of taking us in the direction of the tricorder but it'll never really get there with a lot of the species that we have to identify it does a pretty amazing job on on a lot of species so I want to talk about sort of floras and apps and so forth and and introduce you to some of these so um so I've been working on these for a number for about 30 years now well last uh 18 at UNC and we came up with a way of of developing these kinds of tools basically by creating a database from which the tools are derived rather than writing a flora by typing it in using microsoft word we have information in a database that then we can export in various forms and formats so that's called the flora manager and from it has come the flora of virginia the flora of virginia app the flora quest app the wildflowers of the atlantic southeast wildflower guide and and soon we will have the a sort of the even newer southeastern flora which i'll talk about in a few minutes so this is kind of what the entries some of the entry screens look like it's just in access right now we'll probably put it in in an sql format soon but basically these are sort of the entry screens and so forth that you can tap on you get this for you grow part of a flora so you can enter in habitat and distribution and comments and phenology and this is the synonymy and you can click here to view or edit the distribution here's the keys that lead to it so you can access the key directly from this screen the entry screen for distribution looks like this you can tap in different places in each of these boxes and indicate whether it's common uncommon rare whether it's native or alien or uncertain and all and also record information about its endemism and so forth and and then there's basically this is how you output information so you can create a derivative flora by choosing your geography just choosing your taxonomy choosing what maps you want to use choosing whether you want to insert photos into the into the format what sections you want to include etc and optional fields that you can include like Arkansas remarks or Shenandoah national park comments so that's sort of the basis for all these things so the flora of virginia app we developed a couple of years ago with my colleagues in virginia at the virginia national heritage program and it has a number of great features one of which particularly I want to introduce to those of you aren't familiar with it and that's the graphic key so it has dichotomous keys it has the standard kind of dichotomous keys but it also has this graphic key it has lots of illustrations it enables you to do geographic filtering by where you are in the state of virginia if you enter in your counter county it will basically filter the results for species that you're likely to encounter in that area so if you were working in southeastern Kentucky you could use this app and enter in lee county virginia the southwestern most county of virginia and it would work great it has essentially all the content of the flora virginia book but you have it on your phone and it's a pretty expensive app it is available in in both platforms and a major upgrade is about to be result released in the next couple of weeks and if you already have the app it'll basically just refresh with that new content so the graphic key is basically a an elaboration of an idea that first came up about 30 or 40 years ago and people did it using punch cards and holes punched in them and so forth but it's basically a polyclave where rather than a dichotomous key where you get asked some question about some feature that you don't have your plant is in flower and it wants to know what the fruit type is or your plant is in fruit and it wants to know what the flower color is this kind of key basically lets you enter in whatever information you have on the plant in your hand and each time you enter in information it subsets the possibilities so in virginia 3164 species if you enter in that it's a broadleaved woody plant you're down to 474 already almost anybody can do that right you don't have to have a lot of sophisticated terminology or knowledge to do it and then if you keep on entering things like the leaves are opposite they're simple they're palmately veined lobed and toothed you're down to 10 species and then you click show and it shows you maples and viburnums and you can make a choice based on the possibilities I mean maybe if you enter in enough information you'll get down to one species it depends but you may end up with a number that you have to consider and once you've selected what you think might be right you can look at photographs you can look at description and habitat you can look at range map you can see the breadcrumbs of how you got there what what choices you made that got you to that point so a great app so we also decided to do a wildflower guide we did this with timber press eye and colleagues at the garden it covers this area so not Kentucky but it'll work well in eastern Kentucky so it covers eight states and it also has this kind of simplified approach to kind of keying the plant rather than just in Peterson's wildflower guide you know if you had a plant with yellow flowers there were like 131 pages of yellow flowers that you had to flip through right so this gets you down to just a few pages by entering in basic information on flower color flower symmetry number of petals and some simple leaf type questions but then you get to a page like this that that where you can sort through and compare the photos so the big flora the flora of the southeastern United States so back in October we dropped 29 floras onto our website they're freely downloadable there's the full flora there's five regional floras and there are also individual state floras for all of the states so this is the area covered and Tara asked me to explain what part of Kentucky is in the southeast and in my opinion all of Kentucky is in the southeast so there's a pretty major floristic boundary that is essentially at the glacial boundary so north of the glacial boundary you have all this country that was flattened by the by glaciation the drainage was deranged it has bogs and so forth and so on and in the unglaciated interior low plateau and southern Appalachians and so forth you basically have this these relictual residues of species that were that were not ridden over by ice so so this is how I define the southeast so these are the little maps that are included that contain a lot of information about the distribution and also the distribution outside of the area these arrows show you where the plant occurs outside of the map or Ian means that it's endemic to the area on the map um so 10 000 species here's the download site 1848 pages we've had over 3 000 downloads since late october there are the five physiographic regions sub region floras and the 22 state floors so if you're going to work in Kentucky you can download the Kentucky flora and it will work pretty well it's not super customized these are where the downloads have come from Kentucky is doing nicely I like that little yellow dot in Kentucky and the flora contains scientific and common names detailed habitat description phenology distribution the little map comments detailed synonymy and hyperlinks to the nature serve global ranks the rarity ranks and to show you sort of how the keys work so when we produce all these different floras basically the keys get automatically simplified so here's the key to magnolia for Kentucky and the key to magnolia for Louisiana and you can tell by the key numbers here that something is missing um so each well Louisiana has five magnolias and Kentucky has six not too much overlap between which ones are present in each state but you end up with a functional key that excludes all the species that aren't in the state so the basically the computer automatically does that the programming so we're currently working on derivative floras um that uh will involve sort of not just sub-setting but then adding back additional information so we're working on a a flora of Arkansas with Theo Whitzel and colleagues at the Arkansas nature preserves commission we're working on a flora of Delaware funded by the mount cuba center um with bill McAvoy at the Delaware natural heritage program we're working on a flora of georgia with max medley and the georgia natural heritage program funded by them we're also working on a couple of floras and flora yulets with the national park service cumberland piedmont network a little river canyon flora and a guide to the ferns and leica fights of the cumberland piedmont network so that would include mammoth cave and so forth and this will be with illustrations this is a sort of a more public facing um flora so um i want to announce though that uh just uh about a month ago i got generous funding from a conservation philanthropist that will support basically taking this whole project to the next level and we'll be reworking all the keys um and optimizing them um making them easier to use for a broad range of users um we'll be compiling five to ten photos for every species in the flora including diagnostic feature photos we'll be adding in a additional field into the flora on field recognition advice for families genera and species how do you tell a holly from a rose shrub and expanding the kind of idiosyncratic comments about history and edibility and field survey hints that i think makes it less dry of a flora and more approachable to a broad set of amateur users um we'll develop the graphic keys and to do that we'll have to develop a database that will enable the computer to do that kind of sorting um and the plan is that we'll release five probably five sub regional apps that'll be similar to the current flora virginia app but with some additional features even to it and that will cover kentucky and the entire southeast um so um those will be on ios and um and android um one of the limitations of the floor quest app has been that it's um on um on apple devices only um and we're going to be putting in additional things like c of c values improved g ranks um and doing this with an expanded regional set of contributing experts it's really going to i think be a pretty amazing resource for for botanists and conservationists in the region so tara called out that i was going to say something about facebook so i threw this in so here's a few favorite facebook sites for plants uh obviously for you all the kentucky native plant society um i have a facebook page or group weekly's flora of the southeastern united states that i use for posting updates about the flora new species that get described um kind of fun um plants that i see and so forth um the southeastern grasslands initiative has a couple of different facebook groups but one that's more plant focused is the ecology history and biodiversity page and then two um sort of regional pages that were created by edwin bridges are the southern apple action flora and ecosystematics and flora to flora and ecosystematics it's really more than florida it's really a coastal plain southeastern coastal plain but these two each um well 1.9 000 almost 2 000 members of southern apple action uh 5400 members um here they a lot of really great information gets posted on these um sites about that you can learn a tremendous amount from by just going to the site occasionally and and going through things and then i was in i was inspired by tony romano's presentation this morning to just briefly mention also um we've had kind of effort as well in north carolina about finding piedmont roadside native plant uh sites um and this is a an i naturalist page that was set up by uh julie tuttle with 10 000 observations now and 164 people contributing to it um of these um these relictual roadside biodiversity gem sites that are supporting a lot of the sun loving fire historically fire maintained but now mowing maintained um plants in our in our southeastern region um and all so i want to close uh with uh kind of uh what i hope are inspirational remarks sometimes it's easy to get discouraged in the conservation business but uh these are fish and wildlife service and north carolina wildlife resource resource commission biologists searching for bog turtles on a very very cold uh december day um and the dedication is uh just amazing um but we need to train um the new generation of um magi botanists and magi ecologists and magi zoologists um to go out there and conserve the really magical flora and fauna of our region our southeastern region and uh a north carolina centric photo here but heritage and state agency federal agency this is gary kaufman who works for the forest service what i call a combat biologist position um people who work in agencies uh that are of multi purposes some of which are not conservation oriented um justin robinson a contractor for natural heritage we've been remiss in broadening the scope of our human diversity uh supporting biological diversity um in the past and uh and need to work on that and um just to the final closing is kind of a couple of word pairs um and uh and all economy and ecology they both come from the same root oikos the greek oikos meaning home and um um there's often been a tendency to view economy and ecology as um as in opposition to one another and i think what we need to work towards is a realization that economy and ecology have to work together um in the long range and sustainability and so forth we're never going to have a good economy if we um if we destroy the house the oikos that we live in um and all uh they have to work together conservation and conservative that's also kind of a word pair that um has come to sort of be viewed as in opposition to one another um and um it's important for us i think to remember that a lot of great conservation innovations legislation in the past um came from republican administrations uh teddy roosevelt um a lot of great conservation legislation was passed under uh richard nixon um and all and we need to reunify that conservation is conservative uh they mean the same thing really um and i guess i want to highlight the you know the the legacy of land um that uh many of us who who grew up and live in the southeast we appreciate the land for what it is we have connections to the land uh the the younger generation less so um my parents grew up on farms um i grew up in the woods all the time i spent about a month out of every year with no electricity um at at our camp up in the mountains um you know without electricity without screens it's almost impossible for most of us to imagine nowadays um but um that connection to our natural heritage and the legacy of land is something that um you know i think it is a greater challenge than ever to to convey to a broader set of people and in the long run you know our i think our conservation agenda will only succeed if we do uh if we do broaden the set of supporters and participants through citizen science through um through politics through um understanding the the science and having access to tools for identifying and learning about uh what's around us um so i guess that's my um my sort of hope for the future so curing plant blindness or plant attention deficiency uh building uh botanical capacity i've always loved this quote from baba de um from the 1960s in the end we will conserve only what we love we will love only what we understand and we will understand only what we are taught so i'll close with that thank you alan that was awesome um there's been lots of comments and a few questions um but in i guess let's see we'll take just a few we're kind of running a tiny bit over um but i'll ask a few to you alan and then the rest maybe during um uh the next talk uh you can uh reply to some of these yeah and i can also i can also stay i don't know whether things are gonna close sharp at the closing time but i can stay on a little bit after okay great great um yeah there's just lots of kudos um let's see you addressed some of these already uh like the i the ios um in the flora quest you know in 2016 the year your flora quest app was the reason why i switched from android to iphone and it's just been a downward spiral ever since um let's see uh i'm going through um i there has been talk of um you know will there be call for photos how are you gonna incorporate some of the photos how can we help as a native plant society um you know for kentucky information um well that's a great question um the we really just got this funding um like a month ago and um i'm beginning to organize things about that um we we will definitely be putting out calls for photos um we've also found um on some other projects and fio out in arkansas has that i naturalist is a great source of sort of prospecting for photos and then contacting people and asking for permission you know where you can look for the sort of the right photo that shows the leaf arrangement or shows the aspect of the plant or the the plant and fruit or some diagnostic feature um but yeah we will we will be looking for we we've compiled about 50 000 photos of the flora of the southeast but we're going to be looking for another 50 or a hundred thousand uh probably uh to to add that i think we'll also be putting all this up on a website as well as in the app so it'll be sort of um once again sort of multi-purpose um so um i will uh be back in touch with uh the the Kentucky Native Plant Society folks and uh when we have more specific kind of uh requests and and uh have organized how we can receive things so awesome great