So, our next speaker is Devani Wu. She is an active member of the Elixir community for a long time. I remember personally, Singer speaking at Elixir London 2017, and then Elixir Conf 2019, and now here at FOSDM 2020. So, she has also a number of open source projects in Elixir. And, yes, she's going to talk about processes in grain, so let's give it up for Devani. Thank you. So, thank you very much. I think I talked too much. So, today I'll try to keep it very short. So, for myself, I basically write programs, and sometimes it breaks, so I fix them. So, I botch them. And that's my Twitter account, which you could use to, you know, to discuss certain things with me if needed. So, today's topic is about, it's about Orleans and especially within the context of Elixir applications. So, what exactly is Orleans? I believe if the framing of the Orleans paper is correct, then it starts with a problem brewing inside of Microsoft, where it's really related to management of state that doesn't fit on a single machine. So, usually when you write applications, you start with one server, and that server is a database server. So, you have a single server, it's a pet. Then, well, state history in the database. But then, you may end up with something like this, where one server is no longer enough because it gets overloaded. So, now you have one database server, and many stateless application servers. In this case, the state is still stored on the database. But now, in order for this state to be processed and brought to the end user, you now have to have the data travels through two tiers of networking, first from the database to the actual server, which doesn't work. Results and sent to the customer. So, there's basically no consistency. So, basically, sorry, I pressed two times. So, basically in this situation, you end up with a lot of network traffic, and sometimes you still get overloaded in this regard. So, usually, you will add some kind of cache server in this case, where since the database is so overloaded, you add some replica nodes, and they might still not be enough. So, you add a cache server or multiple cache nodes, and you end up with no consistency because now you have to handle cache invalidation. This is a problem that plagues the web community, and has plagues the web community since the beginning of time. In Rails, for example, old cache objects in Rails usually contain some kind of cache key. The cache key is a hash or just a timestamp of the object. So, it was actually revised, I think, in a more recent version of Rails, where instead of doing this, which pollutes the cache because if an object is modified, the old version that has been cached doesn't go away. A new version isn't generated and put into the cache. Now, in the cache, you have two objects, one old, one new, and the old one isn't used anymore. So, you introduce a lot of cache pressure, and so on. I think in the most recent version of Rails, it has been updated to a point where the cache key only contains the object's identity, but then within it, there is some kind of hierarchical structure. So, if the object needs to be updated, the key is reduced, but the content of that key or the content that's linked with that key is updated in this manner. Anyway, there's a lot of management you have to do in this kind of setup, because now you have two places that potentially store state for you. So, you have to consult a cache. If it's not in cache, consult a database, and remember to update the cache. Remember to not update the cache when the database has been updated again, and so on, and so on. There is no consistency, and it's horrible. So, yeah. Basically, two issues.  So, you have infrastructure sprawl. This is what happens, and you have invalidation issues. So, at least that's, that was the framing of the paper, where, since this was the issue, a stainless middle tier doesn't provide data locality. You have to move data from one place to another. So, there's a lot of transportation going on for that data. So, they look at the actor model, and the actor model looks appealing, because you can write a function and put the function anywhere to run the function local to the data. And, well, in a more modern way of saying, it's basically data gravity, but in Microsoft speak, it's basically this. So, they continue on to say that actor models, such as Erlang and Alka, burden developers with many distributed system complexities. The key challenges aren't that you need to manage the life cycle of actors, because you have to create them in one place. You have to communicate with them, and when you're done with them, you have to tell them to go away. Otherwise, they will stay around, and so on, and so on. So, the developer must be a distributed systems expert, unless if audience comes to play, that's essentially what their paper was about. So, their result was basically, each actor is represented as some kind of grain. You can implement code for a grain, like you implement any kind of object in .NET, and the grain has four states, where persistent means that there is only information in some kind of data service, like a database, and there is actually no computation going on, there's no process running in your server, but you can address anything, address a virtual actor by the name. Let's say, for example, a customer pays, the customer pays an invoice, in this case, you have actually two actors, one is a customer, another one is an invoice. So, when the customer pays an invoice, you call upon the virtual actor representing the customer, and you call upon the virtual actor representing the invoice, they will each refer on a database, do the right thing, and then you tell the customer, you are paying this invoice. That's essentially how the virtual actor works, and then you don't have to worry about stuff such as, okay, so now I'm done with the customer, I close down the customer actor, and I close down the invoice actor. Essentially, the Orleans runtime does it for you. So, that's why you have these four states. So, with seeing the construct of Orleans, a grain basically has an identity, which is an identifier, email address, or the customer invoice, number, referring invoice, or so on. There is some kind of behavior, which is linked with that kind of grain, which is, well, in Donetsk, it's basically your class, and in Erlong or Elixir, that would be similar to how you always use GenServers, but in many GenServer implementations, you always do use GenServer, so you comply to GenServer behavior, but you actually implement different callback modules. So, that's the behavior, and the state obviously is state. Well, and obviously, Orleans being Orleans, if you look up on the web today and look for its documentation, it'll tell you, okay, this is how you deploy. So, well, wait a second here. So, all your grains basically represent single-reader processes. One, count two, they can be configured to allow re-entrant calls, which you may or may not want, but it's going to be imported later on. So, they are scheduled cooperatively, so unlike Elixir or Erlong, it's never preempted. So, if any grain takes a long time whizzing Orleans, then everything gets low, or they will block, and the justification provided in the paper was that this was actually better for discipline teams, because it provides better predictability and multitasking, but you have to keep in mind the context. Now, the good part. So, these virtual actors are automatically activated when there are messages. If there are no messages, they are not activated, and also when they're done, they stick around for a bit, and then they go away automatically. So, that's a good part, because in Elixir or Erlong applications, well, you've got supervisors, if you want to use them, you've got distributed supervisors built upon like lusp or swarm or whore, if you want to use them, but you have to create your own processes yourself, and you have to name yourself, and you have to tell them to go away on your own. So, Erlong hosts or silos in this regard, they are responsible for hosting grains, so you can't just take a grain and just run it out of there, because in the context of Orleans, there is no such thing as a scheduler already implemented. So, they went on and implemented a scheduler, which is responsible for executing work that these grains are asked to do, and there was no out-of-the-box clustering, distribution, and so on. So, they implemented this part as well, and there's a global way of scanning the whole world and finding out where the grains are, if you want to do a listing, but the paper also acknowledged this is kind of expensive to do. Well, so, obviously, this is what happens whenever you have something like that. So, Orleans basically resembles an Erlong paradigm. So, obviously, somebody has ported it back to Erlong. So, basically, a fellow Mr. Slalter actually ported it to Erlong a few years ago, and I think he might be here at the event. I don't know where he is, but I think he's based here. So, it's called Erlings, which is basically Erlong Orleans, and the premise is basically, okay, so each grain is basically single-threaded, so it's one process. So, each grain is one process. Process registration in Erlong or Elixir, well, you can use PG2, Process Group 2, which is the updated version of Process Group that's in the core Erlong distribution, or Core Erlong bundle is part of the kernel, or you can use Swarm or Horror to do process registration, but in this case he chose to use LUSP for it, because LUSP, well, it's more modern, and it does provide various scalability, and it works with partisan if you want to. There is another point, which is grains are either stateful, meaning they have state. If you work with it, it has some kind of state, and when it gets activated, the state actually is stored elsewhere, so when it's activated again, that stored state comes back up and is hydrated and put into that grain again. So, Erlings has an ETS-backed or Erlong-turned-stored back provider, which does it for you, it's out of the box available, you can use that, or you can choose not to use that, and obviously grains are implemented in a way that you implement callback modules, so it's very similar to GenServer. So, yeah, and last thing, if you don't want to use Erlings, but you have a similar problem, for example, you already have a system in place running Elixir or running Erlong, that already has a lot of processes, you just want to, you know, they can go away when you're done with them, then use a state machine, or more precisely, use the Bondo-Mealy machine from OTP, the GenStateM behavior. GenStateM is provided as part of the OTP distribution, and it allows you to do something like, okay, transition my machine from this state into that state, but add a timeout, invalidate the timeout if I get any message, otherwise, continue, and this allows you to very easily build something that goes away automatically if not bothered, but sticks around to process more events without having to use a framework. Yeah, so I figured that it would be better for me to show you some examples using Erlings, and my chosen example is this. This is Converse Game of Life, and if you look at the Game of Life, you can find some interesting things. So basically, it kind of resembles nature. Each block is a cell, and there are rules around when cells become alive, when cells become dead, and when cells remain alive or remain dead. So obviously, the implementation here is that, okay, so each cell is a grain, and the grains, either talk with each other, so each grain talks with its neighbor to find out whether it should die or it should become alive, or alternatively, each grain can query a database, and the database contains state as to whether grains should be alive or dead, and you start with a randomized state, and you end up with something like this. So this is just a video, but I have some live versions here, I think. So, okay, so I'm just going to start an interactive elixir console here, and this version contains the version where grains ask each other about what's going on. So I put in some kind of helper function, so it's easier to try. Okay, so I'll explain what all you have just seen. Basically, when I started the helper function, what it did was, okay, I want to first find a reference to a particular grain, or in this case the big grain, the game grain. The game grain is in charge of creating the game and maintaining the game, and more importantly, making sure the cells are updated on precise ticks, and not just randomly or just whenever they want to. So find a game grain, tell it, start building your cells, and when you're ready, because it's a call, so the call will block, when you're ready, start a timer, and this is because although the game of life specifies how the cells will come alive, become dead, or remain alive or remain dead, it doesn't do something like, it wouldn't specify how you should implement it, and if you have a lot of cells, and they can all make decisions that affect not their own states, but also future states and the states of other neighbors, you want to make sure they make these decisions in an orchestrated manner. So in this implementation in particular, each cell can be told to tell its neighbors about its own states. So this is the first thing the cells can be told about to do, and the second thing the cells can be told about to do was to, according to how many neighbors have told it that the neighbors were alive, figure out whether it should die or not. So this is how this version works. Yeah, so it also has modules, this is basically the callback module implementing the early grains, early grain behavior, specify there is a provider for saving state, the provider has a short name called in memory, this is configured elsewhere, and the placement of the grain is preferably on the local host, this is because again in our lanes as in Orleans, the placement of the grain can be locally or anywhere in the cluster. Depending on, also you choose that as a developer, but also you choose that depending on whether the grain should be stateless or not, because in grind designer Orleans you can actually use grain steps workers, although in elixir application you probably want to use gen stage or something like that, which is higher level and closer to the spirit of elixir. So when the sale starts it's going to gain my ID, and an ID, this ID actually is basically the coordinate, and there are some management functions, but the sales can be told to prepare or to commit. In preparation it counts its neighbors, and according to the current status determine whether the upcoming status is alive or dead, and in commit it updates the future state, it grabs the upcoming status, put it into current status, and clears upcoming status. So as to the neighbors, basically each sale is started with eight possible neighbor fields, this is the state held by the sale, so the game grain actually makes these sales and sets the neighbors, so the type spec is incorrect, because actually there are eight possible neighbors and I just haven't updated the type spec. Anyway, in this in this version it just basically the sale asks each neighboring sale, but there is a problem with this implementation, which is that the game grain can tell all the sales to update, but in this case the game grain will be making calls to each sale, and if the game grain were to make a call to that sale, that sale grain, then that sale grain is going to be busy, and it wouldn't be able to answer any other calls. So in this implementation the update is basically singles-rided, because you cannot have one sale answering anything else when it's already answering something else, and yeah, so it's not very good. Alternatively, and also in a in a version that's more closer to the spirit of okay, to the spirit of Orleans, there's another version, which is also simpler, it's got fewer fewer files, and I shall show you this one. So in this case again, I first basically start everything up, and it works, but internally it's entirely different, and I'll explain. In this version, when I start the game, I actually make a game entity, and I store it in a data store, then I grab the grain representing the game, and I tell it, the idea of the game is baking to identity, and I tell it, start, and when a game grain starts, well it just starts the timer, it doesn't do anything else, it doesn't do any kind of perforation, why? Because that logic is in your sales, but more importantly, when the game grain wants to update on the timer tick, it asynchronously calls all the grains at the same time to first pre-create and then commit, and these sales grains actually do not call each other anymore. Instead, what they do is, well first when it starts up it loads from a database, but what they do when they want to count the neighbors is by doing repo queries. So, I have eight possible neighbors, and in this case the game is actually run on the torus, so the top of the edge is basically linked with the bottom of the edge, so there are no boundaries in this version. So the game just basically looks at the database and determines how it should behave, so in this case there is no more communication dependency between the sales, and so you can run this program in very high parallel, basically of very high levels of parallelism, where I think all cores available are being packed at the moment. Yeah, so all the cores are being used. I think this is the most inefficient implementation of the game of the long you will ever see. Basically, like that, so in this case it's like 20, 25, 30, 30 by 30, something like that, so close to a dozen things. Obviously, this implementation imitates nature, and a better implementation wouldn't have done it like that. It would, a better implementation may have just allocated a two-dimensional array, and just have done with a single function, and the function does the thing on the array in an iterative manner, and that would be much faster, and it would probably just use one, a little fraction of available processing power, but if you want to use Orleans, I think this shows you how Orleans could be used. Again, each isolated process, or each isolated grain controls its own state, reading from the data store, persisting to the data store, doesn't really communicate with other grains too much, especially not at the same level, it may communicate with a parent, it may communicate with a child, but not at the same tier, so as to work around the same work around that Microsoft people use re-entrancy to solve. Yeah, so I guess there's, there's only one last question left, which is, should you use it in your Elixir applications? I think it depends, but keep in mind that Orleans, in itself, provides you with a framework that you can base your application design upon, and in any case, if you're creating a lot of actors, you ought to have a way to make them go away at the end of their useful life. All right, thank you very much, any questions? So the question is, do grains provide ways to asynchronously message each other? And the answer is yes, because grains are processes in Erlang and Elixir, and processes can send each other messages. The only problem is, if one grain is busy doing one thing, and the other grain is trying to message it, now you have two busy grains, because the second one cannot get a response on her, the first one becomes free. So this is why in the dynamic limitation of Orleans, there is re-entrancy, so you can call the grain again, and the scheduler does its magic for you. But in this case, I think, I think there might be a very complicated way of doing it, which is, for example, you don't use calls anymore, everything is cast, so this grain sends a message asynchronously to another grain, without waiting for a reply, and then carries on, something like that, and if you move everything to casts only, then yeah, it's doable. Or you can redesign your architecture, so there is no need for such kinds of communication. Please. I see a built-neighbors function, built-neighbors function. That is an Erlang-level service discovery question. It's a no-discovery question, and I am not too familiar with LostPG to answer whether, when there's a new node added to the cluster, whether it picks up that node automatically, but I think it does. So LostPG, according to documentation, uses either its in-house distribution, I'll go, or it uses or falls back to Erlang distribution, and if you add a node to Erlang distribution, obviously that node is there. So related to that, there is a question of low-balancing, which is, if I have a lot of grains running in one silo, and I add another silo, does that silo automatically take over 15% of the grains? I think the answer is no, it doesn't. However, if you use a similar library called Swarm, written by Bitwalker, Swarm does do that. So Swarm moves the processes around, based on available resources, and based on available hosts. So if you spin up another host, I think it does move towards it for you, but if you were to try and implement your system in a way that load is automatically balanced, then the best way would possibly be to make sure you don't have long-running grains. So grains work, and they return to idle, and when they return to idle, they will eventually be recycled, so they don't, they don't really exist. So when that grain is needed at a later time, it will just be spun up, anywhere else. Yes? So question, is this distributed around only? The answer is no. I wrote it on a single node, and it ran on a single node. Can you run it on two nodes when it's distributed around? The answer is yes and no. If you have two nodes and they don't communicate with each other, then you cannot have the grains running on one silo over to another one. So it wouldn't make sense. Um, if you're asking whether beyond distributed Erlang, can you use another technology to achieve the same function so you have grains that are linked, but you don't use distributed Erlang? The answer is yes. You can, so last PG itself supports using something other than distributed Erlang. Does that answer the question? Anything else? Well, if there are no further questions, then thank you very much for coming today.


Today, I just made the crispy skin falling apart chicken taco style. Check it out.  You wanna turn them skin side down. We're gonna squeeze some olive oil. Use the taco. That's the orange top. We're gonna sprinkle generously. Flip them over. More oil, more taco. 350 degrees for one and a half hours. Let's see how it turns out. All right. We can peel the skin off. Set that to the side. It's still super juicy. Get some of this good meat. Put it in our shell here. Dano's taco chicken thighs. Wow. Totally taste the taco flavor. It is in the meat. It's on the shell. Wow, wow, wow. That's crazy good. Get in that juice. Oh wow, that's so good. Flavor packed. Yum, yum, get you some. Leave a like, leave a comment. Let me know what you think and we'll see you next time. Keep sprinkling, my friends.


Good morning, good afternoon, good evening, wherever you're hailing from, welcome to a special edition of OpenShift TV. This is the call for code for racial justice mini-series that we've been doing lately. Today we're talking about the legit info project and to kind of tell you more about call for code for racial justice and more specifically what we're talking about today. I will hand it off to Demi. Demi, how are you today? I'm doing well. Thanks so much for having us here today, Chris. Thank you for coming on. Yeah, thank you. Hi, everyone. My name is Demi Ajayi. I'm the open source community manager here at Call for Code for Racial Justice. And today we have with us a few people from our legit info team, and I'll let them introduce themselves before telling you guys a little bit more about call for code for racial justice. We'll do it alphabetically or however you want. Yeah, go ahead, Tony. I'm Tony Pearson. And I'm the lead developer on call for code legit info, and we'll be doing a live demo and presenting some hands-on labs to go with it. Hey, I'm Tommy Adams. I'm kind of a miscellaneous person on the team since the beginning with Tony. I did a little bit of testing, very little coding, some, a lot of research and documentation, just kind of any odd jobs that the team had for me. Hey, everyone. My name is Upkar. I am a developer on the project and also work with all of call for code for racial justice projects. Excellent. Thank you, guys. And again, as I said, I am the call for code for racial justice open source community manager. If you've been part of the series, you'll notice that I'm a guest here. Usually we have Sabine Justeline, who is the product manager for call for code for racial justice hosting, but she's unable to join us today. She's going under the weather. We're hoping she gets better real soon. So again, this is our third episode of this series. The first two episodes, we have fair change and open sentencing as part of the series. And today we'll be talking about legit info, but just a little bit more about what call for code for racial justice is. Essentially we're currently all IBMers. So this emerged out of an initiative that was launched by IBM, which is the Embrace Initiative, which is a diversity and inclusion initiative within IBM that really wanted to do some work to really make some change. So as part of that, we had a call to action where we launched a spot challenge for call for code within call for code, which is our tech for good platform. It's hosted by the David Clark cause that has an annual event every year to support global challenges around social issues. So last year we were focused on for the spot challenge on tackling systemic racism. And the plan was essentially to have different different people within IBM and within Red Hat to contribute to open source to building open source technologies that would address issues around three different pillars. We have police and judicial reform and accountability, which we talked about with fair change and open sentencing a bit as well as legislation reform, which we're going to talk about today with legit info and also diverse representation. So call for code for racial justice is now grown to be it's a full fledged program under call for code within IBM. But as I mentioned before, we're open source. We've externalized our solutions. They're now on the Linux Foundation seven of our solutions, including legit info, which we'll hear about more today. We're definitely looking forward to telling you more about the solution about finding a way for you guys to get involved as well. So now I hand it over to the team to talk through more about the solution and how they got to where they are today. And we're kind of trying to make this conversational. So if you have any questions, do you come into the chat and we'll also try to fill the questions to the team as well. All right. Can everyone see the screen? Yes. All right. Tommy, why don't you start off with talking about the problem statement? Sure. Yeah. So I'll give you a little bit of background on on the specific problem that we're looking at. And as as Demi said, you know, we are grew focused on the legislation and policy reform area, and within that specifically, the problem statement that we were kind of most interested in was listed here. And that's concerned citizens and impacted residents don't have a straightforward way of knowing what or how policies and legislation affect them or what they can do in response. And so as we kind of dug a little bit deeper into that that problem statement, what we really found is that there's a couple of, you know, there's several issues here, but a lot of it has to do with legislation itself. Legislation one is it's very hard to find, you know, you have government websites, but they you have to kind of sift through their their pages, go through different sessions. You really have to know how to navigate to find what you're looking for. There's also usually no good, you know, search mechanisms there. So you're really kind of limited to to knowing specifically what you're looking for and knowing what bill number you're looking for to actually find to find it on their website. Once you find it, you run into the other problem of it's it's very hard to understand, right? And the legislation is typically it's there's a lot of legally is there's a lot of political speak in there. So it's it's it's difficult for the average person to really understand what they're reading and get to the kind of the core of what that legislation means. And then in dealing with, you know, call for cover racial justice, we also found some impacts specifically towards people of color. You can kind of see the figure two over here we have on the right. People of color are more likely to report racial discrimination when trying to vote or even participate in politics. And so this could be something, you know, that's easily seen like, you know, poll blockers or somebody interfering with elections. But it can also be things that are a little bit more discreet.   And so what we were trying to really do with with our project with legit info was to make legislation more accessible and more easy or easier to understand for everyone. Right. So Tony, if you want to go into the next chart here. There we go. All right. So what is legit info? So our solution is a web based application. Its primary goal is connecting users with the legislation of interest based on their preferences for impact areas and geographical location. So basically what this means is we're trying to make it easier to access this legislation, bills that are upcoming, bills that are existing now, things that are about to be voted on. We want to make it easily accessible and easily understandable. We want to be able to be able to find the information that they're looking for without having to jump through a bunch of hoops to get there. And so some of the current features that we have, it's an interactive website, both mobile and tablet compatible. It allows users to research either anonymously if they choose to, or they can register with the site and save their settings for future sessions. And then of course, they can search the legislation based on their location or on impact areas that they're concerned with. And then they can actually save and share those results with other people as well. All right, next chart. All right, so a really kind of high level process of how legit info works. First and foremost, we have a collaborative effort going on with Legiscan. So Legiscan, if you're not familiar with it, is another third party website. And they do a really great job of manually collecting and collating all of this information from all these different states and federal government websites, basically pulling all of this legislation down and putting it onto a single pane of glass for people to search on. So while they do a really great job of consolidating all that information, they still don't really classify the information or make it easier to find what you're actually looking for. They make it easier to find because it's all in one place, but you still kind of have to know what you're looking for to be able to get to it. And so in our collaboration with Legiscan, we actually run a batch process where we access their REST API every week. We download all the pertinent legislation updates, anything that's been updated over the past week in both PDF and HTML format, kind of depending on which state is being scanned or which state is submitting their data that's submitted in different formats. We take all of this information and we pre-process it to extract the text from those files. And then we send that text to Watson Natural Language Understanding. That lets us identify the key topics for each of those legislative documents. And then from there, we take the process files and the impact areas, saving them out to our legit info database and then make them easily accessible via the web application so that you can start searching by topic and location. Awesome. So I just had a few questions here. This is cool. Obviously, I know a little bit about legit info, but it's always great to hear from the people who developed it a little bit more. Can you talk a little bit about what challenges, like what was the biggest challenge in kind of trying to pull this together as a solution? I'll start. One of the big challenges was that we had a very short time window. We were only given six weeks to try to develop a workable system.  And we were also struggling at that time before we found Legiscan, finding source data to work with. You can't train a model unless you have actual source data to work with. And the first few weeks, not wasted, but spent design thinking, who would use the tool and who are we trying to deliver this for? And trying to interview people who would use something like this and come up with what are their requirements. So one of the requirements was anonymous search, which we were surprised at, but a lot of people are concerned that their searches might be tracked somehow. And so having an anonymous search feature seemed like one of the first requirements we tried to implement. Interesting. And actually, as you were speaking about your design thinking session, do you have an idea now of who your ideal user would be? We've identified the advocate, we call them the advocate, generally someone who is computer savvy, but may be have a set of social media followers or is leading an effort of local activism to encourage people to vote, to encourage people to protest, to encourage people to call their legislators. And so rather than trying to think that this is affecting everybody, we aimed for having an advocate who would use the tool and then use that to drive their next steps, their next actions, being able to share that information with their followers. And along with the information, say, here's how we need to act. Here's what we need to do next. And that's kind of the results of our design thinking effort. Interesting. And did you guys have any ethical considerations as you were developing this? Any things you thought about and how you developed that? You kind of mentioned anonymous search. Are there other things that came up through your design thinking or through actual implementation of this solution? That was the main one. We added later the ability to save, register as a user and to be able to do additional functions if you registered your information. And if you are an activist, one of the things is that you can be an activist for people or things that can't vote. The people for the ethical treatment of animals are representing animals that can't vote. There are people who focus on immigration reform and the immigrants may not be able to vote. Or children and children's safety and children cannot vote. So sometimes the advocates have to act on behalf of the people who are actually affected by some of the impacts from this legislation. Okay, interesting. One of the other things that I thought was interesting, you know, talking about like ethics while going through this, you know, one of the things that we were trying to also do with legit info is to reduce bias in, you know, how we're classifying some of these things. You look at a lot of bills and depending on the news stories you go to, you know, you can get wildly different stories about what the bill actually entails. You know, something as simple as a grant for public education, you have one side that's very pro, you know, education is positive, you know, pro for the bill, but then another side that's looking completely at the financial aspect of it is saying that, you know, it's gonna increase your taxes. So you can get positive and negative. And so one of the things that we're trying to do while we went more with the machine learning approach is that kind of take away that bias and just look at what is the core of the bill? What topics does the bill impact? And you can kind of form your own opinion on it. Yeah, that's a good point. And actually that reminds me of our other solution take two, you know, the one that's about diverse representation and trying to identify that kind of bias, specifically racial bias in tech. So I'm wondering if there's actually a possible synergies between your solution here and maybe also like anything as far as like media articles. Yeah, so that's a great point about natural language understanding. I actually was the product manager for Watson natural language understanding. It's always very interesting to see it being applied in different solutions. Can you talk a little bit more, either Tommy or Tony about how you guys use NLU specifically and how it's been a help to you? Sure. The initial MVP minimal viable product we didn't have Watson NLU. So we were hand curating all the legislation. We were reading the dozens or hundreds of pages of legislation and trying to summarize it into a impact to say, you know, having read this document, the impact is this. And so we were hand classifying individual legislation and then putting it in the database and then you could search from that. And it was terribly manual. I mean, to say the least, it wasn't sustainable. And after the six weeks we're over, our team was given the opportunity to enhance the tool to get additional help from others who were Watson NLU knowledgeable to help implement the NLU capability. So currently today, when we are processing the PDF and HTML files, we extract the raw text and that text is then processed by NLU. We didn't have to train NLU. It already can read English and understand English terminology. So that saved a lot of training time. And what NLU does is gives us 10 phrases that it considered to be the most important phrases for this document. So it could be phrases that then reflect whether it is healthcare related or related to safety or transportation. And then what we had to do was map the thousands of phrases that it came up with to a set of impacts that would standardize the results of that. So it's a kind of a two-step process. And I kind of relate it to using your dishwasher, right? When you wash dishes, the dishwasher is an automated thing, but we had to rinse the dishes before they go in the dishwasher. And then when you take the dishes out of the dishwasher, you have to then clean out all the spots and everything else of the few dishes that weren't quite cleaned. No dishwasher is perfect. And so Watson NLU is like a dishwasher. It does 90% of the work, but we did have to pre-process the text from the PDF and HTML to get it to be usable for Watson. And then once Watson came out with the top 10 themes or phrases, we had to then interpret that to put it into the impact categories that we had chosen. All right. And how did you guys get connected? Go ahead. Go ahead, Rukwar. Well, I was just gonna add that that's interesting to me that it's not a classification exercise. It's not like you're looking at a legislation and saying, does this classify as healthcare versus safety versus, right? It's going the other way around. So not limiting or by the impacts that the solution considers, right? So in this case, Watson comes out with, does it happen, Tony, that it comes up with impact areas or words that are not listed in our impacts? Oh yeah, absolutely. In fact, we anticipate that there's gonna be legislation that doesn't apply to the impact areas we're interested in, right? So for example, military spending. Well, we're not interested in military spending and there could be military spending bills and all of the themes that it comes up with are military oriented and it doesn't match any of our impact areas. So it would be classified as none as not one of the five impact areas we're interested in. So there were quite a few of those. Some legislation, for example, erected statues for famous people or removed statues of famous people. Same thing, did not classify as any of the five impact areas that we had. So it gets summarized as none. So absolutely, rather than trying to come up with a traditional classification machine learning language, we thought this was the better approach. It allows us to add and remove impact categories without having to rewrite or retrain the model. Okay, actually that's a... One other interesting application of this and this is something that I think is something that could be applied in the future but it's not on there now. But the model that we use allows us to do this is we can actually look for riders on bills as well. So you have 10 impact areas and 80% of them have to do with healthcare. But then for some reason, you have two of them that have to do with finance or something that might be a good indicator that you actually have. And it dims them to the bill that's not really part of the initial bill but it's something that stands out. Exactly. Interesting, all right. So that brought up two questions for me. The first one is related to the source of truth and choosing the impact areas, right? Like, how did you guys decide what governs the impact areas that you want to show and how might we think about doing that in the future? For instance, maybe military spending, we didn't think was relevant but if someone else in the open source community thinks like that's actually relevant, how do you go about deciding that so far today? Right, this is the problem with design thinking in that the limited number of people that we interviewed, we then called it down to five impact areas and chose that as our initial set of impact areas that we chose and I'll be showing those soon. And we chose initially Ohio and Arizona because we had websites and documentation that we could use and we had people from Ohio and Arizona on the team who could make sense of some of the more subtle subtleties listed in the document. And that's kind of how it worked out. Now we designed it so that you can have any of the 50 states, we designed it so you can modify programmatically in the database you can list the locations and the impact categories. So if you want to add and remove impact categories, we made that very possible without having to have anything hard-coded into the code. And as a result of that, if we interviewed 20 different people we might have had five different impact categories. It just happened to be, we spent the first two weeks just trying to discuss who was the advocate and what impacts might they be interested in doing in looking at. If we had interviewed 50 soccer moms, we might have had more things related to education for children and school safety and so on. And if we had interviewed other categories in different states they might have different priorities. Okay, that's good to know. I mean, I guess it definitely creates like a opportunity for thinking more about how we want to develop this further down the line as well. Now, the other question I had was related to ledgescan. Can you talk a little bit more about how you guys got connected with them? And it seems like it made your jobs a lot easier, but can you just give them a bit more explanation as to how you were connected with them? Yeah, I would say that we had a real struggle trying to figure out when it was originally just curated data, if we found legislation, we could curate it, summarize it, give it a title, give it a summary, give it an impact and a location and put it into the database manually. But when we wanted to try to get it more formalized with repositories, we found that every state has its own different repository and the different website and different webpage and screen scraping was just not something we wanted to do, a screen scraping is often a matter of getting the raw HTML of each page and trying to extract elements from it and follow links until you find the attached documents. And that's complicated for a single website and multiplied by 50 because there's no standards between each state's websites. Each state develops its own way of doing things in all 50 states. We also didn't want to limit it to United States only. We could have used this in the UK or Europe or developing countries. So we were trying to decide a balance between individual location states websites and some more centralized repository. And we reviewed quite a few attempts at trying to consolidate legislation. And let's just scan, we chose because it had a nice API. It has both a free and a paid version. And because we worked out with let's just scan how we could implement this with just a free user ID. So we wouldn't have to scan and cause too much effort on their servers. We didn't want to generate too much CPU load against them. For the free scan, we can do it with a minimal amount of impact to their servers so that they can continue to use the free user ID and the REST API. And so that worked out well. We talked to the let's just scan people, told them what we're trying to implement and so they're very cooperative. Awesome. So thank you guys. And thank you, let's just scan for supporting this effort. All right, so what is next guys? Okay, so let me first give an overview of the architecture and then I'll do a live demo. And so the instance here is the web application. The user would access it normally through a browser and this could be on a desktop or a laptop. It could be a tablet, it could be a smartphone and we are able to then access the web pages, do our searches. We have the option to save CSV files of the results we find. We're also able to email the results to ourselves so that we can then forward it onto our followers but by adding some additional comments to it. So rather than emailing it directly to our followers randomly, we thought it'd be better that you email it to yourself and then you forward it onto your team to say, look at the list below, let's take action, let's meet Saturday to discuss it, let's, you know, et cetera, et cetera. So we thought that was a better approach based on our conversations with activists. Our data sources, you can see here, we extract from Legiscan and we store them in an internal database already organized, filtered, categorized with the location and impact and we have a user directory so that people can register, sign up and save all their settings so they can come back and everything is where they left off. So that is our architecture. And so let me jump to a demo. And I will jump to here. Okay, hello. Nope, it's not letting me do that. There we go. There we go. Yeah, it's the Zoom thing seems to drop down right when I need it. Oh, yeah. Okay, so this is the Legitifo screen. Can everyone see that? Yes.  And as an anonymous person, you can implement the search and we can hit the search button and you could do that.  The menu on the top gets changed to a pull down menu. And when it's a wider screen, then the menu comes out on cross. All of the choices here work regardless of the size of your request. And as an anonymous search, you can say I'm interested in, you know, let's say Tucson, Arizona and I'm interested in healthcare, safety and transportation. So you can check multiple impact areas. And then when you do the search, it would then identify the search there and you can see the title. We actually have the bill name. So you know that this is from 2018, HP 2001. It affects all of Arizona, not just Tucson and summarizes it there. We list only a few per page and then you can go to the next page and look at those next pages and so on. So we did that. You can at any time print the page out or download it as a CSV file. And the CSV file is nice because then you can run it through other reports or forward it onto your team. Or you can go back and say, I wanna search something else. I wanna search Ohio. So let's say we go to Columbus, Ohio and now I'm interested in environment and jobs and I can now search a different state. And here again, I have the titles and the summaries of the different things. I can say that some of these are based on jobs. Some of these are classified as an impact to an environment. So you can see that for that. So again, at the bottom, you can see that there are thousands of legislations found that you can do this. Now let's take a quick look at which impacts we have. These are the five impacts we have, healthcare, safety, environment, transportation jobs. Again, that was the result of our design thinking. We needed to pick just something that were relatively unique and different that we can find legislation for. And that's how we've classified our legislation. If you look at the locations, we have a hierarchical search. So in this case, all the states relate to the United States and then counties relate to the states and then cities relate to the counties. We've structured this so that it can work with all of the United States and all of the districts and on other countries, it could be provinces and districts and towns. Whatever your country is, we can probably organize the hierarchy to do this. So if I chose Columbus, Ohio, as my designated location of interest, then I would find any legislation that was a city ordinance for Columbus. I would find any counties that were related to Franklin County. So it affects all members of Franklin County, including Columbus, Ohio. I would also look for any statewide initiatives for the state of Ohio. And then I would find any federal legislation related to the state of United States. So when you pick a location, it doesn't limit it to just city ordinances. It can actually find all of the things related to that location at the government hierarchy the way it's organized. So I think that was another key thing there. You wouldn't have to do four searches. I mean, otherwise you'd have to search Columbus and then you'd have to search Franklin County and then you'd have to search Ohio and then you'd have to search United States.  We can also register ourselves. So I can say, Frank one, two, three, right? And let's give it a password. And we register. And here when we do the registration, we can say Frank and we can then pick their location. We'll say that they live in Cleveland, that they're interested in Cleveland and we can save some of their settings that they're interested in these three and we save those changes. And let's go ahead and add. All right, so now we're logged in. It says hello, Frank at the upper right corner. So when I do a search, I have it all preloaded. I have it loaded with Cleveland, Ohio, and I have it all. But that doesn't stop me from doing other searches. I could say, well, even though I'm located in Cleveland, I'm gonna search for Cincinnati. And even though these are my normal defaults, I'm gonna add jobs and remove safety. And I can still do the search from that. It just pre-populates the standard default for that. So you can see here, I have jobs, healthcare and environment. And again, it would be city ordinances for Cincinnati, county ordinance for the county that Cincinnati's in, which is different than the county that Columbus is in. And then the state of Ohio, which would affect everybody in Ohio and then federal laws as well. So that's, you can see a very flexible way to do it. And we designed it so that advocates don't have to be expert search ninjas in terms of trying to figure out how to find stuff. We really try to make it accessible and easy to say, these are the impact areas I'm interested in. These are the location. This is the communities of interest that I want to do. You can always look at your profile. And if you want to change that profile, you can update it to say, okay, I wanna change. I moved from Cleveland, Ohio to Cincinnati. So now I am less interested now in safety and more interested in transportation. And you can save those changes and you can now make that possible. We also have the ability, and I will sign out here, to sign in as an admin. And I already have the user ID and password loaded in which case, when you log in as an administrator of the application, you get the admin button. And here in the admin button, you get to actually see all the database tables we have. This allows you to reset the passwords of individual users who say, hey, I forgot my password. Can you reset it for me? You could look at the laws, for example. And if you look at the laws, you can choose to add laws, you can change the laws. If you wanted to, you could modify the classifications. Let's say that you wanna look at, you can create, you can add a citation in the law if you wanted to, you can change the key. I'm gonna go ahead and cancel this, but you could manually enter that. Who would have access to that kind of? Yeah. Well, I would say if you're running the website for a group of activists, you would have an IT person who would be in charge of that. You could, for example, here's all the locations if you wanted to modify them. Here's all the impacts. If you wanna add or remove impact areas, we have tasks to do the batch jobs. You can schedule when those tasks run, for example. We have our downloading the Legiscan data sets runs. The next time we'll be running on the 13th of June or whenever the next time is. And so on, you can schedule these to run weekly, Monday, Tuesday, Wednesday, however you wanna do that. So yeah, so somebody would have to be the administrator. At least initially to set it up for, if you want to set this up for the state of Minnesota, then you would have somebody who would make sure that Minnesota was listed, that you'd have the impact areas that you are interested in and then make it available to the activists who needed to do the searches that they wanted to search. And that would depend on whoever's sponsoring this or making this available to their communities, to their advocates. Right, and because this is not, there's no central website for anyone to look at this right now. There's only, we can only download local copies of it essentially, right? That's right, yeah. I mean, we have copies running in the cloud just for our own test purposes, but if we wanted to deploy this somewhere, it's open source. People can download it onto their servers and try it out for themselves, sign up for a Legiscan user ID, sign up for Watson NLU, sign up for IBM Cloud Optic Storage and completely have a fully running system with minimal effort. All right, so that is it in the nutshell. I think I went through my entire script showing how that all works.  Oh, I'm gonna talk this chart and then I'll hand the torch over to Upcar. But what we ended up doing was we had three stages for deployment. The first stage was for development and test. We tried to keep it simple so it could run on a local file system with a very simple database as Qlite3. The application has a written in Python has a single user run server that's built into Django and Django also has a Q cluster to do batch processing. You could use Cron if you were on a local system as well and a web browser with a single user. And you can do all of that with curated content without having to use Legiscan or the Watson NLU. And that might get people started so they can see how it works and validate things. And then in the transition phase, we're able to move the local file system which contains all the PDF and HTML of the legislation we've downloaded into either a network attached storage and NAS or we're using IBM Cloud Object Storage. We can support either of those. And instead of just the SQLite3, we're using a more robust database called PostgreSQL that we have running in the cloud. So it's a centralized repository. And now what's on premise is just the application. We're switching from the run server to a more formal Gunicorn and continuing with the Q cluster. But this now with Gunicorn allows us to have multiple users. In this case, you could have a browser, tablet and mobile phones accessing the application to make sure that everything looks responsive and is spaced and positioned correctly. And then once you're finally ready to run in production, here what we do is we move the application, the Gunicorn and the batch processing with the Q cluster in the cloud. And so the only thing you have locally would be your browser, your tablet or your mobile phone. And so this transition really helped getting people on board. People could download the code and do the development and testing locally, get familiar with it without affecting anybody and then getting into transition mode, learning PostgreSQL, learning cloud object storage, learning the Watson NLU algorithms and keeping the application itself running on premise so you can still debug it, stop and trace activity there. And then once you are ready to move all of that into the cloud for production, you're in a much better position. So I will then pass the torch over to Upgar who will then explain our latest deployment using code engine. Yeah, thanks Tony. So this diagram is showing the current site that Tony was demoing on. And essentially what we did was each component of the application, we created separate Docker files and containerized that portion of the app and have them run separately on this platform called IBM code engine. But it's all containerized so it should be possible to move it to other platforms as well. And we look at the code pretty soon here but code engine has two components, sort of big components. One is the concept of an application that users can access from the outside. And then there's this concept of a job which are basically running the background, process something and then die. And so the top part here is the application user comes in. Sorry, the numbers, we didn't put that in the chart here but the numbers basically tell the flow of users coming in and then how the application interacts with Postgres. We are using a Postgres service on IBM cloud. And then at the bottom you'll see there, the three jobs that Tony talked about. So there's the get data sets job which retrieves the different data sets for the different locations. And then there's the extract files which extracts the legislation out of the zip files that came through in the get data sets job. And then finally the analyze text uses the NLU function. And these jobs are set to run on a weekly basis. And that's again, there's a way to do that with the code engine stuff. And then again, using cloud object stories as the persistence layer in our deployed application. Tony, if you don't, if you don't mind moving to the next slide. Cool, so yeah, we have a couple of places for folks to start on, right? Tony and I created this lab and actually let me, maybe I'll share my screen and show it as well. You wanna do that? Or do you wanna go through these charts and then switch to the screen? Okay, let's do that. I always listen to Tony. Yeah. Okay, so go through these next two or three slides and then we'll go into the interactive portion that you can show. Gotcha, okay. So, but the links right on top, that's the first thing. So if you wanna try it out on your own later on, Bill, Chris, if you don't mind sharing that link. I already did. In the chat. Thank you. All right, so when you land on this page, there's a button to log into the lab environment and you have a couple of different choices or options here to log in. I have an IPMID, so I use that. A lot of folks use GitHub if you have a GitHub account and you can use any one of these social buttons to log into the environment. And the lab, there's no cost to it lab to do this? There is no cost to that. Correct, there is no cost to the lab. It's self-paced and the next screenshots they'll show you, it's a very sort of guided concept. So thank you. So you can see this is the lab environment, works in the browser. That's all you need to start it. On the left-hand side, we have the different steps that you will go through. If you do the lab and on the right side, we have a editor, looks very much like VS Code. And so you can, if you're familiar with that, you can open the terminal, you can click on the little files thing to see all of the files that are in this lab. And then at the end of the lab, you also have this launch application button on top so you can tell it to open a certain port and actually see your application running locally in the lab environment itself. So what do I need besides a browser on my laptop to do this? A keyboard? No, that's all you need. I don't have to have Python running on my laptop. I don't have to have... Correct. Any of this, right? I can just use a browser and I am launched into this skills network that has all of the stuff on it. Nice. That's correct. So it's like it's a lab environment that has Python already installed. The steps go through installing some of the other dependencies that the lab needs of the solution is built on, for example, Django, right? And then it has a couple of other utilities. So for example, IBM Cloud CLI, there's a portion of the lab that has you deploy this application on IBM Cloud under your own account as a Cloud Foundry application. And so that IBM Cloud CLI is also in there. So yeah, the reason for doing it this way was, versus the other route you can take of course is go through the readme file and do it locally on your system, but then you have to install those different things. So all of that is given. It should be the same steps, installing Python, installing Django, installing Bootstrap, but going through all of those procedures. It would be the same steps, correct? Yeah. So if we go one more, there we go. And then this is the other thing we try to make it easier for our users. So I think Demi mentioned part of this is, you know, you are interested in helping out with this, with this project, but also you might want to learn more, right? So let's say you want to learn about Django or you want to learn about these, how the Django queue scheduler works to under the hood or the Django admin site. And so this is, this lab goes through all of that as well. So it's a good learning opportunity as well. So to make that a little bit easier, the instructions, we kept them short on the left-hand side. There's the next and previous, or previous and continuum button that you can use to move back and forth between the instructions and then any command that we want you to type, there's the handy copy button. So you click on that copy button and then you can paste it in the terminal, which is shown on the next slide. Here we go. So on the editor side, you can go terminal, new terminal. And in there you can, you can paste it. I think Tony took these screen shots. So very nice of you, Tony, to say control V or control shift V. It depends on whether you're running a Windows or a Mac. You might have to be a command V instead of a control V. It seemed it was just slightly different in each operating system. So I kind of said, figure it out for your own, how to get the copy to work. There we go. I think the right click also works the right click. The right click and then paste, yeah. There you go. It went in doubt. Went in doubt. A lot of different ways to paste copy. I think that's the one thing developers get really great at very quickly. So that shouldn't be a problem. That's a good skill for all open source people to learn how to do is copy and paste. There you go. Okay, so I think this was the last one yet. Yes. This is the last page. Let's switch now. I will stop sharing and let you take over the screen. Thank you. Ooh, wow. Ooh, how that happened? Oh my, nice. Okay. My terminal does that sometimes. All right, so I do have, I have the lab open here. By the way, thanks to Tony, I've been using Linux for like what, five months now. I thought Tony using it looked pretty cool.  I haven't opened my MacBook for a bit now. Nice. And it's been great. All right, so here's the lab. Ashley, let me open the other window just so you can see. I should mention that we designed this tool while you're doing that. That the Python and Django can run natively on Linux, Windows or Mac, the way it's structured so that all the instructions are based on Linux, but don't let that scare you. It can be implemented on Windows or Mac as well, fully supported. So if you have a Windows machine or a Mac machine, you wanna download the code and run it locally with your own local Python compiler and your own everything else, have at it because it's not limited to Linux and don't let that stop you from anything. Cool. All right, of course, I'm super secure this way. So I'm gonna switch to the other window where I'm already signed in instead of pulling up my phone and the authentication app and all of that. So I'm already signed in here. So let me do another access quick lab. This should open the environment. We do say this lab takes 120 minutes to complete, but we noticed you can get through it a lot faster as well. That 120 minutes, if you wanna learn more about the different components and the technologies used, so you might do step one, then you Google a bunch of stuff to learn more and come back. But yeah, I think maybe 20 to 30 minutes is generally good to go through the lab, good enough. Okay, so you can see here on the left is a quick project overview. On the right is my editor. I'm not gonna go through the lab, but let's just do the first couple of steps. Tony, I love the music keys. You're getting it. Yeah, I'm turning off all my phones. I apologize, I should have done this. No problem. No problem, it's nice. All right, so here's an example. Too many phones. Here's an example. So get version, I copy it here and I can paste it here and enter, boom. I have get install, which is nice. Oops, let me go back one, one more. And let's see if I have Python down here. And I do have Python 3.6.9 that comes with the environment. And then we're using Pipeenv to create a virtual environment and manage all of our packages in that virtual environment. So right here, I installed Pipeenv using Pipe3 and I can use this next command to see what got installed. So Pipe version 2021, 529. This portion of the lab does use Escalite 3 as the database. So we've shown you the different tables that get installed as part of this. So if we go one more step, and this is probably the last thing I want to show you. So CD into home project and then get clone the actual code, the legit info code. So as soon as I do that, let that finish. You'll see if I now open the browser or the file browser in VS code, here are once that finishes. Here are all my files that come with the get clone, right? And again, you can read through Tony and team and Tommy did a wonderful job documenting all of this in the different read means that are in the repo. So take your time, go through that. I'm gonna go into legit info and then create my virtual environment with Pipeenv shell. And we also right in here, we did try and describe or put a little bit more writing on what these different things are, right? So Pipeenv, what is it doing? Pipeenv install. And then the one thing I do love that Tony did here was using all these scripts. So you can see here, this command is calling the stage one script and passing in an argument and you can go in here into the files and look at stage one, so you can just open that and see what's happening there, right? So it's really, it's a very open, transparent and again, I thought this was a great way to learn Django and Python. I got to, I learned about Django custom commands and was able to create a new command to add in new states into the solution. So yeah. All right, and then let me, so I'll stop there with the actual going through or doing the lab, but I do want to show you what happens at the end of it. So once you have followed all the directions, you use this launch application button on top, give it a port and basically you see this. So you'll see the legit info solution that Tony was demoing before. And then the next couple of sections, explore the application goes to the different panels, the different admin things you can do, exploring different parts of the solution. So that's this whole step. And then we also have, as I mentioned before, we have a step to run this application as a Cloud Foundry application on IBM Cloud. Again, using SQLite 3, we're not hooking into Postgres at this point, but the steps goes through deploying the application on IBM Cloud. And then finally we have some challenges. So one of the challenges, for example, is update the location. So add a new location, what does that look like? There are other challenges at the bottom as well. Update and impact is another one. So it's a good way. So once you have learned the solution, then you can try some of these challenges. The whole pair, apart from the education bit, learning about these different things, is also hopefully at the end of, what is the step nine of 10, I think? Step nine, yeah. Yeah, step nine. So hopefully by the end of it, right, you are at a place where if you wanted to contribute back into the solution, you understand the different components and the technologies that are being used. And it makes it easier for you, hopefully to contribute back. That's it. The next steps then, once you've done this, again, if you wanna join the community, call for good information to this community, you can do so by going here. But I also know Demi has a slide on that. And then if you want to see what the team is working on, the different issues that you can pick up, then you can go to this issue section of the repo, right? So let me bring that up real quick. So a lot of these, the team has been working on. We also have quite a few of these are actually feature requests and not bugs. So, and properly labeled as enhancements. And then Tony's gone through and we have marked quite a few of these as help wanted or good first issue for folks who are new to open source to come, to contribute back. That's great. Fantastic. That was a quick, quick intro to the lab. I'm gonna stop sharing. Okay, and then I will get the last screen. All right. So, Upkar has done a great job walking us through the lab. And thank you, Tony and Tommy as well for getting us through what legit info is. I also learned a little bit through this and I have like a lot of ideas as well. You guys have done such a great job with this and there's a lot more that we can do. I just see so many possibilities where we can take this. So we want people to join us in this effort. So if you're interested in learning more about legit info or a call for code for racial justice in general, you can follow along with a link that should be in the chat. It's also on the screen here. You can use your phone to scan the QR code and that will basically give you access to our Slack community. First, you will register on through our IBM Cloud platform and then from there you'll get information about opting into the Slack community. And then you'll be joining the Racial Justice General channel and then we'll get you plugged into our solution. So we are looking for all sorts of skills whether you're technical or not. As you heard today, we are interested in more like design thinking around these things. If you're a policy expert, you can also lend your voice that way. If you are, of course, a developer, we're definitely welcome your skills there. So do reach out to us when we look forward to hearing more from you. Thank you guys. Fantastic presentation. I really appreciate the issue triage as well. Those good first issue markers are always helpful for a project. So thank you very much for coming and showing us this project and the ways we can give back to it. I think Ohio and Arizona, we need some more states on there. And obviously that's hard because you're right. It's not consistent data coming in, right? Like it's 50 microcosms of data coming in or 50 plus actually given territories and national federal level type stuff. I mean, Howard, I'm trying to ask this without it being too long of a question and response, but what is the data sanitization process look like? I mean, how laborious is that sometime? Actually, it's all automated now. So we take the PDF file, we extract the text and we run it through the tool and occasionally it'll spit out a phrase that we didn't see before. We need to add it to a little list. So we have a list of phrases we've seen and what impact we would associate that with. And other than that, we're seeing about the same 3000 phrases. So we are able to classify them correctly. Of course, if we had a new state, a new phrase might pop up that we never saw before and we'd have to make some updates. But we have, it's a table that anyone can edit a CSV file that says for this phrase, consider it to be part of this impact. And that's what we use to standardize it. And we had to standardize it because we can't process 3000 phrases. Right, yeah. That's awesome though. I mean, what great functionality, right? Now we can take the project and run with it in our own states potentially. That's right. That's awesome. Thank you so much everybody for coming on and demoing this. This is fantastic. Thank you so much for having us. Appreciate it. Thanks Chris. Truly my pleasure. So again, thank you for your efforts, call for code for racial justice in general. Thank you very much. And we look forward to seeing you in two weeks, I think, Demi. Is that right? That sounds about right, yeah. Yeah, there we it, yeah. 27th, right? 27th, yep. Yes, yes. Cool, so yeah. Come back in and we'll be talking more about that on the, more about the other projects in this call for code for racial justice ecosystem in two weeks. All right, thank you. Take care everybody. Thank you, bye.