Again in high school and Mr. Morgan's class, again there were three main things we learned in there. It was supposed to be a programming class and we learned Corel or Carol. Watched previous videos to get that. Then we learned some QBasic but then we also learned PowerPoint which isn't really programming. It was kind of silly but I wanted to take it a step further so I figured out how to do these animations and I was able to animate some stuff but I wanted to put some real animations in there. At home I was playing again at the time Duke Nukem was popular and Shadow Warrior was an awesome game which was based on the Duke Nukem game engine, pretty sure, made by the same people anyway. There was an opening scene shot with a few frames of animation of Wang and he was doing something in his dojo. I took those five frames of animation and put them on a floppy disk because I wanted to put this in one of my PowerPoint presentations because it would be so cool to have an animation like that. I popped in the floppy disk and of course they disabled the floppy drive in Windows but I quickly found out in that class and this was true for many, many, many years later up in college in the mid 2000s this was still true. You go to a Windows machine and they have certain drives disabled but you open up your CMD prompt and if CMD was disabled you run a command prompt, command.com, which was very similar but a little different than CMD so lots of times they would disable CMD but they wouldn't disable command so you'd be able to start with shell that way and you'd have access to all the drives so I was able to copy over the pictures from the floppy drive and I showed the teacher and he's like, how did you do that? I explained to him because we didn't have internet in the classroom at that time. I'm like, well I put on this floppy how did you and I showed him how I got access to floppy and he freaked out. Not that he was mad at me but he was afraid that he was going to get in trouble that I did that so he asked me not to do that anymore but I did learn a skill there which is basically Windows is completely insecure and although nowadays it's a lot easier to disable stuff because I'm telling you all the way at Windows XP every Windows XP machine which I know Windows XP has been dead kind of sort of for many years now through all Windows XP life I would go to schools or libraries or these places and you would see they would have things disabled but you were very easily to get around it just by running the command or CMD it's like Windows they would be disabled in Windows but not in the shell so anyway good class learn some stuff thanks for watching have a great day


All right, ladies and gents, you're probably wondering where I am. I'm actually in the ocean right now. I'm in La Paz Resort in Halong Bay and To be honest with you, the reason I haven't been uploading is because firstly, there has been no internet in Cambodia the whole time. There was very very little internet. I was in Bangkok a few weeks ago and I posted that video saying like I'm gonna be posting now. There was then no internet throughout the whole time in Cambodia. I was ill in Bangkok and I also was ill for part of the time in Hanoi as well in Vietnam. So okay, so right now I'm in La Paz Resort, Toan Chao Island in Vietnam. Now, this has been sort of a break from the whole traveling thing, which you can see on my other channel, Transcend Your Limits. Now I'm gonna be moving on to Ho Chi Minh City, which is where I'm gonna be staying for a month, and there'll be much better internet there. I'll have much more time to sort of focus on these videos and actually talk about lucid dreaming, which is why I'm here. So with that being said, please don't give up on this channel. I know that you're probably wondering when the next video is gonna come and it's not like me to have a break this big. Normally, I sort of say I'll post a video in a couple of weeks or a couple of days or whatever. This time it has been a bit different because of the internet and because I was ill for a bit. But now hopefully we're gonna get back to doing the regular videos. I've got a lot more stuff to talk about, a lot more content to share about lucid dreaming. So hopefully you can enjoy that, but if you just want to watch what I'm doing in the meantime, you can actually see my vlogs on Transcend Your Limits, which is where I'm spending most of my energy there because I'm enjoying these travel videos a lot more. It's just something I'm becoming quite passionate about.  If not, then a couple of days time I should be posting the next lucid dreaming video. For now, I guess I'm actually a challenge.  So don't just... don't just rely on doing loads of different reality checks. Try and just focus on one for now. And until you see my next video, just try and do that reality check really, really well. Like do it every day for 10 times, 15 times and be really mindful about when you do it. So don't just say, don't just sort of ask yourself am I dreaming and then not really be concerned or, you know, care about the answer. Try and really work out the answer as if you don't know the answer. As if you're trying to find out whether you're dreaming or not, because that's the best way. That's how you're gonna actually lose the dream properly. Anyway, I'll see you in the next video and don't forget to subscribe, like, leave a comment. Let me know what you think of this beach, by the way. I think it's pretty cool. Anyway, you can see more about that on Transcend Your Limits.


How do you find out what is material to a relationship to someone? Um, well, let me let's see material. What's important to them? Well, you first you ask them questions What what does a relationship look like for you? What does commitment look like for you? Do you want to get married? You want to get remarried? Do you like the institution of marriage if yes or no? Why see if someone says okay, let's think about the okay. I didn't say this earlier Why don't you like the institution of marriage? What bothers you? Well, you could take half my money You know, I don't want to be tied down. I want the freedom. I want to move on Why would you want to be even dating someone? Who who is fear is fear-based? Why would you want to date someone that's fear-based?


They're forces of dehumanization that are inhuman and trying to basically lower us into something subhuman that they can more effectively control. I don't see that as evil. I think about these forces the way that we think about diseases or parasites in nature.


Hello, thanks for joining me today. I'm Charles, software engineer at Netflix, and I'm also a TaiKV maintainer. So today, I will share our experience of building a scalable and reliable change data capture service for TaiKV. So today's talk will recover several topics. Firstly, we will explore why we need to build a new CDC tool for TaiKV. Then we will talk about how TaiKV works internally. Though TaiKV currently only support desynchronization from TaiDV built on TaiKV, the same principle and experience can be used by users wishing to sync data from any system built on TaiKV to other systems. This is particularly relevant as syncing data from a distributed database like TaiDV is often more challenging than from other systems. Once we have a better understanding of TaiCDC internally, we will cover the performance gains achieved after TaiCDC version 6.5. We received a lot of feedback from the community last year, like concerns about performers in assistant reliability. So we put a lot of efforts in time to enhance the reliability and optimize the performers. First, let's talk about why we need TaiCDC, what is a major use case of TaiCDC. There are two common scenarios where TaiCDC can be particularly useful. The first one is incremental data synchronization for heterogeneous systems. This means that if you have multiple database that need to be synchronized with one another, TaiCDC can help ensure that all the data is up to date in real time. The second scenario is cross-region disaster recovery, which based on primary and a secondary replication. This can be critical for business that rely on their data to operate. Compared to traditional database system, TaiKV can hold much larger volume of data, which makes capturing change data for TaiKV very challenging, as we want to ensure not only the high data syncing throughputs for large volume of change, but also different level of data consistency. So compared to some other systems, what kinds of features do TaiKV provide? First, TaiCDC provides, so first TaiCDC supports low latency incremental data replication for various downstream. So that is to say you can use TaiCDC to replicate data from upstream database to Kafka using Canon, JSON, Avro, or OpenProtego. Second, TaiCDC support database in a table filtering, which enable you to filter specific data based on your requirement. This feature helped reduce the amount of data transfer and makes the replication process more efficient. Third, TaiCDC support most operation through OpenAPI. This means that you can easily integrate TaiCDC into your existing application without having to worry about compatibility issue. Last but not least, TaiCDC supports bi-direction replication between TaiKV cluster. This means that you can easily replicate data between two TaiKV cluster, making it easier to change your data across multiple clusters. Now we have understand why we need TaiCDC. Next, let's talk about the design goal and the challenges of building CDC service for TaiKV. Our first objective is to ensure high availability. We understand that the critical natural of syncing data to downstream system for users, but accidents and the disasters are inevitable. Therefore, we must guarantee that even in the event of system faults, the data syncing process will continue uninterrupted and in the CDC cluster will remain fully functional. The next goal is to achieve high throughput and a low latency. As previously mentioned, user typically store vast amount of data in upstream cluster, spread across thousands of nodes. Consequently, a capable CDC service should be able to concurrently sync data from multiple TaiKV storage regions with optimal throughputs. Our third goal is to ensure consistency and ordering. Unlike other systems, maintaining the consistency and order of events operated upon is crucial for TaiKV, as many users are using TaiKV as the backend storage for distributed system. Therefore, we intend to provide snapshot isolation and eventual consistency to address this requirement effectively. However, all these goals are not easy to achieve, as we face many challenges at the same time. First, we need to capture the change data instantaneously, as any delay can lead to large synchronization lag. Additionally, we must be aware of and able to catch up with the scheme revolution. As I mentioned before, many users using Tai CDC or TaiKV to build their database system. The structure of the database can change over time, requiring Tai CDC to adapt accordingly. Another challenge is striking the right balance between ordering and high throughputs. While maintaining the order of events is crucial for date integrity, we also need to handle the large volume of change without compromising the performers. Similarly, we face trade-off between consistency and low latency. Guarantee eventual consistency while minimize latency is a delicate task that will require careful consideration. Furthermore, Tai CDC needs to efficiently fetch data that is spread across multiple TaiKV nodes. This requires effective coordination and communication to ensure the seamless retrieval of data from various source. Lastly, we aim to minimize operational complexity as a complex system can introduce challenges in maintenance and troubleshooting. Now, let's take a closer look at how Tai CDC addresses these challenges. Here is the overall system architecture of Tai CDC. Tai CDC talks directly to TaiKV, which allows for more streamlined communication and a faster date transfer. Tai CDC captures change data by watching the change log of TaiKV, which provides reliable and up-to-date source of information. The system is horizontal scalable using table as the basic scheduling unit, which allow for greater flexibility and ease of use. Additionally, Tai CDC provides an extendable downstream syncing interface, which can be easily customized to support different third-party downstream platforms. Currently, we officially support MySQL, Kafka, and Cloud Storage service, but our flexible architecture allows for easy integration with other platforms in the future. Okay, now let's take a closer look at the Tai CDC cluster internally. The Tai CDC cluster is composed of multiple capture server, each of which can be either an owner role or a processor role. The owner acts as the leader and coordinator of the Tai CDC cluster, responsible for scheduling change feeds to different processor managers. Each processor manager manages many small sub-processors, which serve as the base working unit of the Tai CDC cluster. After a change feed is scheduled to a processor manager, the manager will initialize and assign one processor to one table of the change feed. The processor will then start watching the change log of the table on a TaiKV and begin pushing change data to the downstream. When pushing a change feed, the owner will calculate the watermark of the overall change feed process based on the watermark of each individual table managed by each processor. The metadata of the change feed and the Tai CDC cluster itself will be stored in a PD cluster of the TaiKV, which is, and you can treat it as an ETCD cluster. This process allows for efficient and reliable change data syncing even in complex and large-scale environments. Notice that in the Tai CDC capture server, each server will always launch a processor manager to capture change data and push it downstream. However, only one server can act as the owner and the coordinator of the cluster at any given time. After the owner has been elected, we can start scheduling and dispatching tables to processors. The owner initialize a scheduler for each newly created change feed. The scheduler's primary function is to assign syncing tasks to different processor managers with each syncing task mapping specific table. Once the processor manager received the task, it initialize a new processor and begin syncing the corresponding table. During the syncing process, each processor initialize an agent that communicate with the coordinator to track and report on the syncing progress of the table. Now, we have introduced how Tai CDC as a cluster work from high level. Next, we can dig into the details and see how each processor work after being assigned task of syncing tables. Here is the topology of different components. The component topology of Tai CDC involves several key elements that work together to ensure accurate and efficient day syncing. Firstly, one change feed can be spread to multiple capture servers. Each capture server will initialize a new processor that is dedicated to handling the syncing task for the change feed. Inside each processor, a pipeline is created for each table, allowing for efficient syncing of individual day set. The owner of the change feed is responsible for syncing the DDL event which is the data definition language event to the downstream. Such as table scheme update while the processor pulls the DDL events but it's not responsible for pushing the DDL to the downstream. Instead, the processor applies the DDL schema to the monitor which will deserialize the DML which is the date manipulation language. Even later, this component topology ensure that Tai CDC can handle large volume of data syncing task in a maintain accuracy and efficient throughput throughout the whole process. Here is what happened inside a table pipeline. From the left to right, we have firstly the Tai KV CDC subcomponent which is a subcomponent embedded inside a Tai KV. Allow us to read the change log of the Tai KV for ending updates. Tai KV provides interface allowing client to watch the change log which can be useful for building change data capture servers for any system built on Tai KV. Next, the puller component connects to the Tai KV CDC associated with the corresponding regions and watch all change events. When the change event occur, it is sent to the puller. The puller streams the change event to the sorter which sorts the event based on its time stamp. After sorting, the sorter push the event to the monitor. The monitor decode the key value entries into table format and sends them to the sinker. Finally, the sinker syncs the events to the user specified downstream location. We have talked about the Tai CDC syncs to DDL events. Next, let's talk about how the Tai CDC synchronize the DML events. First, how the puller work. The puller is a critical components in Tai CDC state syncing process that is responsible for mounting for watching change events of a table and pulling them from the source database. S table in Tai KV can be split into multiple date region and store in multiple Tai KV nodes. A puller will connect to multiple Tai KV nodes to watch all the related regions. Tai KV nodes push change events to a channel through the GRPC string and the puller creates a worker for each region which periodically reads the change events from the input channel and all region worker share an event channel that connects to the output channel. The output channel dispatch the change events to the DDL job puller and the puller node based on the event type. Next, let's talk about the sorter. Why the sorter is required? Sorter provides two major functionalities. First, buffering the incoming events allows the sorter to smooth out the peak in the valleys of upstream date flow. This ensure that data-sinking process is efficient and it can handle large volume of data without becoming overwhelmed. Second, incoming events may not be in time order which can make it difficult to provide snapshot isolation or ensure eventual consistency if the system crush. The sorter ensure that the events are sorted based on their timestamp allowing the processor to reconstruct the cluster snapshot correctly. Overall, the sorter is an essential component of high CDC ensuring that the data-sinking process is efficient, accurate, and consistent. So how the sorter work internally? We use an external KV store adopting the log structure merge tree. On the top, the memory buffer area is called map table. The rights from the foreground are initially sorted in the map table. When a buffer reach its size limits, the data is flushed to the disk forming SS table, which is the sorted stream table. All map table and the SS table are divided into multiple events. Levels, the layer that includes all map table and the SS table directly flushed from map table to the disk is called a level zero. As sorter needed to sort incoming change events based on their timestamp. However, different disk files may have overlapping timestamp intervals for an SS table. So an iterator needs to wrap to the merge sort this files when accessing them. Managing memory and disk files for hundreds of thousands of table can be challenging. So multiple tables may share a fixed amount of memory buffer and then flush them out to disk together. This can result in file and disk having overlaps not only in timestamp but also in tables. To address this, the files can be periodically read, merge sorted and rewritten back to the disk. Once the events has been sorted, the mounter converts events with a timestamp smaller than DDR barrier into table role format. To do this accurately, the mounter require knowledge of the corresponding table schema. Although the owner is responsible for pushing the DDR events to the downstream, the processor must also store the table schema to achieve the processor DDR puller retrieved the table schema from Type AD CDC components. Filter out unnecessary information and store the table schema in the processor schema storage for later use. Next, the sync components in Type CDC is responsible for transferring change events to the downstream. Each change feed has a resource manager, a resource manager that connects to the disorder of all tables associated with the feed. The sync manager periodically pulls the source manager which retrieves events from all orders and sends them to the sync manager. The sync manager then distribute the events to the corresponding table sync which writes them to the designate downstream. Type CDC employs a pool-based sync to ensure efficient and uninterrupted event transfer. Push-based sync can result in block events on the sync, leading to performance degradation. Type CDC's pool-based approach can manage a large volume of data with precision and consistency, guarantee accuracy throughout the data synchronization process. So while Type CDC is theoretically horizontal-scalable, certain performance issues such as high CPU utilization and unexpected lag between the upstream and downstream may still occur. We received feedback from users last year about the concerns of CDC's performers. In response, we have put significant efforts in a time to optimize the performance of Type CDC. So starting with 6.5, the throughput of Type CDC has been improved by seven times. In the following section, we will discuss the approach that has been taken to enhance the performance of Type CDC. First, let's check out some performance baseline of Type CDC before 6.5. I put some odd number here because I want you to have a more intuitive feeling of how much performance gain we have already achieved. The first case we consider is using Type CDC to sync data between two Type DB clusters. The hardware specification of our Type CDC node is 16 CPU and 64 GB of RAM, which was proven to be more than if sufficient for our case. Single-table throughput weighs a big table, 1,200 bytes per row, is 80,000 writes QPS, weighs a throughput of 120 megabytes per second. Even before 6.5, we found that Type CDC is able to handle very large tables, up to 30 to 40 terabytes in size. And Type CDC was able to keep up with the throughput demands without any problems. And there are no limits to the amount of upstream cluster data that Type CDC can handle. This means that as our data needs continue to grow, we can rely on Type CDC to keep up with the increased demand. The second case we consider is using Type CDC to sync data from Type DB to Kafka. Still, it is the old data before 6.5. The hardware specification are same as previous case, which is 16 CPU and 64 GB of RAM. When testing a single-table throughput on a large table, still 1,200 bytes per row, we have achieved 35,000 writes QPS with a throughput of 52.8 megabytes per second. While this is a slightly lower than our previous test with only two Type DB clusters, we have also found that Type CDC is able to handle very large tables. We have a test table up to 30 to 40 terabytes in size, and Type CDC was able to keep up with the throughput demands without any issues. So, the throughputs of Type CDC table pipeline is an essential performance matrix that measures the efficiency of each of its four consecutive subcomponents, the puller, sorter, mounter, and sync. This pipeline can be visualized as a water pipe, where the narrow list part determines the overall throughput. So, by measuring the throughputs of each component separately, we have identified that the sync is the bottleneck in most cases, which has the average throughput 76,000 while the sorter has the largest throughputs. Therefore, to improve the overall performance of Type CDC, optimization efforts have been focused on sync, puller, and mounter. To optimize the performance of the puller in Type CDC, several measures have been taken. Firstly, the processing of resourced time stamp has been optimized by processing them in batch, which resourced time stamp you can treat as the watermark. Reducing the time taken for the puller to calculate, this measure helps us to reduce the time taken for the puller to calculate and advance the checkpoint. Secondly, we rewrite log have been implemented instead of using just exclusive log, enabling concurrent access to resource and increasing the efficiency. Lastly, the frontiers inspection of region split merge has been optimized to reduce the time taken for the puller to access the update data. An unnecessary memory allocation has also been removed, which help us to further improve the puller's efficiency and reducing overheads. So the performance, so here is what we try, how we try to do the mounter improvement. So as you can remember that mounter acted as the deserializer decoder in our pipeline. So once it will receive a DDR event or key value entry, it will try to decode it and make it like a table format and send it to the next components. So in order to improve the throughput on the mounter, we try to use a decoder pool instead of just a single thread decoder to decode the key value events. So the performance of the sync components in TICDC has been significantly improved through the implementation of a pool-based sync, as I mentioned before in the previous slides, which help to improve the management of the event flow, reducing the risk of data log on the sync. So we achieved a significant performance after applying a formation optimization. A comparison between TICDC 6.3 and TICDC 6.5 reviews after actually it's the version larger than 6.5 reviews a substantial increase in performance when a downstream is Kafka using different protocols. When using a Canon JSON protocol, the performance has improved from 5,000 per seconds to 41,000 per seconds. When using an open protocol, the performance has improved from 8,000 per seconds to 58,000 per seconds. And finally, when using an Avro protocol, the performance has improved from 9,000 per seconds to 63,000 per seconds. So specifically to address the checkpoint lag when syncing data to MySQL using TICDC, we apply a batch-syncing approach. Rather than executing events one by one, events are now synced in batch, and multiple connections has been established between MySQL sync and the downstream MySQL-compatible database. So to measure the effectiveness of this measure, a sysbench workload has run with 500,000 rows per transaction. The results show that this measure has led to a significant reduction in the update and the delete lag by about 30%. And lastly, I would like to share some of the valuable lessons we have learned. Throughout the journey of building TICDC. The first lesson we have learned is to decide the system architecture in align with the upstream database. This means that when creating the data pipeline abstraction, we should closely align it with how the database organized its data. Instead of focusing on adapting the system to Golan channel, we should prioritize the table pipeline approach. The second lesson is the importance of establishing clear boundaries between subcomponents. For a significant period, we lack precise regulation and scope for each subcomponent, making it challenging to optimize or enhance the system. Any change often requires modification in multiple places throughout the entire pipeline. Hence, defining clear boundaries is crucial for scalability and demandability. Careful consideration should also be taken into account when choosing between push model and pool model. Previously, we utilized the push model for sending data across the pipeline, which made it difficult to isolate each table pipeline. However, by transferring to the pool-based sync, we gain more flexibility in supporting different downstream systems and efficiently syncing data from various tables. The third lesson involves implementing the old-value features from the early stage of the development. In the initial version of the high-CDC, only put and delete events were synced due to performance concerns. However, we discovered the usefulness of the old-value features in many scenarios. After months of struggle, we finally decided to implement the old-value cache and fetch the old-value from Tikev using historical snapshots. And then the final lesson resolved around implementing an efficient sorter while considering scalability. Initially, Tikevc used an in-memory sorter, which increased the risk of OOM, especially when dealing with large volume of changes in the upstream cluster. To eliminate this risk, we engaged in discussion and eventually adopted an LSMG-based DB sorter. This feature took approximately half a year to mature and was designed with scalability in mind. Alright, so that's all for today's talk. We'll keep adding more features and optimizing the system. You can find details for advanced features like large table scale-out, which will split the large table syncing process into multiple table pipeline in our official documentation. For the GitHub repository of Tikevc, feel free to check out. If you have any questions or comments, just show me an email. Oh, okay. And also, you can scan the QR code and provide the feedback for this session. Thanks. Any questions, comments? Yeah. Could you use the microphone after you? I think you mentioned about filtering capabilities. Initially, when you're explaining the Tikevc, where do you put the filtering part in that architecture? The filtering, right? Yeah, like if you want to selectively get records. Correct. That's a very good question. So the filtering part is very important because users usually have huge amount of tables. So to ensure that we are now going to transfer unused data from Tikevc to downstream, we put the filter part on the Tikev side. So from the very beginning of the pipeline, we are going to filter out all unnecessary events and the tables. And just one small follow-up. On the sorter, is there some kind of a configuration as to how long that sorter waits for all the events to arrive from the change or how does it know when to trigger a sorting? So your question is about is there any configuration we can use to configure the trigger of the event pooling? Is that correct? No, no. You had that sorter component in the middle, right? You were saying that sometimes the events may not arrive in the right order and the sorter in the middle is going to reorder those events before it sends it to the sink. So my question is that sorter component in the middle, how long does it know that, you know, this is the trigger at which I'll have 100 records, for example, and I need to reorder them and send it through. Or is it a time-based threshold or is it a record-based threshold? Yeah, so I think that is I don't remember the exact number of the frequency we trigger, but I believe that is configurable. We can config how frequently or how often we wanted to push the sorted events to the downstream. Yeah, I think that is configurable. We can double-check the configuration file. But that is a very good question. Thanks. I have a question about the consistency guarantee part. So it looks like Thai CDC pulls single row level changes from Thai KV and you said it ensures eventual consistency. So it doesn't mean that if there is a transaction that involves multiple row changes, does the asset property is still guaranteed by Thai CDC? I don't think so. Currently we do not support transaction level consistency. So it's difficult to achieve that because the row or the data can be across multiple Thai KV nodes. So we do not recommend users like on a downstream database you try to sum your transaction on an upstream database and then you will receive the same kind of transaction on a downstream database. Yeah. One more quick question. I just want to clarify the puller's working mechanism. But one follow-up is when we're syncing the data to the downstream database we are actually submitting the MySQL events to the downstream database. So if we really wanted to implement the transaction consistency, it would be very simple. So there is some price in the trade-off. If you wanted to do that the throughput and latency can be low and latency can be large. That makes sense. One quick question I had was about the puller's working mechanism. So the puller's pull the changes from the Thai KV periodically or not Thai KV pushing changes from the internal container. So it's like the stream. So Thai KV have a change. So inside the Thai KV we have a sub-components that is watch the change lock. When you receive the events it will sort of have a stream push to the puller. Even though this one is called puller that is for Thai CDC itself. It looks like a puller pushes those changes into the entire Thai system and then pulls from there. That makes sense. Thank you. More questions? Great. So I guess that's all. Thank you.


Okay, welcome everybody. This is Sartreau with LSNTOW. I am here with William Guyton, and today we are discussing the LSE baselines. William, would you like to introduce yourself for a second? Sure. As Sartre said, my name is William Guyton. I am currently serving as the Director of Information and Technology at Legal Aid Services of Oklahoma. Excellent. So, a few quick things about the webinar here first. Number one, logistics-wise, this is a smaller group. I am happy to promote anybody up to a presenter so they can talk or unmute them. The default here is for individuals to be muted, but since this is a Q&A, feel free to type something to me either in the question box or use the raise hand function, and I can unmute people. Also, if you have questions any time throughout this, there is a question box. I will be monitoring those, and we will be answering any questions that are in there. Second thing is that this is being recorded. We will be posting this to our YouTube channel. Our YouTube channel is accessible. It's in the upper right hand corner of LSNTOW.org. We've got over 200 videos that we've done, including a whole series on security that we worked on last year with Tech Ground Table. All of our past trainings are made available to the public for free. In the handouts, there are five handouts for this webinar. There is the LSC Technology Baselines that were updated in 2015. There is a call center triage. Let me actually pull up a copy of that here. There are four guides that were put out in 2018, and I think that these are great supplemental readings for the baselines. One of them is on knowledge management, putting together things like a brief bank, being able to share knowledge inside of your organization. The second one is on information security. It's a tool kit that covers a lot of the physical security and digital security items out there, mostly on the digital side of things. We just did a webinar about a month ago on it. We have one on online intake and triage. We've done several webinars on this topic, and then we've got one on call center technology. Each of these are about 20 to 40 pages, and they go much more in depth on technology, and they were each published this last year. The baselines were written in 2015. These four guides work as very solid supplements to those baselines. The baselines that were published in April 2015 were an update of the 2008 baselines, and they had significant input from the community. They go into a lot more detail also compared to what we're going to discuss today. We're just going to go over a few of the basic areas, and then hopefully get some questions from the community with regards to each of those. Back up the slides here, presents. One of the major things in the baselines is how to work on the long-term technology planning and how to focus on the strategic aspects of hardware and software, management for client data, production and supervision, legal work, records, management and information management, intake, and telephonic advice. William, what is some of the stuff that you've worked on that you think is kind of essential on that intake or the ability to serve clients on the phone side of things? Well, my first real introduction to call centers was post Katrina in Alabama, where we spun up some call centers in 2005-2006 specifically to handle the volume of folks that we knew would have Katrina-related legal issues. Alabama got into online intake and call center as a mechanism pretty much by accident because we had to. The most important thing that we learned in developing that process was it really does take a cross-functional team. It takes leadership, it takes attorneys, it takes intake staff, it takes paralegals. We had one or two of just about every staff member in the program in Alabama involved in that process because at one point or another each one of those types of staff are going to have to touch that online intake or do something with that call in the call queue. This is my 15th year in legal aid and I've watched it go from a couple of different iterations of physical intake, online intake, call backs, call in. I think we're still experimenting on the best mix and match of those technologies and techniques as it applies to quality client service and we're doing that in Oklahoma. We're experimenting with our legal server integration for online intake and again doing some interesting things with call distribution using Amazon Connect to route calls across the state to try to deal with the volume that we know we're going to get. So there really is no one size fits all solution. It's a state by state or program by program solution based on technical capacity and capacity in general but there's probably not much of anything I haven't tried or seen in the last 15 years. So in kind of on the disaster preparedness side of things, what have you learned from that experience of that could be done in advance that would have made things much easier in the future? In terms of Hurricane Katrina, well there wasn't, I don't know if there was a whole lot we could have done because we didn't really know which way the government was going to be shipping people out of Mississippi and Louisiana. So Katrina impacted the entire southeast. There was a bunch of folks that got shipped to Houston and parts of Texas, so Texas programs got impacted. There were a bunch of folks that got shipped to Alabama and got shipped to Atlanta. So until we knew where people were going to end up going, we didn't really have a good idea of what to expect. And then you've got the long-term knock-on effects of that kind of disaster. In terms of foreclosure and family law and all of these knock-on effects, we were still seeing cases three, four and five years down the line after Katrina because of just the knock-on effects of those legal issues. But in a statewide program, at least from a hurricane perspective, you can try to put assets across the state if you're a localized program in New Orleans and there wasn't anything you could do. So we didn't really have a sense of how many people were going where until FEMA said, hey, we're shipping 50,000 evacuees to Alabama. Now that's a very good point. In being a statewide program, you get much more of an opportunity to be able to geolocate resources across the area. And in Washington state, we have put our phone system and our email system both on the east side and on the west side of the Cascades here so that we've got some redundancy in those systems and the ability to keep it going if there is something on each side. Next area that we've got here is on staffing and this, I was part of the process that helped discuss these and this was one of the more contentious areas in just making sure that as your organization grows that you've got enough IT staff. When comparing this to what is out there in the general nonprofit kind of advice area, I would still say that the staffing here is a little bit light in having two FTEs per hundred individuals. Those FTEs do not necessarily need to be in-house. Some of that can be an external help desk. But additionally, making sure that you also have someone at the strategic level of the organization that is looking at your longer term policy things that they're reviewing what your data retention policies are and data destruction. That's something that the baseline doesn't currently cover but there is a session coming up at the MIE conference and right before, I believe it's called ITCon now, the new TIG conference in New Orleans and data destruction is something we're going to look at there. But making sure that you've got enough essential staff to be able to answer people's questions, help them with a bring your own device policy, that type of stuff and as your organization grows or if it merges with other organizations in the state making sure that that IT staffing is there. So William, how do you handle the IT staffing there? Well, we're 180-ish staff and so we're currently at three and a half FTEs and that really is based upon the fact that people want to go on vacation, people get sick, people have kids, you always want to party on.  We're large enough and diverse enough and have enough people across the state that we're lucky, we're probably above average in terms of bodies but we're doing a tremendous amount of technology in Oklahoma and no one of us knows it all so we know obviously we're multiple halves and we've got some cross-training and some redundancy built into it so if I want to go on vacation I can do that but we've lifted the entire program to the cloud so even if I'm on vacation I'm not really on vacation in fact there's a few people in this day and age that are really on vacation unless they want to turn their phone off but again it's one of those cross-functional conversations you have to have with leadership and it's nice that the baseline is at least addressed because I think on the first one I don't know that it did. Right, the first one did not cover staffing issues in this type of detail. Additionally we've got a question here, is it common to have a consultant in addition to in-house IT? I'll answer that first. At Northwest Justice Project we've definitely used consultants for particular projects and at other programs that I've worked with especially smaller programs they will if they've got one person who is in-house their second person is often a consultant or a help desk company or someone else that supplements what that one person does. I dealt with a program about five years back that was about a hundred person program they had one IT staff and that person just left and they had no idea even how the infrastructure was built how they could support that type of stuff and it became a crisis for them and they ended up spending a lot more money on consultants to come in and deal with the crisis of that individual leaving. Having at least two staff members that are aware of the IT infrastructure, the strategic plan, the password management system I strongly recommend LastPass which individuals can do for free. I use an enterprise version for LS NTAP and everybody who works with me which allows me to instantly like pull all of the passwords from somebody when they leave for all of our websites that type of stuff but just having at least two people and even if one of those if you're a smaller program is someone external consulting that knows what your strategic plan is when you need to rehire that tech position it just goes so much smoother. Great question there. Yeah absolutely. Yeah we've those of us that have done this long enough have all been in that same position. What's the router password? Where does that go? But absolutely. We use Dashlane for the same reason and you're using LastPass and we've looked at LastPass but not only do you get visibility into other secured passwords for critical infrastructure and that kind of thing. For us we use it as a tool to enforce our security policy which is you need to have a unique strong password for every website in every account that you visit and we've encouraged staff to use Dashlane for their personal use because invariably their personal account will get breached and they've used that same password at some other lasso account therefore we've been breached. So again we're trying to use the tool to raise all boats and to make it convenient to do the right thing in terms of password strength and that policy aspect of it. Right and I would definitely recommend with regards to the password stuff just do a training with and help staff set it up even setting up a personal account because if they don't use it personally the chances on them reusing passwords is really high. Yeah. Spend the extra time and help them create good security practices in their personal life also. Absolutely. So budgeting there's really three main things that the baselines hit is ongoing maintenance and upgrades of hardware and software personnel consultants and training in use of technology on the maintenance and upgrades. I've dealt with several different legal services organizations that kind of had the well we wait till somebody complains about their computer or until the computer crashes in order to save money but the loss of productivity of waiting on those computers is really really high. Most programs that I know have somewhere between a four year and the longest that I've seen as a six year replacement policy for desktops and or laptops and being able to rotate those through on a regular basis and not have to deal with crisis is when they crash can definitely help you a lot. Another question here that was back on the staffing is it common for an organization with 100 plus staff to have software hardware maintenance handled in-house? A quick way in how Northwest Justice Project handles that is that some of our some of it is in-house and some of it is external. So for example with our call center software and when we were using SharePoint before we moved to 365 when we had it on site we had external individuals who maintain those pieces of software and the rest of our software was in-house. SharePoint just had we had a lot of customization and it wasn't a skill we had in-house. When we moved to 365 we basically dropped the customization and in a lot of different both in using open-source tools or closed-source tools I strongly recommend limiting the amount of customization that you put into it so that when upgrades come along you're not beholden to a consultant or an external coder to make your customized system work with whatever the new standards are. But almost all of our other software is in-house although that has been recent. We are probably 150 almost 200-person org before we started to put all of that stuff in-house and we're at about 230-240 currently. Yeah and and we did the same thing we've we've we lifted the entire program to 365 Azure so we've reduced our on-prem footprint by probably 95%. So the maintenance there's not any maintenance the upgrades there's not any upgrades you let Microsoft do that. Your budgeting is fairly straightforward because you're paying a per user per whatever service that user has per month type of type of budget so the budget should remain relatively constant unless you've got a project or a program that you're deploying. So our so our cap X has been flipped into an op-ex in a big way so we're not having to save $100,000 over three years to do a big server upgrade like we used to do you know five years ago. So the the simplification of all of that is has gotten has gotten very very good with 365 and Azure. So that really has a positive impact on the budgeting obviously and it has an even bigger impact on the maintenance upgrade side because you're focusing on very specific things to your legal aid program versus exchange and active directory and you know all the other things we used to do five years ago. Yeah we we evaluated going online five years ago decided to stay in house we are redoing the evaluation now. The online tools have continued to significantly change over time so I think looking at those cloud solutions and how much you're paying in-house to maintain those things is definitely worth it. On the training side of things here I recommend trying to put technological training as part of your regular like CLE schedule throughout the year and pair the tech training with another training that individuals are going to show up to. We've tried to do like here's just a tech training and the attendance just dives but if we can get people coming in for something else and add on 10 or 15 minutes of tech training at either the beginning or the end of another substitute legal issue we get much better attendance and we just try to keep it short and practical. We don't want people sitting around for an hour or two listening to security standards or something. We want some identifiable short tips that people can use and so they feel comfortable asking the silliest questions possible like just treat them well answer the questions help them out because when your staff feels comfortable in being able to ask those questions they're more likely to ask you about that weird email that's asking you to click on stuff or when their computer is running slowly then they'll come to you and figure out what's going on so just creating that rapport is very very important. Absolutely big time you know ultimately the technology department in legal aid is a support organization you know we're about supporting staff to advocate for clients and you've got to remember your role in within legal aid organization for sure. We're in the same vein we find ourselves having to do the same training over and over and over again depending on which office you're in. We've taken to developing very short videos that we upload to the video section of SharePoint and so if you went up to our video section of SharePoint you'd see from very basic stuff to some fairly advanced stuff how do I log in what happens when I log in how do I load Skype where's my headset you know we took it from an onboarding perspective because we were we were redeveloping our onboarding process so we looked at it from an onboarding staff member's perspective and you know what are the what are the five things you need to know when you sit down at your new job at legal aid you know what are the five people I need to reach out to and so we approached it from that perspective and it's grown kind of like the LSN tab video library over time where we record the CLEs now and the CLEs were up there by subject matter and by location and by date and it's just and you know we've we've we've made people comfortable in terms of the open source tools and you know webcams and headsets are inexpensive now and the ability to have a subject matter experts sit down and do a 10-minute teaching session on some complex legal issue is really impactful and because you've got it in the can you can point somebody to it and we're developing those on a on a on a vertical basis as well so family law will have their own section of videos you know those kind of things so we took the video approach just because we kept hearing the same question so we got a question here William would you be willing to share some of those onboarding videos sure they're they're not glamorous but there's no reason why you can't have access to them the you know at the same time we we do a a about a quarter about once a quarter we do a Midwest IT director's conference call so all the all of the directors in the Midwest part of the United States get on a call and do this same kind of thing that we're doing today and we've opened it up to just anybody so if you want to pick the brains or or talk about a specific subject within legal aid technology we're doing that on a regional basis because it's you know we know none of us know it all there's some really fantastic people and great ideas and we need to we need to leverage all of those ideas that's the great thing about TIG I steal most of my ideas from TIG well and if if there is a piece of technology or something that you're using in your program that you would like us to either help you create a video for or create a short five-minute video for that is definitely something you can ask LSN tap to do we would prefer to collaboratively do it so that you get the skills to do future ones but then we will take those videos and share those with the community we were working on one currently for last pass because we're encouraging staff to use it but that those type of short videos do not worry about the production cost it is more important to keep it short simple practical and that is going to go so much further it does not have to yeah we don't do we ten minutes is our max if you can't if you can't condense that concept concept down into ten minutes then you don't know what you're talking about and we use OBS studio we've got OBS studio loaded on a bunch of machines specifically to encourage folks to capture that because we you know time is limited and we all get the same question you know advocates the advocates get the same questions leaders should get the same questions I'm like take your expertise pull it down to ten minutes and and put video out yeah I'm gonna drop a link here in the chat to OBS studio I I've used it for streaming and for other stuff it's very very powerful open source software that allows you to do screen captures video captures it it's great definitely recommend it I got another question here how do you guys handle hardware maintenance at your program well we from a from a desktop perspective we have a 36 month four-year replacement schedule for desktops and we we just went through a major hardware upgrade in fact we've got two more offices to visit for the end of the year in fact I'm heading to Muscogee Oklahoma tomorrow to do to do phones and and security cameras but the it is sort alluded to earlier it's significantly easier to develop a replacement schedule than to figure out five years down the road that you have to replace an entire office with the machines and and the negative impact of on on that staff's product productivity and their morale is going to be in the toilet I mean you gotta you have to if you want good quality legal aid out of support staff for clients you've got to get a you have to give them the tools they need in order to be able to supply that high quality legal representation there's just no way around it and and we're so you know we're so technologically focused now at the court level at the legal aid level you know being without connectivity being without a tablet being without the right software is in a lot of cases a disservice to the clients we're trying to serve but we we budget we do and you know within that budget since we're on the budgeting page we budget for disasters you know what do we do if the roof blows off an office what do we do if an office burns to the ground what do we do if you know you you can't budget yourself to the penny and then not not be able to respond to some emergency because invariably over the course of a year there's some emergency at some office somewhere but that's you know you have to make it part of your and that's part of the reason the baselines exist is you need to be able to articulate what this document is recommending as a good as a best practice for a legal aid program and you'll have you'll have leadership that push back on that well we simply can't afford to do that and then that that opens up that conversation about well what type of service are we willing to give clients great and finding the practical impact examples of these things does tend to work better with the executive directors and your leadership staff when when you end up having PCs dying or things the impact on clients is really high so that's kind of a follow-up to that it was does your in-house staff in install software that type of stuff on new PCs over here at Northwest Justice Project and we've basically got have a five-year replacement program and wet and we do go through and install the software but we pick up a series of computers that are the same we set one of them up and then we replicate that setup on all of them so the the having a bulk process for doing that and having it as part of our annual cycle cuts down on the time and the cost of doing those because we're not loading up every computer and then installing one by one each of those things but yeah we do do it in-house yeah and we have a similar approach we're we're a big we're a big Amazon smile customer I mean most most everything 98% of what we purchase is is on Amazon smile and so we've done our testing we've got a base package this is what your your you know your new hire staff members going to get in terms of the assets here's the desktop here's the laptop here's the phone here's the headset here's the webcam software and the software in the OS side has gotten easier only because we're really digging into the in-tune aspect of Office 365 where we can start to automatically deploy and give staff access to the apps via 365 and ultimately if we continue to work with in-tune it'll have that same kind of impact on the security side in terms of mobile device management endpoint deployment and notification and those kind of things in fact I'm looking at conditional access so you can now put a policy in place in in-tune that won't give somebody access to email a certain baseline criteria have not been met so in-tune at a buck fifty the nonprofit price per user per month is a has turned out to be a real asset literally because we use it for inventory as well so so we're again we've done the whole Office 365 Azure thing so we've we've had a big helping of the Microsoft Kool-Aid and part of that Kool-Aid is is using in-tune to manage your devices so there was a follow-up question here is who handles a procurement for IT so at MGP we have a a budget overall of what we're doing and then our head system admin currently covers all of that with oversight from one of our executive team we are in the next year going to be hiring our first chief technology officer that those responsibilities or at least oversight will move to that individual so the long-term there there will be somebody on the executive team there it's currently overseen by a member of the executive team but there is an annual budgeting process and then also check-ins on that budget throughout the year yep we're I'm a member of the executive team so as a director level position I'm that technology representative at the table but at the same at the same time we have a CFO and we have a formal budgeting process and I'm the person that's held responsible for being over under budget on an annual basis and we I sit down with the CFO quarterly and see where our spins are you know I'm over on hardware I'm under on software I'm over on services you know so you get you get feedback and reports that show you where you are on your spend and that all ties into your technology plan and what do we want to do in year two and year three what do we anticipate as a big technology project they all tie together because obviously without the financial resources and also dovetails into consultants if you don't have the if you don't have the internal capacity to develop that particular solution then you may have to hire a consultant to help you deploy a particular technology solution so the budgeting and the technology planning and and the leadership all dovetail together because they all impact each other. I think you're muted Brian. So case management systems is kind of the next thing that we've got here we keep a list on LSN tab of case management systems the features do tend to change on a regular basis these are some of the major systems currently used although Salesforce should probably be added to this slide as I know a few programs have started to use Salesforce also the feature sets for the case management systems are also when they get upgraded for a single member of the legal aid community those get pushed out to everyone so we're having discussion I know one of these case management systems has upgraded recently to make it easier to do data destruction and there's talks about doing it for other case management systems but the the ability for staff to be able to access them from multiple locations your report functioning out there is very important and then the ability to share those case information that you need when transferring it or sending it over to other programs are some of the fundamental pieces that are really in there so William in looking at a case management system what are kind of the major things that you look at and what was a deciding factor in the case management system you're using well we've I think I've done three conversions now between three different CMSs so I've I've been through the good bad and ugly of just about each of these on the screen it's you know for us the the watershed moment was when you went from something like Kemp's or Pico which was really client server based and if you weren't in the office you didn't have access to it and there was a time when when we knew that we were gonna have staff representing clients that needed access to that data and really we spent a lot of time and a lot of money and a lot of effort trying to figure out how to take what is what was traditionally a client server architecture and virtualize it and put it up in the cloud and give people access to it and in most cases it didn't seem to work very well so we ultimately went with legal server primarily because it it it's specifically written for legal aid organizations which in and of itself is is unique although Kemp's did that and I get to that to a certain extent but it's it's it's biggest selling factor is accessibility it's a web-based application and more and more of the grants that we're competing for at a regional and national level require an embedded staff member and you you're not going to be able to support that embedded staff member in that hospital or that domestic violence shelter the way you support a staff member that's physically in one of your offices so we had to we had to think about you know how do we support a staff member that's not physically in an office anymore and that really drove us to to to legal server and the office 365 which gives everybody the ability to get to everything they have regardless of wherever they are which is a which is a which allows a lot of people to especially in Oklahoma you know we have some pretty dramatic weather events and invariably we've got internet out at an office somewhere in the state probably every day so that so from a disaster recovery capacity but more importantly from a from a staff accessibility capacity that's what really sold us on legal server yeah we we are also on legal server part of it is that our whole state is on legal server all of our volunteer lawyer programs and the other legal services orgs which has made it easier to transfer files I do think there is a need for a case data standard that was interoperable between all of them because the lock into a particular system I think is a long-term it prevents kind of competition and others entering that marketplace that's not something covered in the baselines but it's one of those things that could would be useful another question here is who does your RFPs bids solicitations etc so at NJP we've got a policy where over $25,000 we have to do a competitive bid process we write the RFPs in-house we often ask the community on the LSN type email list does anybody else have an RFP here we occasionally also actually in our last two RFPs that I've been working on we run those RFP also by the TIG staff because they've seen so many RFPs they're willing to just make improvements give you feedback on an RFP to help improve those and then we post them publicly we've got a RFP board and job board that was launched in the last six months on LSN type where anybody can post an RFP let me see about bringing up a copy of that but it also it covers jobs also additionally we then send it out to an email list that is everybody from the LSN type community and then we contact vendors one-on-one and we look for email lists or community groups that deal with those things so with our last Drupal upgrade we reached out to a bunch of different Drupal developers and some Drupal email lists to kind of share that broadly how do you guys do RFPs bids solicitation that type of stuff it's a similar approach for us we don't have a lot of projects anymore that that hit the RFP threshold only because a lot of what we're doing is a per month per user service type of charge and that's probably a consequence of being in the 365 Azure orbit there's not a single huge $30,000 project anymore you know most of those were done several years ago like the case management conversion and some other things we did in the past we just don't see a lot of very large dollar-denominated projects anymore they're really just made up of you know clicking this box and saying we'll sign up for N10 for 30 days and evaluate it and clicking this box and clicking this box it's we've decided what our technology architecture is going to be which right now is Office 365 and Azure and we know what our base hardware and software package is going to be for a given staff member so most of those you know there's not a whole lot of wiggle room at least in the last couple of years because one of the one of the major benefits of a platform like 365 is we've been there have been significant incremental improvements in all of the 365 services over time that haven't cost us any more money so it the way that you the way that you budget and plan for technology in a cloud-centric environment really doesn't lend itself to those large denominated projects anymore if that makes sense definitely the yeah moving things on to a per user or monthly or yearly basis definitely changes kind of the threshold there although even when we're picking up services like that although we don't go through a formula formal RFP process we definitely do an in comparative analysis point between what's available out there and that type of stuff so I think still having a process that looks at the features looks at the cost and looks at the impact and compares it is very very valuable up on the screen right now I've got the job and RFP board so if you're looking to hire more tech staff or if you have any RFPs out there please send it to us along with a close date we will put it up there we'll share it widely and then we will also remove it when it's done but we're we want to help share those things as broadly as possible the next area that we've gotten the baselines is the basics around supervision it goes into a pretty good amount of detail in the baselines but kind of the five things we pulled out of it here is calendaring that you need some type of software to be able to do that and share that and let people know when people are out that type of stuff a document production timekeeping most of our case management systems have that integrated I know a few programs use external timekeeping apps though supervised data and then online research and there are a lot of different approaches that I've seen programs take some of them go and entirely open access free tools that are online and train individuals on that which often takes a little bit of training because law schools typically don't teach how you can get access to pretty much all of the case data out there without going through some of the other services that are out there others do it with a Westlaw or Lexus or one of those type of accounts William do you guys use anything on the software side for supervision data for checking in on people seeing where cases are is that all in your case management system or how do you do that yeah those are reports those are canned reports within legal server although we are talking to Ed about some of the dashboard management tools that he's talking about he's got a tag to do some of the dashboard stuff that we're interested in in looking at have you seen what Ed's to grant yes yeah that's an interesting one I we did a short highlight on it and hopefully we'll be able to do some more yeah electronic records this is one of the areas that the baselines covers a little bit says you need things like a document retention policy policies regarding access which is really part of your security also and confidentiality the one area that's kind of missing and I think that whenever next update is we will definitely have a section on is kind of what exactly your digital destruction looks like slightly covered in the document retention but there is a need in the community to find better ways to get rid of pieces of data that you no longer need if you've got a 10 year old closed case you may need a client's name with regards to a conflict check but still having pieces of their medical record or personal information related to that case the longer that you hold on to it the more likely it is to end up in a data breach and I wish there was a way that I could tell you how to prevent data breaches but they're going to happen it's a matter of time no matter how good your security is we can do a lot of things to cut down the likelihood but one of the best things we can do is regularly purging data that is not needed that you don't have a business case or a client responsibility to hold on to what types of policies or procedures with regards to electronic records do you have William we're there they're they're being co-developed as we deploy in tune and some of the compliance tools that are built in the office 365 we're starting to have that conversation it's we're getting down to the point where we're having to tag data and qualify data and in based on those tags and qualifications we get to determine via policy in 365 who has access to them do you have the should you have the ability to print this data or download this data should you be able to email it forward it to somebody you get a you get a lot of digital rights management ish type tools with some of the security suites built into office 365 a lot of that's coming from our friends in Europe because of the data standards they've enacted and but there is a fairly as clear as it can be approach that Microsoft maps out in terms of the like the lifecycle of a particular piece of data it's a work in progress it's it's a heavy lift and we're still getting the infrastructure and the training and the policies in place to be able to do it well but it's it's part of our data architecture that we're working on excellent I know that John Griner has is working on a it's not model policies but it's more things that you a bunch of questions you should ask for creating particular policies that should be out early next year we did kind of a preview of it yesterday in a webinar but once the kind of toolkit there is done we will do at least an hour hour and a half long discussion around that topic overall also there the pace that we're going we're probably not well given 10 hours I don't think we could hit everything that's in the baselines but if there are questions even outside of the particular area that we're looking at currently or comments please drop those into the question or comment box we are definitely happy to cover different areas on the knowledge management side this is an area where the baselines cover a little bit we're looking at about a page two pages of some simple tax but this is somewhere where I really encourage individuals to pick up the handout that is I believe it's a 33 page document on knowledge management covers everything from brief banks to sharing letters to anything where you're trying not to duplicate work or you're trying to share internally the knowledge that your staff has over particular legal issues I think what what William has done with creating some short videos from experts so that if you've got an AmeriCorps fellow or you've got someone coming in that they can get that information as soon as possible and know who the expert is in your organization that is essential the particular software or hardware that's used is less important in my mind than the community management aspects we've launched a SharePoint site to be honest half of our org uses it very very actively and another half doesn't use it at all and the big difference there is the the portions that have used it there's been someone in our telephone hotline area or in the practice area that has really taken it on and decided to separate it and talk to people about it where in the other areas no one had really championed that and if you can find some individuals who are willing to champion that the the management aspect of finding those community managers and giving them positive feedback and showing them how useful it is I'd say it's more important than whether you build something on Drupal Drupal whether you use a Google search appliance whether you use SharePoint there are several other tools out there the tools all at this point have very good search that can be put into them but it's making sure that good documents go up and that only good documents stay in that and that individuals put into their daily workflow sharing that knowledge that they're creating for clients any tips on like putting together knowledge management systems or things that have worked or things that have failed one other thing I have seen knowledge management systems that tend to go outside of your organization do tend to have challenges when it comes to redaction confidentiality that type of stuff putting those together is much more challenging than doing it internally next area that we've got here is statewide websites and we've got several different ways that those have been done in smaller states we see a very basic website that is often built on something like D law which is a free open source distribution of Drupal I've seen a few particular programs or even parts of programs create a WordPress site for particular issues I've seen that happen definitely in the veterans community from time to time or for particular service areas pro bono net also provides some very robust statewide websites and urban insight on top of the D law template does a hosted version called open advocate that several different programs work with putting together a way to manage what goes into or comes out of the statewide websites and when they get updated is extremely important we've got basically two fte's in Washington state that update the website one of them is more on the tech structure side and the other one is a content editor that creates that content one of the things that isn't heavily covered in the baselines but I think should be added is making sure that whatever you're putting out there on the website that you have client user testing as part of the process both test can it be found online and also does the materials that you're creating create the knowledge in potential clients that you want it to and we did a bunch of focus group testing on some videos for part of a take grant and we found that there were a lot of what we thought to be very plain language things that were still embedded in legal terms and that the that we could significantly improve those videos by going through a round of user testing any thoughts with regards to statewide websites one of the areas that the that the new baselines added that was not there before is the legal information social media side of things and most programs at this point have some presence on social media I definitely recommend short videos for YouTube for common questions that come up a lot of individuals are going to be seeking assistance outside of the business hours of your hotline or of your office and YouTube's available 24 7 Facebook also now has an auto responder option if you've got a Facebook page that is an area where we get a lot of people asking us questions we put together an auto responder that sends them to our online intake we did test a chat bot that would interact with that that would try to serve them resources from our statewide website also dependent on what it thought their legal issue was that technology was a little bit clunky but has a lot of potential if you can go to where your clients already are instead of making them come to your statewide website your you're able to reach a lot more individuals overall and going to the clients is really one of the things that the baselines looks at when it's incorporating social media so it's not just a broadcast that we've got a clinic here or there's this informational thing going on or here's where our offices are but also providing a way for them to get access to yourself help resources through those is definitely beneficial overall there's a little bit in the baselines over pro bono support but integrating that into your technology systems is definitely essential and then we talked about this a little bit in the budgeting area but those continual trainings are also essential making them part of what you're doing. Additionally, one of the areas that we've seen a little bit of a lack from time to time is the training for tech staff also most programs do have a standard area where they cover CLEs continuing legal education for legal staff I would although it's not required for tech staff to keep a license it is essential for them to be able to update their skills send your tech staff to one or two tech related conferences make sure that they have access to online tools where they can get education and access to community to update their skills they will be much happier and much more useful as they update their skills and that should be part of the training I recommend considering shadowing what you offer for CLE training to your tech staff as part of a budget for them put that in that for each person we've got X number of dollars for each tech staff member to be able to do this type of stuff and I would also rotate tech staff through either the equal justice conference or the technology conference from LSC because both of those conferences have a dedicated technology legal aid side to it the tick conference is obviously all tech but equal justice conference does have at least one and sometimes two tech panels on every single spot so very useful for tech staff also on the security side once again I recommend the 2018 security toolkit because it goes so much more in depth into this but the four major areas that the baselines cover is maintaining backups updating your software educating staff and then limiting permissions and access and especially having a process in place for when all of your interns go when they leave when we bring in interns or fellows or pro bono staff not always a pro but at least on the ones where we have a limited time we get that end date at the beginning of the process and we set those calendars that we're cutting off permission or access at that date we will send an email to their supervisor at that point in time and make sure that that person has an extended because about 10% of the time somebody stays on or they still got an active case or something but having that proactively from the beginning eliminates the outstanding accounts somebody forgot to tell you so and so was leaving I would also consider having kind of a yearly audit where you go through and double check all of your accounts and make sure that they're still active staff because you'll find some accounts out there that we're just missed for one reason or another now we talk to this is an interesting one so William how do you guys cover bring your own devices you know I sound like an ad for Microsoft 365 because we're we're testing and deploying in tune almost specifically to manage the YOD the we we encourage staff to use their own devices to bring their own devices to connect to last resources by any device they want to feel comfortable with or have access to but that's a double edged sword obviously so part of what we're doing with in tune and the baseline policies that were that are being enforced by in tune go directly to setting some baseline standards for all the devices that we have access to so you know so the the Android devices the iOS devices the Chrome OS is the Windows devices there there's a policy and sub policies for each of those devices so we're we're beta testing the rollout of iOS now we've enrolled a bunch of people the endpoints on their device and it's only on their device because they downloaded the outlook app and connected to their mailbox and when they did that the endpoint got put on the device where we could actually audit the device to help us set our baselines for each device type or each operating system so that's that's the approach we're taking is we don't want to limit anybody from accessing the data they need in order to advocate for clients but we do need to have the ability to set some baselines and and to enforce some policies just from a risk reduction perspective or with regards to Northwest Justice Project we have developed a policy I will share that out a little bit more broadly we do have the ability to remotely wipe individuals who have installed the outlook app if they are just accessing their email though through a web browser on their phone we do not we strongly encourage people to both install the Skype app so that they can do calls from their cell phone that go through their regular phone number instead of their personal cell phone number and the outlook app so that we have that ability we don't sit down with everyone and double check that but that is definitely what we encourage out there and additionally with regards to laptops we do have a policy that if you are going to be doing anything that accesses client data through your laptop that we set up a VPN for you and we approve those on a case by case basis seeing that our case management system is accessible via the cloud we also have a separate policy for telework and for telework we require an antivirus as part of that a computer at a certain level and IT staff to have the ability to kind of review what that is I've seen some programs that will just provide the computer to use in that remote work situation as if it's a home situation I think that gives you a lot more control it does occasionally mean that someone has two computers at home one that is their personal computer and one is the computer that they are using for case related data but I think that long-term that is the better policy because it gives you a lot more control and you don't have to worry about external software or other things that individuals are doing with their own devices. Backups we've definitely seen a lot more organizations moving towards the cloud over the baselines do not specify whether you should do onsite or the cloud either way that you do backups it is essential that you test these on a regular basis we've definitely seen some problems with some of the case management systems in the past where they've gone down and the backup wasn't able to recover stuff backup was sitting there for five or six years and had never been actively tested I think you have to go through a process of making sure that the whatever system you have in place that it works get your IT staff together take a Friday or take a Saturday and pull everything down and try to pull it all back up and see what happened or see if you can pull up that backup to it. When I took over LS NTAP we were using a hosting company that literally had all of their backups in one closet and they lost everything that we had early on. We have switched entirely to very professional hosting company that has a backup service and we've tested it since then and we have lost no data in the last seven years but yeah any other advice with regards to backups. We talked to a pretty good amount about training. There's a bunch that is covered in the communication policy although there's also this is one of the areas that is really important especially when you include social media have a system in place for what is posted what isn't posted and who will review something if you have any questions because as you empower staff to be able to share things which is very, very useful just make sure that those processes are in place. This is one of the areas that I know the least about I've only dealt with the one telephone or the two telephone systems we've had here at Northwest Justice Project. We had an on-site telephone system that was literally dying when I got here and we replaced it with a Skype system that has had some minor hiccups but once we got through that has worked extremely well for us. The toolkit that is on here for call centers does cover a little bit about telephone systems but it is covered heavily in the baselines. William do you have any advice on telephone systems or? Sure. Well I was in a similar situation both in Alabama and in Oklahoma. The same situation that you found yourself in I found myself in here where the PBX has been hanging in the D mark for 10 or 12 or 15 years and literally pieces of them were failing. You'd lose the voicemail card on the PBX and you wouldn't have the ability to leave a voicemail anymore. At the time I got to Oklahoma that was pretty much the situation in every office. It was the same PBX, it was the same handset. They were about the same age. They weren't under contract, they weren't being maintained and so there wasn't really a good on-prem solution that we wanted to go with which again really pushed us to the cloud as a solution and then it became which cloud VoIP provider do you want to use and we spent a good six months testing about a half dozen VoIP providers and initially went with Dialpad which we still have some offices on Dialpad and it's a great product and a great service and it's competitively priced and it's been reliable but it doesn't have quite as tight a level of integration with Office 365 that Skype does. So we evaluated Dialpad versus Skype and decided that we really liked the tight integration of Skype for Business with Office 365. So we've begun to deploy Skype for Business instead of Dialpad but right now we're a little bifurcated but we will take the offices that we move to Dialpad and convert them to Skype offices in 2019. Excellent and with regards to telephone systems this is something that if you have any questions I would really encourage asking on the LSN TAP email list. There is a lot of programs have went through very intense RFP processes, redesigning and redoing and there's a lot of knowledge there. That is definitely if you are looking at two or three different systems you can find people who use pretty much any of those systems on the email list and they'll tell you what went well and what didn't, they'll also give you feedback over vendors because putting these systems together is very expensive and there are vendors with very good reputations and there are vendors that people have lost a lot of money and abandoned systems on. So definitely get a hold of that. Yeah and one of the big changes is that both Dialpad and Skype and I would imagine most of the other Voigt vendors will let you go month to month.  So you have the capacity now that you didn't just a few years ago actually spin up an office or a few volunteers on Skype for business with both a headset and a physical phone if that's the route you want to go. We've spun up instances of 8x8 and Ring Central and Jive and Dialpad and a bunch of other vendors and actually I mean until you put an ear to the quality of the call and look at the feature set and develop what you're willing to spend or what you can spend from a budget perspective and what features you have to have and what features are just nice to have you can narrow down that vendor choice. In that same vein we decided to go with Amazon Connect as our call center software because neither Skype for business nor Dialpad really gave us the control we ultimately thought we wanted with our call routing as it related to online intake. So it's going to be difficult to find a one size fits all solution but with only a 30 day commitment on a per user basis you don't have to make that one size fits all decision anymore. You can mix and match different services based on your needs and your budget. We lost your audio sorry. Two questions, thank you so much. Is there a resource on how to implement Skype for an on-premises system that you would recommend? There's a laundry list of resources that we use for both VoIP and 365. I can in fact I just provided the list of a lot of the blogs and video resources that we use at Lasso about 365 in Azure and Skype specifically. There's a tremendous amount of documentation on on-premise Skype from Microsoft. Microsoft is the documentation is pretty good it may be outdated so you can use the documentation generally as a baseline and then go find some authoritative resources primarily in third party bloggers that will give you the correct recipe for a specific implementation case. So it's really a mix of a bunch of different type of resources. Microsoft is pretty good on the planning side and the big picture side and where their documentation seems to break down is I mean their documentation doesn't seem to keep up with the pace of their development so invariably you're trying to solve a specific problem that they haven't that they don't have good documentation on but invariably the blog postings of their development groups are very informative. They all have communities focused around specific products and services that you can become a member of and there's a lot of additional resources like Microsoft Academy, Developers Academy, what is it Channel 9 but I'll I'll I'll send SART the email of resources I sent to the Midwest IT Directors Group just recently on some of this stuff and we're discovering stuff every day as you Google a specific use case you'll find somebody else in the Netherlands that has this elegant solution for something you've been knocking your head against the wall for a month. But yeah there's there's a lot of resources and that's good and bad there's a lot of bad resources but but hopefully we've figured out the ones we trust the most. Excellent now we'll we'll put together a separate blog post for those resources and just make them publicly available. Yeah cool. So I'm jumping to our resource slide which will also be available in the blog post. It covers a lot of the different things that are out there and we will make this whole list available. Additionally I definitely want people to know that our YouTube channel has videos on pretty much every topic that is covered in the baselines and if there is not a video there please feel free to email me we will create a video or we'll put together a training and we'll find some experts out there in the field to help share whatever is going on there. It looks like we have covered all of the questions. Thank you so much William for co-hosting this with me. I like what you're doing in having kind of that director's call and sharing knowledge on a regular basis. We'll look at our schedule with webinars next year and see about putting together like some type of a national like quarterly thing probably on a different piece of software so that everybody could talk so that it's a little more interactive but I really like that because just having those discussions with other professionals is one of the best ways to share knowledge. Yeah absolutely it's a great asset. No this is the last webinar of the year there's going to be a survey going out within the next day or two because we just released our RFP for what webinars you want to see next year. As that survey comes out to everyone please fill that out mention the topics that you want we're going to be looking at different vendors also if there's anyone that you think we should reach out to with regards to doing a presentation there'll be an option there in that survey or if there's something that you would like to present on send your will-collect contact information through that survey and we would love to have other experts from the community present. Thank you all so much greatly appreciate it.