to rebuild and load the data in the case of the restore logical backups usually are small in size in the case of MySQL they benefit from the MySQL approach to the multi versions due to the fact that you do not require logs in other technologies logical backups require a lock at the database level but due to the fact that repeatable read is implemented as a snapshot at the database level in MySQL you can use repeatable read as a method to create an image of the database the problem with logical backups is that rebuilding or restoring the backup can take really a long time because you need to reinsert all the data and you need to recreate all the data structures index and so are really good for partial recovery it's very easy to recover just one table usually you have a readable human readable copy of the data which is very good for example if you need to make a partial recovery but not in the sense of recovering one table but in the sense of recovering a bunch of rows from one table in MySQL the impact is really low so just one query and the last thing is that you need a leaving instance to recover so you need to install MySQL and you need to start MySQL and create MySQL instance to recover the logical backup between the logical tools the first one is the good and old MySQL dump is just one thread just one file it's very easy to stream MySQL dump using netcat so you can do MySQL dump, pipe, netcat send to another server and then apply it the same time you can use it also to dump only the table definitions and there are options which are not really used very often but are interesting but again for example generate the limited data using dashed up you can dump the schema the schema and the data or if you dump the limited data only you just dump the data and recovery can be very lengthy MySQL pump oracle into to have parallel dump it has some interesting features for example MySQL dump dumps or used to dump the user table instead MySQL pump dumps the user creation and it has some interesting features it supports compression it supports advanced object filtering you can define patterns for filtering but the problem is that it is not valid for creating a replica because it does not return the position so in theory it's consistent but yes MyDumper is an open source tool they are two tools MyDumper and MyLoader it's actively maintained by the community and I say actively because it's a project that has been sleeping for some time but now it's quite active and some interesting features have been added recently it allows you to perform parallel backup it allows you to perform parallel restore and it allows you to make a consistent backup it you make filter the tables and the objects you are going to process using both compatible regular expressions it generates multiple files and it also allows you to stream the backup the way the backup is streamed is quite interesting because what MyDumper does is it performs the backup in parallel locally generates the files locally and then there's a thread that takes every file and sends it to the destination so there's an additional thread that takes care of the streaming yes but the stream is always single-thread because usually it's the network yes but in the case of MyDumper there's a specific thread for the stream MySQL shell it has a set of utilities to dump low tables, schemas and instances in the case of the export it supports multiple formats this means that the data is exported and you can define the limiters and things like that it also supports things like rate limits to avoid loading too much the server it also supports storing in S3 compatible storage it can make consistent backup and the utilities are you have low dump to load a dump and then you can dump an instance you can dump schemas, you can dump tables and you can export that is the one that generates multiple dialects TSB is tab-separated values physical backups perform file or block-based operations to do the backup and restore I divide the physical backups in two types the ones that have awareness of the database and the ones who don't have awareness of the database structure so the snapshots are the main hot backup physical hot backup that exists but as we said before they are not aware of the database they perform the snapshot the same way and for example they don't check if all the files belong to the same snapshot something you have to program separately and also if for whatever reason you are forced to have files into different storage types then probably creating a snapshot is something impossible or at least very difficult but how you make sure that you are freezing in the same moment of time moment of time the two disks you flash one and then flash the other they are not consistent anymore it's like copying the and between the physical hot backup tools we have mySQL Enterprise backup and we have the clone plugin what does awareness mean its database page has lsn this is the last sequence number it just tell us the number of bytes written into the the the I was going to say the binary lsn it's into the inodb lsn so it's quite simple it's a number that it always grows and it is used to identify if a page or a block is newer or older than another if we are aware we can make an inconsistent backup and turn it into consistent and for each lock entry that we have we can compare the database blocks and say this lock entry needs to be applied or not why do we need this we need this because in the database lots of changes take place at the same time so we have concurrency, lots of transactions with different lengths are executed in parallel what happens when there's a crash when there's a crash we have a bunch of transactions that were committed and some transactions that were not committed and we also have information stored in the lock files so with all this information we are able to perform the recovery operation and rollback the yellow transactions that did not complete so this is the situation when we boot after a crash and I was explaining this because when we perform a backup it's basically the same we finish the backup in one moment of time so instead of having the crash here we could say backup end and when we restore the backup what we do is we remove the partial transactions the extra backup I like to call it the swiss army knife because you may do almost everything you can perform a full backup you can perform an incremental you can perform a partial restore you can encrypt your backups you can make backups to a cloud you can compress the backups it's portable you can make a backup from one operating system and move to another operating system as long as they are little endian or big endian both the same so for example I've been using extra backup to restore backups on windows even if there's no version of Percona extra backup for windows because you need to do some tricks but you can do it you can do it because the I know the defaults are compatible you can do it as long as you have a Linux machine where you can see the volume the window of volume you can you can stream backups it's compatible with Oracle MySQL and Percona server for MySQL it used to be compatible with MariaDB but that's no longer true and it's free open source to make a backup so the following steps first we make the backup after that we prepare the backup preparing the backup is what I showed yesterday is applying all the changes and then rolling back all the transactions that were cut in the middle of the end of the backup and then we can restore the backup whatever we want next tool is MySQL Enterprise backup it's a licensed software it's multiplatform it supports Unix Linux and windows it also supports backup to tape using Oracle technology and it also supports and this is really very interesting binary and relay lock backup so when you perform a backup you can also copy the binary logs and the relay logs so this also can be done with my damper you usually you do it when you need to restore you could do it after the backup you can run the prepare yes but you can do a partial prepare because this way you can do incrementals in the prepare stage you do two things you do the apply logs and then you do the rollback of the pending transactions okay there's an option that is just apply and don't do the rollback okay backup so you can still apply them okay the way iNodeB logs are being treated is changing so for example now the new versions are already able to dump the iNodeB logs so the dam that extra backup did originally is no longer in it you can start the backup and use the dam provided by my SQL this is a new when you make a backup using extra backup usually what you do is you copy the data files the iNodeB logs this is no longer in it why? because you can tell my SQL to do it to copy the iNodeB logs somewhere this is a new feature so it allows you to so you just need to this is why I said before that this is changing and probably this means that in the future we will have sort of base replication because otherwise it makes sense to implement this well it makes sense for the enterprise backup but if you want to create an iNodeB real iNodeB cluster then you use these two for replicating stuff okay the last but at least is the clone plugin it's mostly used to reveal instances of replication but it also allows local cloning to directory don't try this abroad test and decline your recovery procedures disasters always happen in the worst moment and when there's a disaster you need time to decide you don't have time to do research or learn how to restore and also something very important if the crisis is huge you need to involve business because if the backup will take three days to restore probably you need to decide the order you restore and you need to make some compromises that are beyond the job of a DBA I like I enjoy the backup sound like the shredding a cat because a backup can suffer of quantum superposition it was completed successfully and you can restore it in a relevant time what does this mean maybe it takes too long to restore if you made the backup using bicycle dump or it's an incremental backup and you say I make a full backup every month and I do I do this for an hour restoring this will take a long time for sure or there's something missing in the backup or there's a back in the backup software and this happens unfortunately quite often with the new policy of MySQL of implementing future changes in minor versions we've seen that actually we were forced to add a feature that does not allow you to perform a backup on a version of MySQL higher than an extra backup it has to be you can say go ahead and make the backup but as a default it checks the version and it does because we discovered one feature that made backups complete successfully but they were not able to be restored due to one change that MySQL added or something as stupid as the infrastructure is not how I want so sometimes you make backups but you don't have a support contract and your on-premise crashes and then the provider says that it will need three weeks to deliver server like this questions I've been quite quick I was manager at a company which was quite successful but they were quite they had problems with treasury and the money and they needed to work every day and they could not stop for more than three days if they stop for more than three days they can't return to the business because they don't have enough cash so and it was a large company with maybe 400 400 million euros per year but it was like because they had very low margins and then for them being able to recover quickly was critical not because it's LBM who does the snapshot so LBM is able to control the snapshot because LBM you have LBM is quite similar to other companies you have the files but you have a lock so it's able to perform a snapshot using different devices what I'm saying is if you are using for example a snapshot on storage and over that storage you have LBM and this storage actually are two network storages so how you make a snapshot and make sure that this snapshot is good or two sands on the same LBM volume and sometimes this happens with things like VMware where you have data stores and also very often the providers try to move the snapshot to the lower level so you think you are making a snapshot on VMware but there's a driver that moves the snapshot to okay so as snapshots you have to test so that actually are consistent on the paper it's safe we've seen in RDS I think it was in RDS I don't remember it was RDS or Aurora an instance running on X3 okay the underlying file system was X3 the underlying file system and this brought a lot of performance issues so with those providers the problem is that you have no visibility you have what they say this means that usually snapshots are reliable but you don't know if the underlying storage is 100% reliable I'm talking about for example EC2 instances a different thing is RDS if they say the snapshot is consistent it should be consistent because they are selling this as a consistent snapshot but it failed probably if you use a replica it's easy otherwise why are you using a snapshot I mean yes but why are you using a snapshot instead of using Percon extra backup it depends on the parallelism and how fast are your disks I don't have a number for GCP but I've seen terabyte size backups running really really very fast an hour, sort of one hour I mean I'm not saying that the snapshots are not good what I'm saying is that the snapshots um yes if it's a snapshot provided by it's not a snapshot provided for database I'm saying if it's cloud SQL then it should work because they have methods to make sure that everything is flushed and when everything is flushed it's really flushed because and this can happen the file system driver for your storage tells you that it's flushing and it's not flushing it can happen that it ignores the sync or the file system driver thank you very much hello, hello, hello you hear it? is there delay? too loud, too quiet perfect, thank you not a lot of people yet yes, it seems so and it's Friday night 4 so I'll start in a minute or so I don't have a mouse clicker unless so maybe some questions that I have for you in the meantime so you can interrupt me at any time, right? if you have questions but maybe I have some questions for you so who's using MySQL in this room? okay, some people are not okay, and from those who are using MySQL who's using replication? MySQL replication okay, most of everybody and then from the people who are using replication who's using replica set cluster or cluster set no, that's Galera that's Galera, okay, but that's good to know so have you heard about it or are you using it? but are you using inodb cluster then? okay, so good we have a user are you using inodb cluster cluster set? so that's what the talk is about today so have you heard about group replication and extra db cluster from Prokona? good, good oh no thank you so need to change and I need to be quick okay, so good, so some people have heard about it, are you using it? so who, good so we're a small group so really if you have just feel free and talk, right? I am a person with a lot of kind of slides and I'm skipping some of them already because of audience which is fine but first before I want to start so I'm going to start talking about database architectures and I'm one of the product managers at MySQL and one of my products is MySQL replication, high availability middleware products so that's where I'm a product manager of at Oracle and the database architectures that I'm talking about is mainly high availability and disaster recovery solutions so everything that I'll talk about and it's not very clear I'm not trying to sell you anything, I'm just trying to show you some products that we have that are open source, right? so whatever I'll talk about is freely available everybody can use it, try it out, break it whatever, right? but before we get to that I want to give a little recap on business requirements, right? because it's important when I talk about different MySQL database architectures is that we kind of say like how do you choose which architecture what is good for whom or what business, right? and the two concepts I suppose most of us are familiar with this is a recovery time objective and recovery point objective so RTO recovery time objective is how long does it take to recover from a certain failure and recovery point objective is how much data are we or can we lose when a failure occurs, right? so that's important we have this lined up here so we have time, there's a failure so the RTO is how long can you afford to be down and then the RTO is how much of that data will you lose when a certain failure happens so this is important question to ask your business, your application what it's important to know this before you can go and design and choose a certain database architecture for anything really now the type of failure is there's a couple of different types of failure that we have and to kind of summarize we have three we have high availability and that is within a single kind of region or data center however you want to see it a single server failing so let's say we want high availability so we want no data loss RTO 0 when a single server fails or when a network partition happens then yeah that's high availability disaster recovery is when a region fails how long does it how much time do you have so what's the RTO for activating another data center or like is it seconds minutes, hours and then we have a third kind of failure is human error like little like drop database by accident or an application bug how much, how fast can we recover from that there's a couple more different types of failures but yeah human error maybe let's say that's where you have to recover from a backup do point in time recovery something like that now important to know is that when I I've done a lot of like consulting and like architecture design and when I ask the business how and this that means I'm talking too much about one slide when I ask okay what's your RTO RPO it's always like uptime needs to be 100% so no RTO 0 almost RTO 0 no data can be lost but it's important to look into the context of things because I think when we design something we shouldn't we I think it's fair to say we can achieve quite relatively easily an RPO of 0 in high availability within one region but if you say I want no data loss when a disaster happens when a whole data center goes out things are getting a little bit more complicated and I hope you're not using the internet for doing all your communication because the internet is not really a necessarily a stable environment right so it's a it's important different RTO RPO depending on the different type of failure and of course it's very important that the MySQL or we're talking about MySQL here or the database is just a part of it right the network also has to be stable or the network also has to be designed to sustain that I have had customers ask I want no data loss automatic failover but my network often goes out I have or I'm using the internet for doing wide area network and I want automatic failover between regions well you're gonna have a lot of issues with that so the network also has to kind of support these claims right so these are the concepts and I'll explain a little bit about the different types of architectures that we kind of that I kind of recommend or solutions that we have built in MySQL and then we'll tie them on RTO RPO and what they do what they not do right but before I go that and that's why I kind of ask these questions is what it's what is it with the how do people deploy and I'm sorry I'm gonna put in the power here because I think this will help yep it won't go out as fast now so it's what is important is that we talk about the past and what people are doing so one person said I'm using you know DB cluster but I feel a lot of people are still using MySQL replication or maybe Prokona XRDB cluster or Galera based solution which I think is good but if we talk about the past and I see many companies even bigger companies that just deploy MySQL is a lot of the work that a DBA has to do is all about setting up and automating the database architecture you have a primary and maybe you want two secondaries well in the asynchronous world or in let's say more than five years ago like 2016 or before and you use MySQL then you would have to set up primary then restore a backup of the secondary configure replication you have to add a user you have to kind of set the GT IDs correctly things like that so there's a lot of work to be done and what MySQL did was they gave you technical pieces like okay you have replication there's a way for you to kind of take a snapshot of the database there's pieces that we give but we never fully integrated it and that's what we have changed since 2016 when our first product in a DBA cluster came out we started integrating this all to really make it easy because there's a lot of work DBAs have to do and for example my partner is a DBA as well and a lot of her work that she does MySQL DBA I met her at a MySQL conference by the way interesting and a lot of the work that they do is automating this setup and making it work and make it their own but I think a lot of companies and there are special companies out of there that use MySQL a lot of companies kind of need to solve the same solution the same problem and if we talk about high availability and disaster recovery it's about RTO RPO basically like we have some common needs that companies have there might be different but they're going to be all in the kind of the same ballpark I'm not talking about read scale out things here in this slide deck we're talking about HA and DR specifically here so in 2016 MySQL released something called MySQL InnoDB cluster and that is a basically a name it's just I wouldn't say it's a marketing name but it's the name of the product and that is just a set of using a set of different technologies making something and by that I mean we have MySQL group replication that is being used and I'll get into that more MySQL shell so you have the MySQL client but since MySQL 5.7 we also have something called the MySQL shell and then we have MySQL router which is kind of a load balancer that can load balance traffic reads writes reads and so on so these all these products together together with the MySQL server as well like we fully integrate this and we make it very easy to set up an architecture like this a standardized architecture one important thing and there's a lot of features that we added to MySQL 8.0 which went GA 4 years ago so it's nothing new is InnoDB clone which is a way for example to when we add a secondary in the past you had to provision it you had to like restore a backup and so on now with InnoDB clone and you don't have you can use the feature of standalone yourself if you don't want to use the integrated solutions that I'll be talking about is that it would basically allow an empty server here and you just say clone from this there and you do MySQL authentication MySQL connections are opened it sends a physical snapshot of the data to the secondary restarts and it's up and running and you can configure replication and you're up and running so this is very important because InnoDB cluster and the other solutions I'll talk about they all are run from MySQL shell and MySQL shell runs somewhere remote and it says okay you're a cluster add a member add another member and it does everything for you for router you just say bootstrap I want a bootstrap connect to this cluster and that's all you have to do so really it only takes a minute or two to set up an environment like this of course it takes longer if you have two terabytes of data to copy I mean it's just for an empty setup it's just super easy and otherwise it's the time it takes to take a snapshot and copy everything over so what we have here with InnoDB cluster is an RPO of zero so no data loss in the case of one server failing FYI we have three nodes here you can go up to nine nodes with group replication no data loss if one of those fails if more than one fails at the same time then we do not guarantee no data loss right and then the RTOS seconds and by seconds it's configurable but the minimum is five seconds so you can have it fail over after five seconds of being a node unreachable or network partitioned and so on I can give a whole hour multiple hours of talking around this but I'll try to be brief because there's more to be coming right this is this is only the beginning so 2016 we released this we have a lot of users a lot of our customers are using InnoDB cluster but then we found some other need like maybe the user so the first thought was okay we have group replication I'll get more into detail about group replication and that's easy to set up and that's high availability built in but not everybody needs this but what if as a user you don't need automatic fail over maybe your network is not as stable but you still want the same ease of use well that's why we have InnoDB replica set and what is that it's basically asynchronous replication configuring so the shell configures everything does the provisioning configuring of the replication adding the users and the same router integration so basically the same commands you type but you just say change cluster into replica set and you have basically an asynchronous replication setup you can add as much read replicas or secondaries to this as you want there's no limit question or just here okay okay no question okay no problem so that's 2020 that's when we released this what we released was it last year end of last year beginning of this year is InnoDB cluster set and this is just a very frequent request that we have from our users we had InnoDB cluster which really works fine but we want disaster recovery give us disaster recovery so a lot of our users know the time before InnoDB cluster so what they do is kind of like okay let's just set up another InnoDB cluster and manually configure replication between them trust me that won't work why because this is an integrated solution there's a lot of automation happening by shell and it won't work as you expect it to be because like a metadata schema here the integration will fail but a lot of users did set it up despite all these drawbacks that they had to take care of so that's we took some time we took quite a lot of time to make it work we introduced InnoDB cluster set and basically it's InnoDB cluster multiple InnoDB clusters so you can say region in one in Rome here and in Brussels you have a cluster in each and then the asynchronous replication happens from one cluster to the other and also like instead of three minutes it takes five minutes to set this up again very simple fully integrated all automatic what we then provide with this is and of course in replica set maybe I forgot to mention this there is data loss because it's asynchronous not semi sync we're not doing semi sync here if you want semi sync you're in InnoDB cluster territory but asynchronous so data loss is okay and failover is manual this is replica set is like a basic version why don't we do automatic failover here is that the only way to do automatic failover for us is to have a monitoring node make decisions like orchestrator or my school MHA there's been several tools open source tools available that do this but we don't want to do that we do not want to make a monitoring tool that makes decisions does failover why because often false positives happen with this monitoring issues if there's a monitoring host has issues then it might make the wrong decision and so on so that's why we have InnoDB cluster it does have the automatic failover it's basically using a PAXOS based algorithm it's PAXOS but it's some variant to it so it does automatic leader election does the failover it guarantees that the data is replicated to the majority before it acknowledges so there's a lot of features built in that make this not need a monitoring host so this is a lot better solution so what we did with InnoDB cluster set is that we still have high availability with RPO0 and RTO of seconds within a region but then we have a manual failover with some possible data loss but this is asynchronous replication so we're talking about if the network bandwidth is not limited and the servers are not overloaded we're talking about milliseconds data loss usually it's not going to be a lot of data loss if any so this architecture very important it might look easy but we had to do a lot of features for this one is we have router and router is everywhere and router knows about the whole topology router knows about it router can connect to both topologies depending on the setting that you use and I'll show you more what the kind of things that we have built in into this now the good thing about this is that when you have Rome and Brussels it's asynchronous meaning that there's no right performance impact if you write a transaction here you don't have to have consensus or make sure that another node from the other region has to acknowledge it so you have the same performance with a regular NLDB cluster but you have a disaster recovery solution yes so why can't we do automatic failover between regions again for that we would need the third region and the one that kind of like witnesses and then makes a decision the kind of one that so there's a couple of ways to implement it so if you want automatic failover with this we can't because we don't have this monitoring host in another region that kind of decides who's the primary we also this is group replication stays local we do not in group replication have the concept of an arbitrator at this moment so something that you're using actually because the arbitrator is a possibility but the problem with the arbitrator is that it needs to receive and takes part in every transaction it receives all the data and that might you need a real data center with low latency as well because it will be impacted if you want automatic failover I'll show you later you just deploy a single NLDB cluster across multiple three regions then you'll have automatic failover yes because it's asynchronous it's not yeah we need this monitoring host to make that decision that's why it's not automatic oh follow up sure it's hard coded is there a reason there's there's just concern from engineering that if you do more than nine nodes that the communication overhead is too much so we haven't had users ask for more and I think if you like I don't I don't necessarily see the use case and maybe we can talk later of having more than nine nodes because usually you want to do this when you have kind of like re-scale out or something you can add asynchronous nodes to a cluster so and there's a lot of actually nice features that would help you and I can show you that later on that make that possible yes sure no problem okay so in NLDB cluster and I think that's my next slide we can do you can try to all nodes but there are some limitations foreign keys in MySQL are implemented in NLDB layer which happens under like in storage engine level so cascading foreign keys are not known to the binary log which is used to kind of we don't have the information the upper levels in MySQL don't have the information available about foreign keys so cascading foreign keys are a problem and DDLs are still a problem in multi primary you can't drop a column on this node you can actually and it will replicate but if you do a write and use that column it might break so there are limitations and we need to do quite a lot of refactoring in MySQL to actually solve it properly yeah yes but note that every member has the full dataset and needs to apply every transaction already so but I understand yeah in cluster set we even do not allow multi primary mode it is a single primary you cannot enable multi primary in a cluster set so we're gonna have only one primary and all the rest is gonna be read only even this one is primary it's gonna be read only so this is some of the features we implemented if you do a clean failover here we guarantee no split brain because we control that this is constantly read only so if we do a failover manually we make this one read only this one the primary and we do it in the right order so we can guarantee like no split brain will happen which is what some older products kind of did not do very well they would have the potential for split brain things like that okay but I'll talk more about InnoDB cluster now if that's okay so InnoDB cluster as I mentioned it has four different components so we have MySQL servers we have group replication we have router or routers and then we have MySQL shell and I'll elaborate on each of them a little bit and again I'm staying very high level here because I have 49 slides and I'm only at slide 14 and I skipped the first five but anyway it can go quickly soon okay so the goals of InnoDB cluster is we have MySQL and we have so many products in MySQL but this is about bringing all those features together we have InnoDB cluster and that drives the features that we need wherever it is right is it in the router and the shell group replication they drive okay this has to work and they all have to work well together we integrate it all together and we test this all together so we have a QAT that does testing on this environment right so it's really good to have right if we before that we didn't really have that we did test replication we test our technologies but we never tested the full solution because it wasn't a product right so that's important easy to use so it's very simple to use and I'll show you a little bit how the commands look like it's JavaScript or Python mode it's quite simple and then if we talk about group replication so what group replication provides you fault tolerance automatic fail over as Alan asked we can have active active anywhere but with limits and I mentioned those limits with foreign keys and DDLs it's automatic membership so adding and removing members is automatic provisioning happens automatic a physical snapshot or like an asynchronous recovery that's all part of this shell doesn't do anything there shell doesn't copy any data shell just initiates add the member and all the rest happens automatically network partitions failures all handled automatically within the bounds of course of what type of network partition it is we have conflict detection resolution so if you write to multiple at the same time and they conflict somehow we have basically first committer wins system so the first one to commit will win and the other one will roll back that's some part of the protocol so we prevent data loss in this case so RPO zero no data loss when a member fails so the difference between kind of the Galera based solutions and group application is in Galera if you have three members each member has to acknowledge every that the transaction was received before the the member that executed the transaction will send back to the application okay we're good to go the commit has succeeded in group replication the majority is enough so if this member is kind of slow the transaction will still continue because there's majority has received it so that's a little subtle difference in group application versus Galera this also means that if I have one member in another region you're not gonna have that overhead where we don't it's gonna the consensus will happen local and in kind of a Galera solution it has to kind of get an acknowledgement from the other side so if it's 200 milliseconds every commit really seconds to it in group application not sometimes it does it doesn't there's like a mode single primary mode by the way it's there's some I'm not gonna go into detail I can talk about it more later where you can enable that the downside of course is with Galera you know that the other side has received it but in group application you're not sure because one node there majority can be reached here in one region so if this region dies there's no guarantee the other side has received everything but yeah anyway I drift off sometimes sorry so group application so it's it's total order so rights are ordered it's basically one bin lock sequential that's what kind of what ends up with right yes question yeah yeah the majority will continue and if the majority does not have the primary it will get the primary and the right role will enabled and the minority will go into isolation like it will wait it will wait until the network partition is resolved yes three and two yeah yes correct okay and the other two are isolated as well already okay yeah so let's say so we have five and then the two here two of these go away then the majority will between those three because that's more than 50% these two will isolate themselves like we'll go into read only and not accept any right transaction if the one isolated it was the primary a new primary will be elected among the majority one of the majority members you can configure wait right and you can prefer this one if the others are in another location okay that's still the same thing but when they become the primary like they were chosen to be the primary they will you can have a setting that applies the queue before it activates so it will take a bit longer and you can also disable that setting so there's two modes either wait or not wait yeah transactions to apply yes the queue to backlog to apply everything yes correct that's the group group consistency group application consistency level before on failover there's like a setting there okay so we have a half an hour left so group application writes are ordered it's XCOM which is a kind of a Paxos implementation it's more based on Paxos Mencius which is another variant of Paxos we have configurable consistency guarantees similar to kind of what Galera has implemented differently so you can have eventual consistency meaning that if you read from a secondary you're not guaranteed to see the absolute latest but if you want you can do it in several ways you can kind of configure read or write consistency levels depending on your desire it is using MySQL replication framework so the same team that develops MySQL replication also develops group replication it's the same engineers well it's different teams but they all know about it and it's using binary logs gtid's everything that you use an asynchronous replication group replication loses as well it's been GA since 5.7 so 2016 end of 2016 I believe it became GA and yeah we supported on all our platforms so whatever platform we support group replication is supported as well including Windows not that that matters often but okay who's running running on Windows nobody okay yeah but we do have it we do have some users we actually have some big users using Windows but yeah it's important okay so no data loss rpo0 I've mentioned that splitbrain is prevented because of the way it works with the quorum calculation I'm not going to go into much detail automatic failover automatic partition handling you can do kind of read scale out with it so you can add more members and you can send read traffic to them all works there's automatic handling of replication lag using flow control there's like 20-30 settings of it to control that if a secondary is lagging behind what do we need to do the primary will start slowing down the throughput that's what we can configure how much it delays all of this you can configure that with those flow control settings the consistency levels I briefly mentioned you can do active active but you kind of need to be careful on you know the cascading foreign keys are a problem doing a lot of DDLs will pose a problem so that's important now good about this solution is optimistic locking so the right performance is not too bad I think we can we have environments that do like 20,000 transactions per second so that's actually right transactions per second in group application I think that's kind of the highest we've seen so far in low latency networks it doesn't really matter so we've seen that so that's good and the added latency because of the way it is implemented we only add one roundtrip it's not like a two-phase commit that we need to do we don't need to like prepare all nodes okay everybody prepared commit so that's two messages we have to send back and forth to the nodes this is not done like that it's basically you send the traffic and the nodes say so that's one roundtrip added and that is enough for group application to know okay the majority has received it I can continue we then have a certification that happens to guarantee that there's no conflict so that happens without communication between nodes also because every member has the full data set during a transaction no extra communication happens it's only at commit of the transaction that any network communication is happening so you're staying local within a transaction you'll get inodb performance just like you're used to with a single or asynchronous replication okay so briefly a slide on mysql router this is our integrated routing solution at this moment there's actually new features that we've released we released some features last quarter it's basically you configure your application to connect to the router and the router will figure out where to go and at this moment the router is a layer 4 load balancer so it does not do transparent read write splitting there is a port for reads and there's a port for writes so be aware of that and router will know reads I need to use the secondaries writes I need to go to the primary that's based on port it also has tls offload so you can do the tls ssl connections happen locally if you install router on the application servers and then the server we have connection reuse even now recently where the connections remain open and then we just reuse another connection authentication never happens by router router always proxies the authentication to the server so if you're familiar with proxiesql proxiesql does the authentication your user management has to happen in proxiesql with router no this is as lightweight as it can get just very small binary it will proxy the authentication to a server that does the authentication sends it back and that's it it will never authenticate but we have this kind of connection reuse so we do a local ssl because ssl is a lot of back and forth so that adds latency you can do that locally very fast and then we reuse another kind of open ssl connection that we already have and that way you can kind of reduce latency router we have users that have hundreds of routers per group application cluster or per inodb cluster why is that? well we have like features so mysql has the mysql port 3306 that's the mysql protocol but we also have the x protocol on port 33060 and in one of the features of the x protocol apart from docstore and kind of that's our no sql api and our regular sql api is also notices and what we have there is that when the topology changes it sends a notification to all the routers who have a connection open and they don't have to pull every half a second for the status they don't have to monitor the status of the group they'll be notified so that's why we support like hundreds of routers on a single group we don't need to do health checking that often because it would overload the database servers so there's like a lot of little features there I wish our documentation was a little bit better so I hope we can work on that because router has quite a lot of features but I think we need more how tos written so one of the I think I'll give a presentation soon at another conference I think cloud world about all these use cases all the things that router can do also router works for replica set cluster and then cluster set it doesn't matter it's the same router so that's also very easy and then we have shell so shell also was released in 5.7 so you have javascript python and sql so the default when you log in it's javascript and you can write your own javascript plugins you can actually write plugins that do all kinds of things so very very easy to use it does this doc store our document store it supports just like our connectors do as well so it does both protocols the mysql protocol and the x protocol and maybe I'll have an example here of how does it look like I don't have a demo here today because I'm yeah I still have 25 minutes but that's not enough to do a demo if I wanted to give all the other 20 something slides so here's how you connect you start shell and it's in javascript mode you connect to user admin on mysql one and you say I want a cluster object and that's dba. create cluster that's how you create a cluster a one node cluster by default they were error out because the configuration isn't applied yet so you're connected remotely on a server and it will say oh the setting this been binary lock setting is not good or this or that or server ID isn't set yet well dba it will configure instance on admin at mysql two or one so let's say we want to add member two it will configure it it will say okay all these settings are we need to change you want me to apply that yes you want me to restart the database if it's not dynamic yes so mysql eight you can change my set persist set persist only there's a restart as a command so you can restart the database so we can do with everything we can do remotely from a machine so if you have containers which is kind of common nowadays you can just connect to the mysql you don't have to get a shell somewhere on a container to execute commands everything happens remotely over that mysql connections and you can configure it set any variable restart and so on it will do that all for you let's say you want to add the instance so we have the cluster object that we created here and it's like a one node cluster and we do cluster that add instance mysql two it will say okay I have no data you want me to provision it I can do either binary logs or I need to clone it what do you want if it knows that there's not enough binary logs available it will say you need to clone it so you can do clone it does all the cloning for you and it adds the member to the cluster if you then want to add mysql three it's the same command add instance mysql three but if you then want to configure router this is only thing you need to do mysql router so on the wherever you want to install router as user mysql router that's the shell that's the kind of Linux or Unix user you want to run as bootstrap that's all you need to do well you you say the host name right mysql one for example and then you start it and that's all you need to do when it bootstraps it connects there's a metadata schema it knows based on the metadata schema we should have mysql one mysql two and router generates a config and then updates kind of a state file as it runs it monitors that metadata to see if there's any new members whatsoever what's the status of these members then you can get the cluster status so it's one of the commands there's many and then it gives you a JSON output with the status so we have the primary as mysql one it's in read write mysql one and then we have two read only members here mysql two and mysql three you can do like extended status you can remove instances there's all kind of operations that you can do but that's basically how you deploy a cluster and if you want to deploy a cluster set which I'll talk about later it's just basically cluster dot create cluster set another member and there you have it you have another cluster so it's almost the same command to run so replica set one slide only on that it's less frequently used but still we have users for it is same integration but it's asynchronous it still does the cloning and the user handling and the configuring of replication and it's manual failover but also shell provides all the functionalities to do a manual failover even in cluster you can say set primary you can change the primary here you can also say set primary oh I want this to be the primary because I need to do some maintenance here sure you can do that shell will automate all this for you if you want to make it automatic it's up to your responsibility you can just write a wrapper that does health checking and then sets the set primary or force primary so we have set primary which is a switch over like a clean changing of primary and then we have force primary which is a failover that we do which of course can create some split brain but it will tell you all that and then you have the opportunity to kind of do a stonet or whatever you want to integrate on how you want to use this okay any questions so far now we go to cluster set I'm just talking all the time and nobody else is so I see people nodding so that's good a few people are falling asleep but that's also fine you know no questions now this is one of kind of the latest development right is cluster set so we talked about inodb cluster we have cluster set now so the same exact ease of use as with the commands how to set it up and what so on you can have one cluster but you can also have multiple clusters so let's say we the primary cluster is in Rome we have one in Brussels while you can have one in Lisbon you can have as many as you want there's no limit actually we didn't define a limit like there's nine nodes per inodb cluster there's no limit on the amount of cluster set clusters but I think there's be practical limit or some kind right you also are not required to have every cluster to have three nodes you can add a third data center with a single node if you want you won't have high availability in Lisbon but you can if you want nobody's gonna prevent it it just works okay simple here let's say we have this environment this is okay a little bit of questions sure ask the question first I'll explain it I have a couple slides but I'll explain it here already because I kind of have to put some context on that so let's say we have two data centers so Rome and Brussels the primary is in Rome we have four routers two and each we have a target mode it's a bit hard to see I need to change the font here you have different target modes important to know we're nobody has asked yet because I get this question a lot can I do bi-directional between clusters and can I do it active-active between clusters the answer is no why it's asynchronous we cannot guarantee that there's not gonna be any conflicts so we are not gonna make a product that is gonna make you lose data or have a split brain so either we will have a fully supported solution that will guarantee that there's no split brain or just go and deploy your own group application and whatever you want to do but not in our solution right this is you're laughing but a lot of people are doing this right so many people and I have to talk a lot of people out of this yes and you too right at your work so okay router knows so the default way router works is target primary which means it will send writes which is red and then reads in green it will send it to the primary cluster but you can also change the target mode this one is in target mode Brussels this one is in target mode Rome which means that I made it a reporting application because it only does reads but if you send writes to this one writes will go to this one because the primary is in the primary cluster so Rome is the primary cluster it has the read write node here in target mode Rome if we send it right here it will go here and it will send it here now it's a little bit of a decision we have to make if this one is in target Brussels if a write happens here the write will not continue because there is no writer here this is the current situation you cannot say target Brussels but send the writes here we can add that to the feature of the code but it would delay us the release of this product so I was like we were like we need to release this as is and not delay it and one of the things important for this is this router is so lightweight just run a router with a read only and run another router with a read write mode and then you can have it as target primary and then send your write traffic to that one and that one will follow the primary and then this one in target Brussels will send the reads always local so it's because router is kind of lightweight now this router will know within seconds that a machine has shifted let's say this became the primary router will detect it within seconds what else is important here oh you can change target modes and the features that we're doing now to router as I mentioned is lightweight we do not want to do authentication there we do not want to kind of we want to keep it lightweight and the way you configure router for these modes and other settings that we're having is you do it through shell you just say this router needs to change its target mode right now what happens is shell updates the data here router will know oh my target mode has changed and it will change the way it behaves so what we avoid trying to do is we don't want you to have to change a config here we also have a rest API to check the health status of every router but knowing that routers can be deployed everywhere we don't want shell to have to connect and send an API request no we can all do this centralized because there's a database hey look we can use the database it's highly available the data is there so we use this we we heavily rely on this to just update its config and router just learns about the changes it's a very neat small simple design but it works so it makes it really really possible and flexible oh two questions yep yes this is fully using GT IDs oh good good question so the question is what about GT IDs so GT IDs global transaction IDs I think was introduced in Moscow 56 or is it 57 I don't know it used to be that if you configure replication you had to configure bit log file position like you change master it was then now it's change replication source is a new syntax it says okay this is the file this is the position the problem is if you want this machine to replicate from this one the binary log file replication is different than this one and with GT IDs globally global transaction IDs there's a way to say just replicate from this one you know what you've applied or it is like just just DT ID said I have you don't I don't have this give me whatever I don't have yet so this heavily relies on it group replication uses one GT ID per cluster so we have two GT IDs here to be more precise there are two GT IDs per cluster there's a management one that we had to implement but yeah it's an implementation detail we had to add some features to make this work we had to add another kind of management GT ID so you'll regularly see four GT IDs if you check the GT ID set yes it's the view change UUID group replication view change UUID the reason why is that if a primary switches here like something happens or you add a member a UUID is sent with the chain the new topology but in a split brain situation you create GT ID inconsistencies between them so that's why we put it on a different UUID so that we can reassemble it and we know it's a management we can ignore the management GT ID events yeah okay sure yeah we don't then we need to have users and passwords here in the router and we don't want to do that yeah SSL happens local we can have that but then the authentication is just the authentication string and yeah security you have to start replicating users while they could sync it the routers could fetch yeah and we also have this audit logging that we might want to do as users we have like in our enterprise version an audit log so we want to know who authenticated so we want to keep control here and the overhead that we've seen we don't have users complain about that about that specifically we're also adding kind of like connection reuse to it to the mix it's actually the connections are reused so the overhead becomes less you don't have to open a new connection so less overhead you can you can configure it you can configure the pool how you want it yes yeah yeah yeah whatever there's no limit yes you can connect the replica through a router yes that works yeah that works but we added features that we don't need to do that and I can show you that I can show you that in the next couple of slides three four questions yes go ahead yes yeah so it there's a timeout configuration for that one there's a limit on how many you want and it's also not in total how many it's actually per user and connection capabilities because remember the authentication we don't want different users to use the same kind of connections so there's there's a little bit of difficulty around that and then what was the last yeah okay yes correct and it's just I think the configuration is if the if you have that many idle start closing them so keep keep that many idle as you want yes yes but router by itself cannot open connections because it doesn't know how to authenticate so it needs a user originally to open an original one and there's a bit of a trick we need to keep state of the connection tracking session tracking like there's there's a lot of state we have to manage and and make sure that that they are compatible for the different users that things get quite complex yes question the router instance or new connections have to be opened so the application then authenticates again and router router is basically man in the middle that's what it does it kind of yeah it knows some things but it it doesn't know the clear text password but it can use whatever hashes were there to kind of forward it at least but it doesn't have access to the actual password bootstrap once question I'll show you that's what the scenarios I'll show you the after these slides I have some scenarios what if a region dies what to do I'll show you you bootstrap to any of them it doesn't matter because router figures it out what to do router will know oh bootstrap to this one it will see the metadata it will know that this one is the primary and it will register through here and it will replicate to everywhere and that's it you don't have to know the primary it's easier for automation when you automate this just give it one of them and it's enough sure no no yes I think we switch to what is it event based we don't do threads for each connection anymore so that's better I don't the scalability kind of depends on router but it does seem like from what our users say there's no they haven't raised any issues on that yet yeah anyway I have a couple more minutes and 19 slides so yes yes correct you can actually switch if you want okay let's show some use cases and scenarios right let's get down to it I have seven minutes okay good let's say we want to switch the primary cluster so this is manual so changing a cluster this is a clean operations that I'm gonna show you you have two regions you just want this one to become primary because the workload shifts there or you want to do some maintenance whatever right it is one command set primary cluster and you just say change across cluster to here so in purple is everything we do right replication will be reconfigured this one will become the primary and this one will become the secondary so router will also know oh have to connect here now and it will start connecting here target primary writes come here writes suddenly move there these rights will also move to that one so this one command and it will do auto automation all the work necessary for you to do this a second or that's very fast there what well yeah what we need to do is we need to put the primary all primary read only all connections that are ongoing need to commit or roll back so it kind of depends on your workload yes okay then router I think I mentioned most things already so I'm gonna skip some things then because I've mentioned it's a target mode I've mentioned these things so scenarios as the person mentioned what if something happens scenario number one the primary dies not the whole cluster just the primary and the prime the read write member the primary of the primary cluster the primary member of the primary cluster dies what happens is you know the cluster or group application will just elect a new primary this one will become the primary and automatically the async replication will also move to the new primary so that's fully automatic so we had to add some features to asynchronous replication to make this all happen you can actually do this all yourself if you just use regular asynchronous replication MySQL 8 you have the concept of not change by replication source to master host you can give a list of it even and even the replication knows about the group so it kind of knows about the whole cluster let's say this one dies the replica cluster primary which is read only but is one receiving the transactions if this one dies replication will be restarted and will continue here fully automatic so you don't have to do anything so these are kind of the common cases right if you do the clean switch it also does it automatically now that's what you mentioned the use case let's say it's partitioned not it didn't go on they didn't go on fire they didn't lose connectivity we have a two clusters right this is asynchronous so application is stuck nobody can access this application so what do you want to do first is fencing you want to fence it you can actually fence it we have features that we added to fence it you can say fence writes so we know because we don't want to writes can still happen to this one it's the old primary so we need to reduce split brain as much as we can so you can do fence writes and it will put the whole cluster and read only or shut it all down you can kind of use what you want assuming you can get to it yes correct if you can get to it maybe it went power went out that's a good thing right but if you cannot get to it there should be ways to get to that isolated one right yes correct but maybe you can access through some console some other way or there's another internet line to get in yes correct yeah you put the whole cluster to read only yes correct not cluster set correct you do it on a cluster yeah well this one is not this one is still a replica at this point it's still read only so you fence first if you can fence it if you don't have to fence it but it's up to you we do recommend it of course so suddenly everything now is read only what you now want to do is you want to so you do force so not set primary cluster you force primary cluster it and you say you make this one read right so router will notice oh I have a new primary here and it will start sending right traffic here now because we fenced if any if the application still so cron job might be running and might be doing some updates on the database that without fencing that will still happen here and we have a split brain situation so with the fencing we reduce that possibility now what happens if the network partition goes away and it recovers these routers will notice that there's a new primary so in the metadata it knows that this one should be the primary not this one anymore and they will automatically send all the traffic there yes then is this yes it it will know it as soon as it can connect to that cluster again yes so here it's read write and that one will be read write but if we switch it knows correct and I have one minute left lot of questions now to summarize so you can see the slides online to decide what went to use what right if it's HA we have inodb cluster no data loss RTO the time to recover seconds with replica set there is possibility for data loss RTO is minutes with multi-region disaster recovery we've showed you this right inodb cluster set which means RTO is zero RTO has seconds within the region write performance is great because it's asynchronous the RTO is higher on disaster recovery because you have to manually fail over now one thing that I haven't talked about is that you could also deploy a single cluster across three regions and you'll have full automatic fail over you'll have no data loss when a region fails you can add two nodes on each fine because then you have some kind of local high availability so this is also something possible but you need very good network for that and your write performance will be affected by the latency of this so this is another one this is well synchronous none of them are considered synchronous right it's or virtually synchronous whatever concept you want to call it but you can get synchronous if you want to yes anyway that's it there's some links everything is available open source use it so and that's kind of the direction and a lot of effort that we put in so if you have any more questions that I have to stop because the next talk is the slides I will put online slide share or on Oracle Tina will add it and I'll upload them now today yeah okay thank you sorry Dale sorry I have to say it again and then I'm used to having we drift off all the time right you have questions because you were falling asleep I was listening I know I know I know there's more people there's more people it keeps people more awake I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I


Hey, Aloha. Welcome back to the Think Tech Hawaii studios for another exciting episode of Security Matters Hawaii. Today we're going to be digging deep into an area a lot of people I don't think pay enough attention to. We're going to talk about identity and privacy and then we're going to get into consent. We've got one of the, I think one of the foremost gurus in this space with us today. Sal D'Agostino is with us from Open Consent. Sal, welcome. Thanks for taking some time out of your schedule to join us, man. Appreciate it. Aloha, Andrew. Always nice to be back on Security Matters. Thank you. This, this, so I saw an article today that was in I think Security Watch Magazine. There's some folks in California in a district there that want to outlaw facial recognition technologies. I think until everybody gets a handle on how they're being used. What do you think the overriding sort of problem that we're having with identity in our country is? Did we forget how powerful it was? Was it, is everybody too open with it? I mean, I don't know what to say other than people share anything, but then they don't want anything shared. It's kind of this dilemma I see. Yeah, so I do think the pendulum's swinging a little bit. I also think that the concept of privacy, you know, something that's evolving. I don't think it's very, very well developed as would be evidenced by the fact that both sort of like the, well, both the International Standards Organization and our National Institute of Standards are both right now working on putting together privacy framework. So it's not, it's not as if there really was one that people had before. That notwithstanding, you know, there's been a lot of people who've done a lot of work around privacy for a long time. And, and so coming down, and I think that what we're, what's happening right now is that you're beginning to see a change in what, what privacy is. For, for, if you look before at both security and privacy, because they're very closely related, what, what you would see and what most of the effort and regulation and other stuff was around was the concept of personally identifiable information and sensitive information and protecting that. And that was considered the goal of privacy and, you know, and security was an important component in accomplishing that in terms of being able to do things, you know, in a secure channel or, you know, encrypted or whatever the case may be related to security way. What's changed is the fact that a lot of it, well, in some ways due to GDPR, which is the general data protection, you know, regulation, which basically said that users have privacy rights. So what is changed from protecting people's personal information and then trying to make them a hole and a breach in which they never consented really to have their information used in any way, near the way it ended up being anyway. And something which was very much trying to minimize the risk through an end user license or, or a I agree button, like the wrong kind of consent form. Yeah, no one ever reads, right?  Right. So so on one hand, that's that's your introduction. And then your recourse is when you need when there's been a breach. And it's all been about simply that well, it's too much been about that. And again, again, what I would I think is changed is the fact that now there are places where there are people are establishing privacy rights. And, and those rights very much are then something that actually people have to respect and are enforced. And, and, and so if you then look at what the goal of security and privacy becomes rather than protecting personal information, it's it's ensuring people's privacy rights. And so if you then begin to look at how you do security and privacy from that perspective, it has a profound impact on our industry. And I so I think you then need to look and, and as someone who has done this for a very long time, I've been reading people's license plates and doing facial recognition for a pro 40 years. And so, and so security, security surveillance as opposed to commercial surveillance, and there's a very big difference. Security based surveillance system should basically be public. And be and because the purpose and justification for it is that you're there to provide public safety by general. And so that's so it's very holistic in that sense. And so when we were doing things like putting in place license plate readers as an example, we would be over saying we're doing this to so that you don't violate the toll booth, which otherwise you drive through and then the road can get funded and things don't work. And so there's there was a purpose for surveillance there. And I think what's what's happened is because, because the deployment of surveillance cameras since 9 11 has gone through the room. Along at the same time, Moore's law continues to march forward on multiple different vectors, one of which is the resolution of the cameras, then the processing power of the cameras themselves. And at the same time, the development, the proliferation of images that could be correlated to along with the open source software to do some of the things that we were used to be very complicated. Like when we were doing it 40 years ago and how to make a license plate reader, it was a it took a lot of stuff. It was a band full of equipment. It's hundreds of thousands of dollars of stuff. And today you can do it with your smart phone and open source software. Right. So so in some ways it's been commercialized and weaponized if someone were to get all hyperbolic about it. But it's generally available. And so so so then now that it's generally available, it was then used, I think, in some sense, with the lack of foresight about what people would think after the fact that you began to use machine intelligence to do identification and location. And there's also ways that you can do this intelligently. Because when you do machine vision, you try to use AI with video, there's stages, right? So you can you don't necessarily have to be processing and trying to get every bit of information out of the picture. There's a concept of when, you know, maybe something was like an alarm condition occurs, then maybe you begin to try to do stuff. So so there are privacy, there are privacy forward ways of doing things. And and a lot of that has to do with, again, goes back to the fact that if you do it from a user rights design perspective, you end up then focusing on consent and notice and and having that the part and parcel of any activity, including, you know, surveilling yourself. Sure. In the in the in the municipality thing that I started with, like in San Francisco, for example, so do you think that people could opt in or out of being surveyed in the public, you know, just as an example? No, I don't think I don't think you can. I mean, I don't think I don't think I don't think that I don't think people can get blacklisted from the database facial database store, right? So I mean, you could you could say, OK, well, I know I've gone to Facebook and I bought their graph and I've got all these images. And so or, you know, I've been just streaming the internet forever and I'm correlating people's pictures. It's not that hard doing Google image search to correlate a name to a face. Put your name into Google and look at images. Your face shows up, right? So because because that's so easy to do is is why you now need to be careful about whether or not it's OK for people to do that, right? So do I want that to be the case?  But then there's the other one of, you know, what what what is it being used for? And the fact that machine vision analogy. So you have the raw data. I mean, you have sort of like a general detection of stuff. You then can have sort of like classification of activity. And you can have identification sort of further downstream. So there's a yeah. And so what happens sometimes is people grow all the switches on all at once because because they're there on the machine, right? So like in some ways, you know, security people say, oh, if I can get metadata, let me go get it, right?  And so and I think that I think that's exactly what that law is going to, right? It's like, no, no, and the other thing is that if people want it, then they can say it. And so there's, so if you look at what's going on in China, a state and there's no, there is no opportunity for people to say, wait a minute, I don't want to have, you know, you know where it's at, right? I don't want the government to be tracking me when I go in and out of public or any or in some cases, private restrooms and count the number of pieces of toilet paper I use, which is actually true, right? There's no chance in China for people to say no way. And, you know, and so I think it's in some ways really important and encouraging that, but I'm not surprised at San Francisco, but that somebody says, wait a minute, we prefer not to have public identification and surveillance and tracking. And in under GDPR, I mean, this is a special category, right? I mean, this is a high risk processing of personal information called out in that rule. So in some ways all that San Francisco is doing is saying, wait a minute, this is high risk. Why do we do we need to do it? And that's what you should always, that's a question you should always ask whenever you get the high risk personal information processing. And it's not just the risk of data, but you know, the overhead of getting people to buy into it is would not be trivial. And so that should, that's also kind of a clear indicator of whether or not it's a good idea of if you're looking at it from the kind of open consent or protocol or design principles that we think can be very useful. Okay, I agree. So let's go, let's go pay some bills. And I want to get into a little more in consent and what's possible with consent when we come back. We'll be back in just about one minute. Hi, I'm Rusty Komori, host of Beyond the Lines on Think Tech, Hawaii. My show is based on my book also titled Beyond the Lines and it's about creating a superior culture of excellence, leadership and finding greatness. I interview guests who are successful in business, sports and life, which is sure to inspire you in finding your greatness. Join me every Monday as we go beyond the lines at 11 a.m. Aloha. Aloha and Mabuhay. My name is Amy Ortega Anderson, inviting you to join us every Tuesday here on Pinoy Power, Hawaii. With Think Tech, Hawaii, we come to your home at 12 noon every Tuesday. We invite you to listen, watch for our mission of empowerment. We aim to enrich, enlighten, educate, entertain and we hope to empower. Again, Mabuhay and Aloha. Hey, we're back with Security Matters Hawaii. Thanks for joining us. After the break, we've got Sal D'Agostino on the line and we are talking about privacy. We're at Intersects where identity and privacy and security intersect. Sal's been getting into consent and he's a co-founder of Open Consent. Sal, I wanted to get into we talked just a little bit about how people accept these in-user licensing agreements and they don't read them, they don't understand what's in them. Where can we go with consent? What's possible and what's the vision for what you're doing with Open Consent? Talk a little bit about that for us, for our audience. I mean, a lot of it, Andrew, has to do with changing the balance in terms of how that interaction with my information takes place online or yours or personal or an individual's information. If you think about your current privacy experience, it's basically accepting cookies, consenting to downloads, operating systems in the end user perspective, which is a little different than consenting to your tracking. But even in both of those cases, it's not the usability of it near zero. There's no opportunity for the individual to actually do anything. It's very one way. It's not like you sent something back. You first are sent to other places to go look at other stuff to tell you which are also internally difficult experiences as well. What we're looking to do is to change what that looks like. The idea that in particular, a lot of it comes through the concept of notices coming not you agreeing to things, but you getting notices back about what's happening with your information. And the notice is generated based on your consent. Again, with that comes the ability to do certain things. In cybersecurity, Andrew, we promote the concept of people having a cyber landing page where if you've got any questions, there's places where you can go to file with DHS and CVEs and there are places that you know you can go and do something about security.  I think you've got to pick up a phone and talk to somebody. I would say is your only But that isn't even available there. So if you look there, and this is what I'm saying, the analog to a cybersecurity thing would be for there to be a cyber point of contact and something and some workflow that you can follow. It's a dead end today online. And so what we're trying to do at Open Consent has changed that. So instead of making a dead end to begin to make it usable and operational, and I have had the fortunate ability to work in different kinds of automation and machine intelligence, not only surveilling and machine vision, but robotics and other sorts of automation over the years, the IDOLA platform that I have with my other ID machines is tech automation for the physical security industry. A lot of what Open Consent is about is privacy automation and how to make it usable. Because again, there are gaps in understanding and there are certain things that you can propagate as a result of having rights automatically because they're just there and there's a legal basis for and that's the thing. So in Canada, it's very far along in Washington state, it's very far along California, it's very far along. So in Europe, it already is. You're seeing substantial, you're beginning to see a variety and some very substantial enforcement of rights here. And so there's going to be a need to be able to provide users with those rights that I was talking about before. And if in fact, if what security and privacy both manage or they're trying to manage risk. And many of us are managing and doing a good job putting in countermeasures or modeling, changing risk equations and for the better is in some ways what we do for a living. And so not being able, and so there are many, many benefits to having the user involved and actually put in control of managing their risk. I mean, doesn't it make sense, right? So if you think about it, where is the risk that the enterprise suffers from the breach of people's personal information when the, you know, when the OPM was hacked and Sarah, my wife's stuff was taken and then they were laterally. Yeah, sure. Yeah, exactly. You know, that was a breach of personal information. You know, they had to then go through this process of doing all these sorts of things. I mean, they're the ability for you to have received a notice and have been able to take action should just, you know, need would necessarily be built in. And, you know, it's more efficient. Even when that hack happens and you're on notification, if you knew ahead of time that that's how it worked, it probably would have, you know, you're prepared for that to take place. And whatever next step she needed to, as opposed to this whole sort of snake eating the antelope gulp that, you know, suddenly everybody can convulses at the time of breach. And that's kind of maximizes the damage of the breach too. So if in fact you had sort of automation in place with the users already ahead of time kind of knew what to do and how this mitigates. And yeah, it's one of the things that could happen. I'm not saying you design for breaches, but I'm just using breach as an example of how if you bring the user in, there are light layers of good things and benefit. And the same thing when it comes to deploying security systems. You want people to know that there's a security system in place, right? It's being public about security. And so I think open surveillance and democratic surveillance, as opposed to state sponsored, where we were alluding to earlier that you didn't have any, you don't have any say, is the only way to go. And, you know, and so I think the article in San Francisco is an example, a very healthy example of how you work that out. And I think that then we can become much better at maybe it's no longer surveillance and maybe maybe it's actually structured sharing or I don't want to be too optimistic given the fact that most of the world economy is based on this stuff. But who knows what might occur. Yeah, I'm just picking a number out of the air. But it does provide a certain amount of lubrication to some international machinery. For sure. Do you think that that surface is going to expand with your identity being tied to like biometric signatures today that can be shared now? You know, what I said in the PLAA demo a few weeks ago, and they're making your biometric templates portable across different systems and things like that. So is the scope of your privacy surface expanding and do we have a way to deal with that? Well, it happened already with facial images. And now the commercialization of print, unfortunately, and face, right? So, you know, I've been doing some work recently to look at how face is used in modern devices and how you can restrict it to the trusted platform and if you know how to wipe it. But, you know, it's not trivial to keep secure your biometrics. And I would not be a fan of my biometrics on server any time, unless I absolutely have to. I mean, that's just no thank you. That's just, you know, really, you know, explain to me explain to me why you don't what why there's not another option. Okay. Fair enough.  Yeah, well, that's, that's the thing that concerns me is that there's I mean, there's no in no way is anybody building in the workflow to inform users about what might or might have been a compromise their biometric information. And this is just raising a flag for all that, right? I mean, so what I think it's good because what it's not like you have to do it that way, you can like the match on card for in the smart card world is an example of how your template gets captured at the device sent to the the secure element in this case matched on the secure element and then an answer comes out. No one ever stored your my template, right?  You don't need to keep it. Yes. After it's after it's been sent, you got an answer. So you can decide, right? Exactly.  And unfortunately, no one is considered by the excuse me, going over the top there. Unfortunately, not as often as should be are people considerate of the implications of schemes that don't adhere to some of those design principles. Yeah. So yeah, I mean, I think I think that's a that seems to me to be an unfortunate and more stuff in the news over the next several years, certainly about bad biometric thing. Leakage. Hey, so we got a couple minutes left. Why don't you give us give us the sort of open consent pitch? What can how can people get engaged with some of this information? What should they be trying to do to protect themselves? So again, we're going to provide people with an ability to do that sort of home page for privacy. We're going to let you know, we're going to we're going to open that up for free. Open get sense going to do that in the relatively near future, operating out of the UK and in Canada. And I mean, this is some extent anywhere, but I mean, in those cases, there's some stronger incentives, I guess, though, in fact, I think the same incentives exist anywhere. There's stronger laws, I guess, would be the better phrasing. And then we're going to do that. And then and then anybody can do that, and we're going to make it easy for people to do that. And then we're going to offer a service keeping track of whether people keep that up to date with a few simple things. And so there's then going to be a paid service, which is having us sort of basically give you a lot of some some some current information about the state of privacy for people. And and and so you know, again, so rather than going to the privacy page and reading a privacy policy, there actually be some you'll be an iconish thing there, which you can do something to and it will tell you a lot, or as it will tell you as much as that company is smart enough to tell you about what's going on with your your your personal information and how how they're treating, as I said, your privacy rights and affording you the yeah, from there, you know, in order to, you know, and then from there, from the paid business perspective, we can work with companies to then make operational and automate, you know, to the full extent, some of some of those rights, the free version has a few as a few simple things made operational, there's a, you know, the paid the paid for version, the live version, it's, you know, it's not terribly much more. But you know, I mean, it's just it's just keeping a check on it. Awesome.  I mean, and it's easy to use. I mean, that's that's the goal is to, you know, is to is for it to be relative to me or individuals. And and for you know, and for us to be able to do something with it. And and and it'll be interesting to see how companies respond to that. And yeah, and in particular, I am looking forward to working with the security industry, because, you know, we should have a very good understanding of this. And, and in fact, you know, rather than, as I said, the commercial capitalism case, we have a public safety and other, you know, other basis for and justified purpose in which in all of these cases, I mean, purpose is a very important thing. Right. And so if you're and so having knowing how to provide just because we have a purpose doesn't mean there is a requirement for notice and for being public. And I think that, you know, and I think the the it will be to the industry's benefit to to adopt that kind of forward approach.   And, you know, we continue to be collaborating with them on this. And, you know, then there's some good references there, actually. And there's more and more available at open consent. And yeah, anytime people like to go take a look, you know, we should be out, we should be out, out from one of the covers in the very near future. Awesome. So folks, if you're watching out there, your privacy is something you need to take a good look at, check out open consent, check out the information that's maybe on SIA's website and find out what's coming at you, because you shouldn't just be giving it away. Okay. So thanks for joining us so much. Yeah, security industry.org, Andrew, I think. Yeah, see a dot, see a dot org. Yeah. All right. Thanks so much. And we will talk again soon. Hello. Hello. Yeah. Yeah. Always, Andrew. Mahalo.