Hi guys. I'm John Schwartz. That's like the last one. This is Ann Tarday. She works at Mirantis. Kevin Benton also works at Mirantis. I work at Red Hat. It's a small company. You don't know it. Today we're going to talk about the race conditions of Neutrons, L3, HA solution, specifically the scheduler, under scale performance. So that's like a lot of subject to get through in 45 minutes. Please bear with me. So the talking points are quite unusual. Unusual. We're going to cover some HR routers 101. What are HR routers? What kind of resources do we need in order to represent an HR router both physically namespaces devices and more specifically database structures on the server side. Then we're going to continue on to some race conditions that we have encountered in Neutron. Continue on to scale testing that I did on both Liberty and Mitaka and finish up with some future work. So what does it mean to even schedule a L3 router? So basically scheduling means binding resources. In our case it's a router to one of the network nodes that we have. So in this example we have three network nodes or more and we have created a router and now we've gotten an API request to add a router interface, connect the router to some internal tenant subnet. So this requires an actual binding, an actual representation somewhere on the network nodes to let everyone know, hey, this is the route in which the package should flow. So binding a router is basically scheduling it and letting everyone know this is where it's at. So once we have the request to add a router interface which triggers a scheduling the scheduler has some kind of an algorithm either the least used network node or the least used L3 agent should pick this up or a random choice among all the valid network nodes. So in this case the scheduler decided to assign to bind the router to the second network node. So that's basically the scheduler and that's what it's supposed to do. On the legacy router that is the non-HA cases of non-HA legacy or DVR non-HA. It's a simple case because we have one router we need to choose one agent among all the possible possibilities, the possible network nodes and we just choose it and create the bindings. Once it is created the server lets the network node, the L3 agent know, hey, this is a new router please handle it. On the L3HA case it's a bit more complicated because we need to choose a bunch of network nodes that will handle it. So in this case we have the node 3, node 4 and node 5 are all network nodes and the router is scheduled to all these nodes and it is active only on the node 4 network node. What does it mean that it's active? It's active as in all the data flows through it and the standby nodes or the passive nodes are just waiting until node 4 will fail or will crash or something will happen to it and then they can fail over and take over the responsibility. This algorithm of who should be the active one, who should be the passive one is handled solely by KIPA LiveD and KIPA LiveD uses a VRRP protocol. It's a non-OpenStack project you can look it up, it's probably open source and the KIPA LiveD they all sit inside HA networks that are unique per tenant which basically means that if I have tenant A, tenant A has two routers, tenant A will have separate HA network than tenant B which is the red one. So this is apparent here, we have the blue HA network and the red HA network and all the connectivity that goes between KIPA LiveD that are in different HA networks goes through that HA network. So basically we have a KIPA LiveD for each agent, for each router. So if we have router 1 here and it's also in layer 3 agent number 2 it will both have KIPA LiveD. The KIPA LiveD uses different VR IDs per router so they won't collide. So if we have KIPA LiveD that is serving router 1 it won't think that something happened to router 1 or the other network node in case router 2 fails for some reason. So the separation is both of HA networks for different tenants and for different VR IDs. So that's basically all you need to know here. On the database layer this requires a lot more resources. So the first resource is the router L3 agent binding. It is a one-to-one matching between a router and an agent so this basically means this is the router, this is the agent, it is assigned to it. On the non-HA case we have a single binding as such. On the HA we have one for each agent the router is scheduled to. On top of that we said we have a KIPA LiveD that is used by the HA routers and they see it in HA networks and the KIPA LiveD uses different VR IDs. So all of these require database resources so we have the L3 HA router VR ID allocation which basically means this is the VR ID for this router. We also have the L3 HA router network which comes in addition to the normal network we all know and love in the Neutron database and this one holds the HA ports right because each KIPA LiveD sits on an agent and it needs to be connected to some HA network so we use I forgot what I was about to say so we use HA ports to connect an actual port that connects the KIPA LiveD to the HA network. Next up we have the L3 HA router agent port binding that's a really long name which basically represents a triplet of a router an agent and an HA port. This is the router it sits on this agent and this is the port that connects the KIPA LiveD on that agent to the HA network. When creating an HA router this is roughly the pseudo code. We create the router database object as usual and then next up is creating the HA network if one doesn't exist if one exists we don't need to create another one sorry after we have created the HA network we create some unbound HA interfaces we create unbound L3 HA router agent port binding one pair expected agent so we create the bindings we create the ports we just don't assign the ports and the bindings to an actual agent that comes later on don't ask me why it's that way we couldn't figure out that. Next up we create the VRIDs one per each router and then the scheduling comes along we determine which which agents will host the router there's a real long algorithm that does it it's like five lines of code we assign the HA network interfaces to agents so now we know which agents will get the bindings so we can take the HA port and assign it to this and that agent and finally we can create the router L3 agent point router L3 agent binding which actually represents a binding so an example race condition is this so the starting point is that we have some kind of an old router it is an HA router and user tool likes to delete it because it doesn't want it anymore and the other one wants to create a new router also an HA router so this is the scheduling that is done by the CPU scheduler of the computer of the machine or multiple machines whatever this is the timing that happens so first of all the user to flow deletes the router wanted to delete the router go ahead we don't need it on the database anymore and next up we say okay so this is basically the last HA router we don't need the HA network anymore the tenant doesn't have any more HA routers so let's delete the HA network so we have made a decision let's delete the HA network this isn't committed until later as it can be seen by this dotted line and then the CPU scheduler says okay let's give user one some runtime so we create the new router the router we just wanted to create and then we the next step is checking if an HA network exists and if it doesn't exist we can create it but if it already exists we don't need to create it so in this case the deletion of the HA network wasn't committed yet so it's still there so we don't need to create it the workflow then continues into committing the actual deletion of the HA network and then once we try to create the HA port for the HA network for the specific HA network that we detected here we are yelling because we don't have an HA network so the HA network has deleted from under us so this is an example race condition there are a few others I'll let Kevin go through them or fuel them so the the obvious question that comes up here when you have these big complex operations is why can't we just put this all inside of a database transaction so we start a transaction create an HA network create all the L3 HA ports depending on the number of L3 agents you have set up for HA create the tenant to HA network binding then call commit on the database and if there's any issues any duplicate entries that kind of stuff you just have one exception that you need to handle and you can start the whole process over so that would be problem solved the issue comes when you look at the internals of how neutron resources work we have networks and ports and routers and these are all distinct resources that have separate transactional semantics in their creation and they can correspond to external resources on other systems so if we put everything inside of one giant database transaction and we go through create a port create a network create a router and then encounter an exception the database will roll back but if we've been communicating with external systems we'll leave a bunch of orphan resources on these external systems this example right here of just creating a network and a port shows that there can be up to just even three different external systems with just even one ml2 back end configured creating a network needs to talk to the neutron database then go off to the ml2 back end then come in to create port we allocate an ip from an external ip allocation management system go back to the database here and then go off to the ml2 back end when it comes time to bind the port and then go back to the neutron database so these are all separate transactions that we can't roll up into one giant database transaction otherwise we leave a bunch of stuff left over in ml2 back ends or external ip allocation systems so the next question that comes up is well can we use locks this would be an easy solution just put a giant lock around whenever we have a router operation to make sure we don't have a delete happening at the same time as a create when it comes to ha networks for a tenant so there's threads and file locks db locks and then distributed lock manager two is in this case that we looked at and i'll talk about each one so thread locks and file locks are just based on exclusive access that a process has inside of a server so it's either some internal process exclusive resource like thread mutex or a file open for exclusive access on the file system but this only can prevent concurrent calls within scope of a single neutron server and neutron supports running a whole bunch of neutron servers all in active modes you have a load balancer sending requests across a bunch of servers thread locks and file locks won't do anything to help you because it's only protecting an individual server so then you can do kind of a fake lock with the database you can use transaction level locks which would be okay for a single transaction but in this case we're doing multiple transactions so select for update doesn't work because that's scoped to a specific transaction once you commit that lock is released and on top of that if we try to use like a separate side transaction to hold a lock for the entire time of other concurrent transactions this falls apart when you're using multi writer galera clusters so if you're doing ha availability for your minus ql back end then this doesn't block concurrent lock access it throws an exception at the end at which point it's too late and it didn't actually protect you from concurrent access the other option is just a basic reservation table where you insert a record into the database that says hey i'm doing an operation for this tenant and then it has a unique constraint and then nobody else is allowed to insert a record while they see it there but then the problem you have there is you have to deal with stale records if a if a server acquired a lock by inserting a record and then died all the other servers will just sit there waiting for that other server to get rid of it so you have to come up with some kind of uh kind of hack together timeout mechanism to decide well maybe after 30 seconds maybe it died and i'll just try to take the lock anyway but it's not really good locking mechanism so that brings us to uh using a distributed lock manager which is designed to solve this problem and there's twos which is part of the open stack big tent that's a library provides distributed coordination permanent so it has leader elections grouping and most importantly for our use case locking and then it has this driver architecture to use different back ends to where it stores the locks or the these coordination primitives that can work with zookeeper readus memcache there's several other back ends that you can find on their on the site for it and this would solve the problem that we're running into here but it has a couple of downsides the first one is there's a constant overhead now of lock acquisition across multiple servers so whenever we want to do a router operation on the neutron server we now have to talk to twos which uses a driver to contact a another server or another service and if it's in an ha availability mode then they have to go through some kind of election algorithm to acquire lock across the twos back end and this isn't a huge deal we could maybe uh deal with that because this isn't something that happens very frequently tenant creating and deleting routers but the the big downside we ran into is this would be a new operator dependency requirement for the twos back end right now or when we were originally looking at this there weren't any open stack core projects that had shipped twos as a requirement yet so we didn't want to be the first one to force the adoption of twos on all the operators there's a emailing list thread here where you can see the discussion about deciding whether or not to use the twos for the distributed lock manager so the solutions we come up with for the race conditions or you know use the database as much as possible model model constraints model things as unique constraints whenever possible so l3 agent plus router id there should only be one binding of a router to a single agent it doesn't make sense for an agent to host the same router multiple times that was one one example utilize transactional guarantees offered by the database where we can if it's not a multi transaction thing we can look up to see if there's a binding there if there's not a binding there then we can safely try to create one commit and then catch any exceptions that are based on stale data or db duplicate entry if there is a race condition from another rider and then wrap this stuff up in retry loops so i hit some kind of transient error start over try again they'll either already be something there and we have a code path to return early or the other thing encountered another error and we can safely create again so the approach just broadly looks like this we say does the resource and the binding for it already exists for example the ha network does the ha network uh is there a binding for ha network does it exist use it so that's easy that's the first return if it doesn't exist then we go on and we create the resource and try to create the binding which has a unique constraint for the tenant if we get a duplicate error it means somebody else already beat us to it there's already a now a tenant network for um uh already an ha network for that tenant so now we delete the resource and then trigger retry at which point we just go right back up to the uh this start thing and we'll hit this early exit of use it um same thing with the with the binding if we think the resource exists we'll try to create the unique constraint if we get a db reference there it means somebody deleted the network out from under us so we just loop back and start over at the create resource phase there so this essentially solved the problem it's kind of a complex state machine that it goes through but it'll retry up to 10 times and this effectively solved the issue of racing with uh the tenant ha network management one of the other issues we had is the l3 agent would periodically call the server to sync all the routers that are running on it and it would catch an ha router right after the database object for it had been created but it would be missing important things for the ha stuff like the ha network or the l3 agent ha interfaces because it was still being built on the server side uh so we decided to use the status field to flag partially formed routers so the agent could say hey this one's still being built just skip over it and then uh get it on the next sync uh the existing values we had in the status didn't map really well to this intermediary state of the servers still in the process of building this resource because it's kind of a composite resource of different individual resources so we added a new status field called allocating so now when you first first create a ha router you'll see an allocating state and then the l3 agent won't receive any routers that are stuck in this allocating state um then once the server is done setting up the ha network all the l3 ha ports that kind of stuff it'll flip it to the um active state and then the agent can get it um there's there's a downside to this approach is if a server dies or encounters an error while the router is in the allocating state we can end up with a router that's stuck in the allocating state and it won't get scheduled to an agent and there's uh a fix for that that just recently emerged that john will talk about later so what was fixed in mataka we fixed most of these server side races with api requests so if you're running rally tests that's hammering the server with lots of create delete uh ha or yeah create delete ha routers most of those issues have been solved in mataka and a lot of them were back well a lot of them were back ported to liberty too right so um and then races triggered by concurrent agent operations the agent syncing to the server at the same time um and crashing when it would get a ha router that was halfway being formed so those have all been fixed in mataka as well so now i'll let ann talk about testing for this okay so um when we found out these errors uh that john and kevin described uh we decided that we need to find out how they look like in production like environment so we ran some tests uh on the uh scale labs for liberty and metaka these results are published in performance docs the links will be at the end of the presentation and here i will share the most important results from there so our our testing was performed on the lab uh that consists of three controllers 45 compute nodes uh server's hardware of each node is presented on the slide and uh what was actually tested uh here we speak about races but um other aspects all 3ha was also checked um so at first we check races the best tool to ran out as much races as possible is rally rally is um which marking tool for open stack that allows us to create huge amount of resources concurrently um another uh thing that uh is a 3ha is actually about low for lower time so we want to check and to verify how um uh reboot for example impacts on connectivity for vm or a pair of vms so manual tests can help to measure these and to see how um or start all three agent or stop impacts on connectivity for large number of vms or used shaker tests shaker is a distributor data plane testing tool for open stack i want to highlight here that um stop or start all three agent won't cause connection interruption until crouching in spaces aren't cleaned up as well so at first let me share liberty testing results that we get on environment uh which was deployed uh with uh mirages open stack 8.0 so um at first let's talk about rally tests uh rally gives us um an ability to test uh open stack an ability to open stack to perform simple operations like create delete create update create list uh for us the most important test was create and delete routers uh and here on this slide there is an example of um create and delete routers yaml um so this scenario creates a network giving number of subnets and routers and delete all the routers uh this scenario has uh times and concurrency parameters times means how many times this scenario will be executed and concurrency allows us to execute current uh scenario in with given number of concurrent threats so uh to get um as much race conditions uh with deletion and creation of h.a. routers h.a. networks and h.a. interfaces we uh run this test um with um different times and concurrency parameters um l3 h.a. uh has a restriction of uh 255 h.a. ports pay h.a. network pertinent and if number of bits exceeded this limit uh new h.a. network is not created so uh we have some um tests that fail due to misconfiguration uh this is shown in yellow bars because uh the number of tenant wasn't enough for the number of routers that were going to be created so um with um increased number of tenants this test passed without any errors uh here are some more numbers of uh issues that we faced in so subnet and use error is about the race uh with the h.a. network management uh and session all back is an error uh because of the race between uh three agent fill sync and uh creation of h.a. router uh so um now let's take a look on the data plane side all three h.a. so the idea of uh main node tests is very simple uh at first we boot a vm or a pair of vms then we perform some destructive act uh then we start connectivity check and then we perform some destructive action uh and check uh the packet loss then we perform this test again with increased number of routers uh the number of routers are increased from test to test to uh increase uh the lowered for l3 agents so the most important for manual test is spin to external network from the m during uh reset of uh controller with active l3 agent uh so on the left there is network scheme for this test and uh on the right uh there is a graph um of dependency uh number of routers or from uh number of packet loss so uh there wasn't stability of rescheduling time for 175 routers uh the problem with this issue that it wasn't reproduced on their virtual environments at all so now let's take a look how things are going with large number of the m's and first let me describe how shaker works shaker um deployed uh instances on different compute nodes and execute connection connectivity check between them uh shaker contains server and agent modules uh server is responsible for deploying of instances uh execution of tests processing results and repro generation agent uh lightweight and they run inside of the m's and put tasks from server and replies with results uh the um typology deployed uh using hit blade once um um we can see that there is um agent in slave mode and in master mode in slave mode uh this means that these agents are used as a backends for corresponding commands for example they run a pair in the server mode so once all the agent replies to server the test execution is considered to be finished and um if some of the agent didn't make it in the dedicated time such results marked as lost uh I will share all the basic scenarios of shaker and full or full different scenarios can be found in performance docs so uh a three east west test um tested the bandwidth between the m's that located in different networks but plugged into the same router uh the test started from one pair of the m's on different compute nodes and increased load until all available compute nodes are used um during execution of shaker tests uh around scripts that um stop uh and active l3 agent and clean up coroutine spaces then when another agent become active uh the first uh agent was started so on the graph on the left we can see upload and unload uh bandwidth of with and without restart all three agent the same test was performed for ordinary rotary scheduling uh l3 hc was turned off and on the table on the right there is a table um how many preference were ended uh with errors or lost and we can see that for ordinary rotary channeling a lot of results were lost or ended with errors uh three north south tests check the bandwidth between the m's that um allocated in different networks uh the uh instances with um master agent allocated in one network and uh instances with slave agent are reached there via floating ip's and the same is started from one pair of them until all um available compute nodes are used and the set and the testing was performed in the same way and uh we actually get the similar results so this test show us the benefits of for l3 hc in comparison with ordinary rotary scheduling so and now let's take a look for metaka testing results uh which we get on environment based on um management stack uh eight uh nine point oh um so the rally tests um uh were around in the same way and actually have the same misconfiguration at first uh and uh here are some more numbers uh we can see that uh errors uh some net and you still persist but we uh fix other errors but we've got new error the server didn't respond in time uh for creating delete and creating the date and we suspect that actually this happens because of allocating or change being a texano network from the m during reset of controller uh help us to catch and investigate bug to masters of the reboot of controller uh and on the um left there there is the same network scheme and the graph uh with dependency so actually this uh problem was happened due with this problem was fixed during newton and on the right we can see that during reboot of the note a lot of packets were lost so um here are uh the shaker test which also was performed uh with and without to restart for 3 ad chain uh and uh we got uh the similar results for l3 east west and l3 north south uh so summarizing metaka results we see that although some things were fixed we um have new issues and now john will describe us how we managed to fix them thank you um so during the the testing and did we found a specific bug uh this is the bug number you can see uh so basically after the uh rebooting of our controller there were two uh master nodes we fixed that we fixed other uh bugs as well that's all uh can be divided into the three main types of races that kevin talked about uh that involved um concurrent api requests create and delete our outer concurrently or sending uh incomplete or partial data uh into the l3 agent or the l3 agent um issuing some kind of a request that will eventually cause problems in the schedule or like uh we saw before a timeline of the bugs and the uh patches that fixed most of them can be found in the etherpad um and very close to the end of uh newton at the uh mid-cycle in ireland um we sat down and we figured out that we can add a binding index to the router l3 agent binding so basically uh now every time we create a router l3 agent binding the object that matches a router to a single agent we also add a binding index which is a unique field uh for each router which means that basically when i create um a binding for uh some router to some agent i can i need to use some kind of a binding index which means that if two um two different workers or two different schedulers are trying to create the same entry using the same binding index uh this will fail because uh the binding index is unique uh in the non-HA we only allow binding index uh with the value of one and in the HA case uh we use different uh binding index between one and the maximum number of uh l3 agents that we will uh assign to uh that we will schedule to and this basically prevents over scheduling an HA router um that um that might happen uh in concurrent scheduling because now in case uh two schedulers try to bind uh an HA router to uh two different agents that will both use binding index equals one and that will fail uh and this allows the removal of the allocating uh status so basically um we are going to uh looking forward what we are about to do is we are going to refactor the l3 scheduler to allow us to first create the router l3 agent binding change the ordering such that instead of creating all the HA resources and then creating the router l3 agent binding we can first create the router l3 agent binding which is now considered safe uh and then after we know that the assignment was um was successful we can continue on to create all the other HA routers uh resources and this will allow us to remove the allocating status because we don't we no longer need it because now everything is apparently safe uh and even forward even further uh in the future we are planning to use uh push notification which will simplify the communication between the server uh and the agent even more uh so one thing we want to emphasize is that the current code is safe to use uh these are looking forward these are improvements to the code not uh bug fixes so please use layer 3 HA uh it's awesome um some links uh the links uh for hands uh um benchmarks and scale tests can be found here and uh we have time for a few questions if you go back to the links page that they want to take a picture of it yeah if you want to take uh pictures and questions any questions somebody so did you guys ever stop for a moment to think this solution is way too complex do we do do we solve this problem differently I mean what you're trying to do is you're trying to get packets from one compute node to another right there is no routing in the way we know where they are to begin with and you're saying well I will not add any intelligence at the node to know where to send the the packets I'll add another node that will do routing right then I bottleneck it I have a point of failure so I need to start scaling it so I need HA solutions and now you're saying we have races even in the API right so when do you stop and say is this the right solution right so the sending traffic between two compute nodes the DVR plus this is mainly now to handle the source NAT case where everybody wants to share a single IP address for a whole tenant's worth of traffic and we need to coordinate the use of that address so your testing was east west yeah that is just north south right right but if you want you can skip the HA for east west by using DVR plus HA okay so then but then this is just for s then then in that case as it would just be s NAT for the l3 HA okay but it is yeah it's a valid question we kind of go through a lot of work to make this single source NAT IP address usable across a whole bunch of locations by putting it on the network node then we have to make it highly available but this is people are concerned about ipv4 usage so it's a difficult problem to overcome oh hey look chris go first um as someone who actually ran into the uh the uh multiple actives on after a reboot uh one thing that i saw was uh is there the the option to do a policy uh or some other mechanism to actually verify there's a stat there's a configuration that is in a valid one such as multiple actives um do you remember what the the bug fix was for that yeah actually the bug fix this uh we started using port status so we check the port for 3ch the bug that was with two masters the problem is that um the l2 agent started later than l3 agent so the ch port wasn't active and when uh the uh the problem actually so when l3 agent started rescheduling router he thinks that the ch port was okay but it wasn't so we start using uh start checking uh there if a chip port is active so now actually this problem shouldn't be there but uh we're at in newton and i think we ported it in mitaka even somehow so in mitaka and later it should be fixed and we're also working on uh liberty i think and uh back porting it to liberty yeah but actually at this moment there are some problems so it's we're working on liberty uh i know my question uh hang on he was and he was next any plans for back putting all these bugs to mitaka liberty so quite a few have been back ported to mitaka i think liberty's hit a point now though where it's a lot of security like critical only fixes um so i'm not sure how many of them will be able to get back all the way to liberty now yeah we're trying to uh back port as many patches as we can and and and as many uh fixes but some of the uh fixes depend on some database uh changes which can't be bucked can't be back ported because uh database changes all right um we're actually out of time i think maybe just one more that's quick maybe we'll see my question i have two questions but they're related one of them is have you tried to you think you hope your thread safe is it have you proved the algorithm and you're just worried about the code or have you just uh because have you proved your algorithms thread safe that's my question so well so there's no um it's not multi-threading it's eventlet coroutine based switching inside of python there's no shared data structures between coroutines so we don't have to worry about concurrent access to the same data structure it's about coordinating around the the database and so it's just just through testing right is it's there's not like a formal proof of it but it's and and what do you do when you add a new now you add a new api call that might interfere do you test it is is there a way for you to check when someone puts pushes code instead of just running it just when you're thinking about new apis how do you know that they're not going to interrupt it's very ad very ad hoc it just requires testing say if it breaks stuff and then just knowledge of the database and knowing what's a distinct unit and if it's if it's a composite this is one of the rare like resources in neutron that's made up of other sub resources so it's not a very common problem normally we can wrap everything up in a single transaction and be good to go just add here that actually we try i'm trying to check all the changes that we got uh on a 3ha on virtual environment with several nodes at least so ransom rally test and check that it doesn't get worse so from the some point i do it regularly so thank you thank you


I'm going to give you an overview of network visualization and analysis with SiteEscape. It's a pretty simple, quick overview. And the reason why we're doing this now, we gave everybody pre-reading that covers most of what we want to talk about. And you also, hopefully, were able to install SiteEscape on your computer. Was there anyone, I don't know if we asked this earlier, was there anyone who wasn't able to install SiteEscape or had problems installing SiteEscape on their computer that we could look at? So we're going to talk about network analysis and visualization. And then for the later part of the afternoon, we're going to return to enrichment analysis. But we're going to have a network view of the results which are helpful. And so that's why we're doing SiteEscape first. So it's a little bit of a diversion, but we're still going to come back and focus on the main topic of today, which is enrichment analysis. OK, so this morning I introduced you to this workflow where we are this sort of generic pathway analysis workflow. There was another published paper that we published quite a while ago. It's a little bit out of date. We're working on creating an update version of this, but that's not ready yet for this course. That focused on using biological networks and gene expression analysis to analyze gene expression data using SiteEscape. So I'm just referencing this here in case you're interested in that. So I'm going to talk about networks in general, what they mean, network visualization, then SiteEscape, and then a little bit on network analysis. OK, so I included this paper in the pre-reading. That's just a short few pages. That covers a lot about what I want to mention. The basic take-home message is that networks represent relationships. It can be different types of relationships. You can have nodes connected by physical, where the nodes can represent proteins. And the interactions can represent the edges or relationships between the nodes can represent protein-protein interactions. So we might call that a physical interaction, because the proteins are touching each other. You can have a regulatory interaction, like A activates B. You can have a genetic interaction, like two genes are synthetic lethal, which means that if you knock out gene A, nothing happens. You knock out gene B, nothing happens. But if you knock out genes A and B together, then the cell dies. So that means that those two genes are important. They're buffering each other. So one's a backup for the other. And you can have functional interactions, which basically means that the two genes are functionally related, based on the interaction. And there are lots of different types of interactions that are useful for relating the function of gene A to gene B, like sequence similarity. If one gene has very similar sequence to the other, it's likely the other gene has a similar function. If they're both co-expressed across many conditions, they probably have similar function. If they have similar protein domains, they have similar function, so you get the idea. The main reason why we are talking about networks, I'll tell you a few reasons. But they're useful just for discovering relationships in large data sets, which they're just much better than tables in Excel. So if you have a spreadsheet where you have columns of relationships, it's hard to see the big picture from that. So networks provide a visual way of seeing the big picture, and you can identify the relationships. And the patterns that we like to look for are covered in a few main patterns that we like to look for are covered in this primer or primer. It's also useful to visualize multiple data types together to see interesting patterns that you also couldn't see if you were looking at each of the data types separately as a table. And then network analysis is a very useful feature of the reason why we want to think about networks in biology. And I'll give you an example, a very, very basic example. Basically, the idea of network analysis is that there's a lot of biological questions that can be answered using network analysis algorithms. Many of these algorithms are worked out in the field of computer science. So in computer science, people have studied for many, many years. Going back to before computer science and math, even more than 100 years, people have studied the concept of a graph which in math and computer science means a network. We don't use the term graph too frequently here because if you just ask people what a graph is, usually people think of it as a plot, like graph paper. But just so you know that that's the terminology used, the jargon used in computer science. And they have the field of computer science called graph theory. And the interesting thing about graph theory is that they've developed a lot of algorithms for graphs that answer all sorts of questions. So this slide just illustrates an example. So how many people have heard of the concept of six degrees of separation? So a few people. So this is the idea that everybody in the world is connected to everybody else through a social network by at most six links. So if you take any two people in the world, there's some connection through Facebook or something of six links or less. This was popularized by an experiment by Stanley Milgram, who's a famous social psychologist, where he wanted to know how people know each other. He wanted to investigate that. So he designed an experiment where he asked people to email a postcard through their friends from Boston to New York. So people in Boston mailed postcards. And they didn't know the address. They just knew the name of the person and that they lived in New York. So each time a postcard was mailed and the next person got it, they were supposed to send a postcard back to the researchers. So they could track where the postcards were going. And they got a whole bunch of postcards actually through to the person. And they found out that the average length of time was only six steps. So that's where this idea came from. If you have a network of all sorts of people, like the Facebook network and how they're related, you might want to know how people are connected. And in graph theory, from the field of graph theory in computer science, there's an algorithm that has been mathematically proven to give you the shortest path between two points. And it's called shortest path by breadth first search. And if two nodes are connected, it will tell you that the shortest path, or if they're not connected, it will say they're not connected. And the nice thing about it is that it's mathematically proven to always be perfectly correct. So that can be used if you have a protein interaction network and you want to know if two proteins are connected and how they're connected. You can use that algorithm. You can just take it from computer science. You don't have to do all the theory improving, math proving that they did to make sure that it's running perfectly. And then you can just use it. Now, you might ask, well, is it biologically relevant to know that one protein is connected to another by a shortest path? Maybe that represents an important pathway. But maybe it's not really an important pathway. Maybe the path, those proteins, don't actually talk to each other. So you do need to consider the biological relevance of the result. But this example illustrates the point that you can take work from the field of computer science and use it to answer specific questions. And it saves a lot of time, especially if the question is a good match. So people of over the past 15 years of network research and biology have come up with all sorts of different methods to use these network algorithms to answer questions in biology. So tomorrow, Quaid will talk to you about gene function prediction. And this is based on the idea, the problem with those pointers that doesn't appear on my screen. Let's see if this works, yeah. So in gene function prediction, you look to see if you have a gene that you don't know what its function is, you can try to find out how it's connected via protein interactions and other types of interactions to other genes. And if you know the function of those other genes, then that might tell you something about the function of your gene of interest. So again, it's using a network idea. People have found that you can look for dense regions in a network. In a protein interaction network, dense regions represent protein complexes. So if you have a big protein interaction network that you might get from proteomics experiments, which started generating this data in around 2001, 2002, then you can run these graph clustering algorithms and identify all the protein complexes. And you can predict new complexes. People have used these to study network evolution, how pathways are changing between species. You can predict new interactions and new functional associations. So one algorithm that people used was try to find these dense regions in a network where there's a lot of connections, but some of the connections are missing. So maybe those connections just have not been identified experimentally. And I would predict that they should occur. And that actually works reasonably well at predicting new protein interactions. People have also analyzed a lot of disease data. Come up with a lot of interesting ways of using networks in disease informatics, I guess you could call it. So one interesting paper that's this pinnacle z or pinnacle z from UCSD tries to identify regions of a protein interaction network or another network that are transcriptionally active in a disease context versus control. And that might help you identify pathways that are or regions of a network that are correlated with the disease. People have used network information for diagnosis of disease, again, and it has some of the same advantages of pathway information. And also in kind of a GWAS type of approach. So a lot of the types of analyses that you can do with pathways, you can do with networks, and there's some things that you can do with networks that are more natural. But the nice thing is that there are plenty of a lot of methods out there that are useful, that could be useful for you. Networks are fairly simple. Like if I give you a protein interaction network, it's a model of what's going on in the cell, but it's missing a lot of information. So typically networks are frequently, and even pathways as well, are represented as static processes. So I guess I should say gene set versions of pathways. So you might not know that there's an ionic wave going through a neuron or a feedback loop that's negatively regulating the pathway in a network. There are mathematical simulation tools that are available if you really know a lot about the pathway and you want to simulate it and see how it works at every little microsecond. You can do that, but we're not covering that in this course. Usually to do that, you need to accumulate a lot of detailed data, which we often don't have. Another thing that's missing is detail about the atomic structure of the protein. So proteins have a structure. They have domains. They have phosphorylation sites. We usually don't see that in a network, although sometimes you do. And then context was mentioned earlier, the cell type in the developmental stage, we don't always see in the network. Usually when we look at a network, it's the combination of all the interactions in the organism, whether they're in one tissue or another. So if you want to filter that based on gene expression in the brain, for instance, then you can do that and get a more brain-relevant network. OK, so that was a very quick overview, again, because we provided pre-reading. And the main point is that networks are useful for looking at your data and identifying relationships in your data and patterns. It's very important when you see a network to understand what the nodes and edges mean. Often, in biology, nodes represent genes or proteins, but they don't have to. They could represent other biological concepts. And the edges often mean some functional interaction or physical interaction, but they could be sequence similarity relationships or something. So whenever you see a network, always ask what the nodes and edges mean. And there are many methods available for gene list and analysis using networks. So sometimes there's an overwhelming number of methods available. I'll show you in a minute the number of tools that are available for site escape. There's hundreds of them. So it's hard to know which tool is available for answering your question. So there's two ways of handling that. You can try to be an expert and read through everything, but that's very time consuming. Another way is if you have a very specific question, like I have my gene expression data, I want to find pathways that are predicting outcome or something, you can ask the question on a mailing list, like the bio stars or the site escape mailing list. And then with a specific question, people might be able to recommend a tool that exists for you. OK, so any questions? So the question is often the networks are visualized as flat plane. So they're 2D visualizations. You can make a 3D view. The 3D view of a network often is difficult to work with unless you can rotate it. So the reason is that when you view a network, you want to reduce the overlap between the circles and the lines. And so the layout algorithms work to reduce that overlap. If you have it in 3D and you don't get a chance to rotate it, there's a lot more overlap. And that overlap harms your ability to see the relationships, because sometimes you can get lost following an edge. So if you have a 3D view and you can rotate it, then you can eliminate that. And sometimes the 3D view helps you by putting more information on the screen. In terms of more dimensions in that, you could try to add more dimensions somehow, but usually we're limited to two or three dimensions. Can you do it in 3D? Yeah. So it's not often the question is, can you encode something else in the third axis? Yes, you could. And in network visualization, it's really up to your imagination, what you want to do, and how you want to encode the information. For certain types of things like that one, I'm not familiar with a feature in Sight Escape that lets you do that easily. But with simple scripting, you'd actually be able to do that. So you could do that. Any other questions? OK, so now I'm going to talk about Sight Escape. Just the basics, and I'm going to give a demo. So Sight Escape is a freely available network visualization and analysis tool originally developed in Seattle at the Institute for Systems Biology by Trey Eidecker and Benosz Wachowski. And then Trey moved around, and now he's in San Diego, and Beno's now in Paris. And as they moved around, they brought Sight Escape with them. And I'm one of the contributors of Sight Escape as well as some other people in San Francisco and some companies. So it's an open source project that has grown in popularity because multiple people have kind of contributed to it. It's a nice thing about open source projects. And this workshop series in general tries to highlight open source projects because they're freely available and scientists can contribute to them and make them better and customize them. So it's usually a more efficient way of getting things done. So from a development point of view. So Sight Escape provides as a default features to visualize and mostly visualize a network. And then you can download apps that help you with many other types of network analysis. You can, by default, you can manipulate networks, move them around. You can lay the networks out. And this helps you with visualization. You can filter and search in databases. And then the real power of Sight Escape is that there's an app store. And I just checked. I think there's over 280 apps. This is a bit of an outdated number here. But there are over 280 apps that have been developed by people the world over and contributed to the ecosystem of Sight Escape. And you can search on the app store to find analysis packages and other things that might be useful for you. So that's really the, there are other network visualization and analysis tools out there. There's a number of them. There's another one developed in Toronto called Navigator. And but Sight Escape is the most popular one. And it's mainly because it was freely available in open source from an early point. So a lot of people contributed to it. And because there's a lot of contributions, there's a lot of users, and there's a lot more features available for it. So this number is actually old. I think we're at 14,000 downloads a month, which is a lot. And 250,000 people have downloaded it in its lifetime. So because of that, there's a lot of users on the mailing list, a lot of documentation, a lot of answers to questions online, and a lot of apps that people have contributed. You can build your own apps. Right now, it requires programming in Java. In the future, it'll probably be, we're trying to make it easier. And so the main take home message here is that it's an active community, which you can take advantage of, not trying to sell Sight Escape. You can use other tools if you want, but it is probably the most, some of these other network analysis packages might have unique features. But Sight Escape has the, and you can use them if they're useful, Sight Escape has the most amount of features associated with it. It's a nice active community. We had a Sight Escape conference a few years ago where people spelled out Sight Escape here. And OK, so again, I'm not going into too much detail here because we gave people pre-reading, so we could save time during the lectures.


Hello everyone, Myself Prithviraj Thambe working as an assistant professor in Department of Electronics Engineering at Vulture Nation of Technology, Solapur. Welcome to this session. In this session, we are going to learn general purpose IO port of LPC176 set. At the end of this session, student will be able to illustrate the structure of GPIO related memory map registers. And they also configure desired functions of port pins of LPC176 set outline. Let us understand the ports of LPC176 set. So LPC176 set comes with five general purpose bi-directional digital IO ports, port 0 to port 4. They supports fast GPIO that is enhanced or accelerated features. Additionally, port 0 and port 2 pins can be used to generate a single interrupt. GPIO ports are generally used for general purpose input to output interfacing, driving indicators like LEDs, controlling off-grid devices like relays or sensing digital inputs. So let us see each port one by one, port 0, total 20 pins can be used as a GPIO. But you will see that P0.12, 13, 14 and 31 are reserved. They are not available for the user. For port 1, 24 pins are available and P1.2, 3, 5, 6, 7, 11, 12 and 13 are reserved. Port 2, total 14 pins are available, P2.14 through P2.31 are not available. For port 3, 2 pins are available only 3.25 and 3.26. Remaining pins are again reserved. Similarly for port 4, again 2 pins are available, 4.28 and 4.29 and the remaining pins are again reserved. Let us calculate the total number of GPIO pins available. So yes, total up to maximum 17 GPIO pins are available for the user. Now these ports can be controlled by 5 groups of memory map registers. They are byte addressable, half word addressable or word addressable. So fast IO direction control register, fast IO mask register, fast IO set register, fast IO clear register and fast IO pin value register. So these 5 registers you can use to use GPIO port functionality. So all these GPIO registers are as I said byte, half word and word addressable. So byte and half word addressable allows the user easier and faster access to the physical port pins. They are all bit level control registers means a single instruction controls any number of bits or pins in one port. So let us see general purpose IO port direction control register. So this word accessible fast IO DR register is a third bit wide controls the direction of each pin of port X. So here you will see that fast IO zero direction control register is rewriteable and the memory address is shown here. So total number of registers, direction control registers are 0, 1, 2, 3, 4. All the registers are rewriteable and the default values are 0, 0, 0. The meaning is all the GPIO port pins by default set to inputs after reset. So this fast IO direction control register controls the direction of each port pin of port X. Example fast IO zero direction control register bit zero controls P0.0 and bit 31 controls P0.31. So direction control bit is effective only when that particular pin GPIO function is selected. Again you have two values 0 and 1. So 0 means that pin will be configured as input and 1 means that pin will be configured as output. Now let us solve some examples how to set the directions. So let us use fast IO zero direction control register which selects the direction of port zero pins. So we need to follow some steps. So we need to first logical and the default contents of this register with a mask which is used to preserve the direction of other port pins which we are not interested and then logical are the result with a desired direction value. So let us set the direction of P0.4 which is a port zero pin as an input. So here you will see that the there are some predefined directions and we are interested in only P0.4 direction. So this predefined direction is the output and we need to change it to input. So first ended with this mask then or with the desired value we set the direction of P0.4 as a input. So after writing all these combinations in hex you will get these values.  You can skip this step. Also keep in mind refer the standard header file LPC17XX.H for the standard definition of all these registers. Let us configure the P0.5 pin of port zero as the output. So here again we have a 32 bit pattern and we are interested in now bit five because bit five is responsible for setting the direction of P0.5. So here by default the direction of P0.5 set as input and we need to change it to the output.  So zero value for bit five and one for the remaining bits. So after ending you will get that the direction of remaining pins are preserved only the direction of P0.5 is changed and we are ordering our desired value or desired direction for output which is one. So after ordering the desired value we will get final value.  So FIO zero DR register odd with itself zero X zero zero zero zero two zero. So these statements change the direction of P0.5 as a output. You can skip this masking part and for writing your C program again refer a header file LPC 17 XX dot H for the standard definition of all these registers for further learning you can read LPC 16 set data sheet UM10360. Thank you.


The rapid rise of ransomware attacks has added yet another challenge that business technology executives have to worry about these days. Cloud storage, immutability, and air gaps have become a must have arrows in the quiver of organizations data protection strategies. But the important reality that practitioners have embraced is data protection, it can't be an afterthought or a bolt on it, has to be designed into the operational workflow of technology systems. The problem is oftentimes data protection is complicated with a variety of different products, services, software components, and storage formats. This is why object storage is moving to the forefront of data protection use cases because it's simpler and less expensive. The put data, get data syntax has always been alluring but object storage historically was seen as this low-cost niche solution that couldn't offer the performance required for demanding workloads, forcing customers to make hard trade-offs between cost and performance. That has changed. The ascendancy of cloud storage generally in the S3 format specifically has catapulted object storage to become a first-class citizen in a mainstream technology. Moreover, innovative companies have invested to bring object storage performance to parity with other storage formats. But cloud costs are often a barrier for many companies as the monthly cloud bill and egress fees in particular steadily climb. Welcome to secure storage hot takes. My name is Dave Vellante and I'll be your host of the program today where we introduce our community to Wasabi, a company that is purpose-built to solve this specific problem with what it claims to be the most cost-effective and secure solution on the market. We have three segments today to dig into these issues. First up is David Friend, the well-known entrepreneur who co-founded Carbonite and now Wasabi. We'll then dig into the product with Drew Schlesel of Wasabi and then we'll bring in the customer perspective with Kevin Warrenda of the Hotchkiss Cool. Let's get right into it. We're here with David Friend, the president and CEO and co-founder of Wasabi. The hot storage company, David, welcome to theCUBE. Thanks, Dave, nice to be here. Great to have you. Look, you hit a home run with Carbonite back when building a unicorn was a lot more rare than it has been in the last few years. Why did you start Wasabi? Well, when I was still CEO of Wasabi, my genius co-founder, Jeff Flowers, and our chief architect came to me and said, you know, when we started this company, a state-of-the-art disk drive was probably 500 gigabytes and now we're looking at eight terabyte, 16 terabyte, 20 terabyte, even 100 terabyte drives coming down the road. And sooner or later, the old architectures that were designed around these much smaller disk drives is gonna run out of steam because even though the capacities are getting bigger and bigger, the speed with which you can get data on and off of a hard drive isn't really changing all that much. And Jeff Force saw a day when the architectures of sort of legacy storage like Amazon S3 and so forth was going to become very inefficient and slow. And so he came up with a new highly parallelized architecture and he said, I wanna go off and see if I can make this work. So I said, you know, good luck, go to it. And they went off and spent about a year and a half in the lab designing and testing this new storage architecture. And when they got it working, I looked at the economics of this and I said, holy cow, we can sell cloud storage for a fraction of the price of Amazon, still make very good gross margins and it will be faster. So this is a whole new generation of object storage that you guys have invented. So I recruited a new CEO for Carbonite and left to found Wasabi because the market for cloud storage is almost infinite. When you look at all the world's data, IDC has these crazy numbers, 120 zettabytes or something like that. And if you look at that as the potential market size storing that data, we're talking trillions of dollars, not billions. And so I said, look, this is a great opportunity. If you look back 10 years, all the world's data was on prem. If you look forward 10 years, most people agree that most of the world's data is going to live in the cloud. We're at the beginning of this migration. We've got an opportunity here to build an enormous company. That's very exciting. I mean, you've always been a trend spotter and I want to get your perspectives on data protection and how it's changed. It's obviously on people's minds with all the ransomware attacks and security breaches but thinking about your experiences and past observations, what's changed in data protection and what's driving the current very high interest in the topic? Well, I think from a data protection standpoint, immutability, the equivalent of the old worm tapes but applied to cloud storage is become core to the backup strategies and disaster recovery strategies for most companies. And if you look at our partners who make backup software like Veeam, Commvault, Veritas, ArcServe and so forth, most of them are really taking advantage of mutable cloud storage as a way to protect customer data, customer's backups from ransomware. So the ransomware guys are pretty clever and they discovered early on that if someone could do a full restore from their backups, they're never going to pay a ransom. So once they penetrate your system, they get pretty good at sort of watching how you do your backups.  And that's where immutability comes in. In the old days, you wrote what was called a worm tape, write once, read many. And those could not be overwritten or modified once they were written. And so we said, let's come up with an equivalent of that for the cloud. And it's very tricky software. You know, it involves all kinds of encryption algorithms and blockchain and this kind of stuff. But the net result is, if you store your backups in immutable buckets in a product like Wasabi, you can't alter it or delete it for some period of time. So you could put a timer on it, say a year or six months or something like that. Once that date is written, you know, there's no way you can go in and change it, modify it or anything like that, including even Wasabi's engineers. So David, I want to ask you about data sovereignty. It's obviously a big deal, I mean, especially for companies with a presence overseas, but which really is any digital business these days. How should companies think about approaching data sovereignty? Is it just large firms that should be worried about this? Or should everybody be concerned? What's your point of view? Well, all around the world, countries are imposing data sovereignty laws. And if you're in the storage business like we are, if you don't have physical data storage in country, you're probably not gonna get most of the business. You know, since Christmas, we've built data centers in Toronto, London, Frankfurt, Paris, Sydney, Singapore, and I probably forgotten one or two. But the reason we do that is twofold. One is, you know, if you're closer to the customer, you're gonna get better response time, lower latency. And that's just a speed of light issue. But the bigger issue is if you've got financial data, if you have healthcare data, if you have data relating to security like surveillance videos and things of that sort, most countries are saying that data has to be stored in country. So you can't send it across borders to some other place. And if your business operates in multiple countries, you know, dealing with data sovereignty is going to become an increasingly important problem. So in May of 2018, that's when the fines associated with violating GDPR went into effect. And GDPR was like this mainspring of privacy and data protection laws. And we've seen it spawn other public policy things like the CCPA and it continues to evolve. We've seen judgments in Europe against big tech and this tech lash that's in the news in the US and the elimination of third party cookies. What does this all mean for data protection in the 2020s?  So, you know, it's getting to be increasingly complicated because GDPR, for example, imposes different standards from the kind of privacy standards that we have here in the US. Canada has a somewhat different set of data sovereignty issues and privacy issues.  And this makes it even more difficult for enterprises to run their own, you know, infrastructure because companies like Wasabi, where we have physical data centers in all kinds of different markets around the world. And we've already dealt with the business of how to meet the requirements of GDPR and how to meet the requirements of some of the countries in Asia and so forth. You know, rather than an enterprise doing that just for themselves, if you running your applications or keeping your data in the cloud, you know, now a company like Wasabi with, you know, 34,000 customers, we can go to all the trouble of meeting these local requirements on behalf of our entire customer base. And that's a lot more efficient and a lot more cost effective than if each individual country has to go deal with the local regulatory authorities. Yeah, it's compliance by design, not by chance. Okay, let's zoom out for the final question, David. You're thinking about the discussion that we've had around ransomware and data protection and regulations, what does it mean for a business's operational strategy and how do you think organizations will need to adapt in the coming years? Well, you know, I think there are a lot of forces driving companies to the cloud. And, you know, and I do believe that if you come back five or 10 years from now you're going to see a majority of the world state is going to be living in the cloud. And I think storage, data storage is going to be a commodity, much like electricity or bandwidth. And it's going to be done right. It will comply with the local regulations. It'll be fast, it'll be local. And there will be no strategic advantage that I can think of for somebody to stand up and run their own storage, especially considering the cost differential. You know, the most analysts think that the full all-in costs of running your own storage is in the 20 to 40 terabytes per month range.  And so I think people are learning how to, are learning how to deal with the idea of an architecture that involves storing your data in the cloud as opposed to, you know, storing your data locally. Wow, that's, that's like a 6x more expensive in the cloud. It's more than 6x. Yeah, I think you did, go get it please. In addition to which, you know, just finding the people to babysit this kind of equipment has become nearly impossible today. Well, and with the focus on digital business, you don't want to be wasting your time with that kind of heavy lifting. David, thanks so much for coming on theCUBE. Great Boston entrepreneur. We've followed your career for a long time and looking forward to the future. Thank you. Okay, in a moment, Drew Schlessel will join me and we're going to dig more into product. You're watching theCUBE, the leader in enterprise and emerging tech coverage. Keep it right there. Oh, Brandon Sales got an email. Click here for a trip to Bombay. It's not even called Bombay anymore but she clicked it anyway. And now our data's been held hostage. And now we're on a sinking ship and the hackers in our system just cause Brenda wanted a trip.  Joining me now is Drew Schlessel who is the senior director of product marketing at Wasabi. Hey Drew, good to see you again. Thanks for coming back on theCUBE. Dave, great to be here. Great to see you. All right, let's get into it. You know, Drew, prior to the pandemic, zero trust, just like kind of like digital transformation was sort of a buzzword. And now it's become a real thing, almost a mandate. What's Wasabi's take on zero trust? So absolutely right. It's been around a while and now people are paying attention. Wasabi's take is zero trust is a good thing. There are too many places where the bad guys are getting in. And I think of zero trust as kind of smashing laziness. It takes a little work, takes some planning, but done properly and using the right technologies, using the right vendors, the rewards are of course tremendous. You can put to rest the fears of ransomware and having your systems compromised. Well, and we're going to talk about this, but there's a lot of process and thinking involved and design in your zero trust. And you don't want to be wasting time messing with infrastructure. So we're going to talk about that. There's a lot of discussion in the industry, Drew, about immutability and air gaps. I'd like you to share Wasabi's point of view on these topics. How do you approach it and what makes Wasabi different? So in terms of air gap and immutability, the beautiful thing about object storage, which is what we do all the time, is that it makes it that much easier to have a secure immutable copy of your data someplace that's easy to access and doesn't cost you an arm and a leg to get your data back. We're working with some of the best, partners in the industry. We're working with folks like Veeam, Commvault, Arc, Marquis, MSP360, all folks who understand that you need to have multiple copies of your data. You need to have a copy stored offsite and that copy needs to be immutable. And we can talk a little bit about what immutability is and what it really means. You know, I wonder if you could talk a little bit more about Wasabi's solution because sometimes people don't understand.  You're not building on other people's public clouds and storage is the one use case where it actually makes sense to do that. Tell us a little bit more about Wasabi's approach and your solution. Yeah, I appreciate that. So there's definitely some misconception. We are our own cloud storage service. We don't run on top of anybody else, right? It's our systems. It's our software deployed globally and we interoperate because we adhere to the S3 standard. We interoperate with practically hundreds of applications. Primarily in this case, right? We're talking about backup and recovery applications and it's such a simple process, right? I mean, just about everybody who's anybody in this business protecting data has the ability now to access cloud storage. And so we've made it really simple. In many cases, you'll see Wasabi as, you know, listed in the primary set of available vendors and, you know, put in your private keys, make sure that your account is locked down properly using, let's say, multi-factor authentication and you've got a great place to store copies of your data securely. I mean, we just heard from David Friend. I did my math right. He was talking about, you know, one-sixth the cost per terabyte per month, maybe even a little better than that. How are you able to achieve such attractive economics? Yeah, so, you know, I can't remember how to translate my fractions into percentages, but I think we talk a lot about being 80%, right? Less expensive than the hyperscalers. And, you know, we talked about this at V-Mon, right? There's some secret sauce there. And, you know, we take a different approach to how we utilize the raw capacity to the effective capacity. And the fact is we're also not having to run, you know, a few hundred other services, right? We do storage, plain and simple, all day, all the time.  Customers love it, right? They see the fact that their data is growing, 40, 80% year over year. They know they need to have some place to keep it secure. And, you know, folks are flocking to us in droves. In fact, we're seeing a tremendous amount of migration actually right now, multiple petabytes being brought to Wasabi because folks have figured out that they can't afford to keep going with their current hyperscaler vendor. And immutability is a feature of your product, right? What's the feature called? Can you double click on that a little bit? Yeah, absolutely. So the term in S3 is object lock. And what that means is your application will write an object to cloud storage and it will define a retention period, let's say a week. And for that period, that object is immutable, untouchable, cannot be altered in any way, shape or form. The application can't change it. The system administration can't change it. Wasabi can't change it, okay? It is truly carved in stone. And this is something that it's been around for a while but you're seeing a huge uptick, right? In adoption and support for that feature by all the major vendors. And I named off a few earlier. And the best part is that with immutability comes some sense of, well, it comes with not just a sense of security, it is security, right? When you have data that cannot be altered by anybody, even if the bad guys compromise your account, they steal your credentials, right? They can't take away the data. And that's a beautiful thing, a beautiful, beautiful thing. And you look like an S3 bucket, is that right? Yeah, yeah. I mean, we're fully compatible with the S3 API. So if you're using S3 API based applications today, it's a very simple matter of just kind of redirecting where you want to store your data. Beautiful thing about backup and recovery, right? That's probably the simplest application, simple being a relative term, as far as lift and shift, right? Because that just means for your next full, right? Point that at Wasabi, retain your other fulls for whatever, 30, 60, 90 days. And then once you've kind of made that transition from vine to vine, you're off and running with Wasabi. I talked to my open about the allure of object storage, historically, the simplicity of the get put syntax, but what about performance? Are you able to deliver performance that's comparable to other storage formats? Oh yeah, absolutely. And we've got the performance numbers on the site to back that up. But I forgot to answer something earlier, right? You said that immutability is a feature. And I want to make it very clear that it is a feature, but it's an API request, okay? So when you're talking about gets and puts and so forth, the comment you made earlier about being 80% more cost effective or 80% less expensive, that API call is typically something that the other folks charge for, right? And I think we used the metaphor earlier about the refrigerator, but I'll use a different metaphor today, right? You can think of cloud storage as a magical coffee cup, right? It gets as big as you want to store as much coffee as you want and the coffee's always warm, right? And when you want to take a sip, there's no charge. You want to pop the lid and see how much coffee is in there, no charge. And that's an important thing because when you're talking about millions or billions of objects, and you want to get a list of those objects or you want to get the status of the immutable settings for those objects, anywhere else it's going to cost you money to look at your data with Wasabi, no additional charge. And that's part of the thing that sets us apart. Excellent, thank you for that. So you mentioned some partners before. How do partners fit into the Wasabi story? Where do you stop? Where do they pick up? And what do they bring? Can you give us maybe a paint a picture for us, an example or two? Sure. So again, we just do storage, right? That is our sole purpose in life is to safely and securely store our customer's data. And so they're working with their application vendors, whether it's active archive, backup and recovery, IoT surveillance, media and entertainment workflows, right? Those systems already know how to manage the data, manage the metadata. They just need some place to keep the data that is being worked on, being stored and so forth, right? So just like, you know, plugging in a flash drive on your laptop, right? You literally can plug in Wasabi as long as your applications support the API. Getting started is incredibly easy, right? We offer a 30 day trial, one terabyte and most folks find that within, you know, probably a few hours of their POC, right? It's giving them everything they need in terms of performance, in terms of accessibility, in terms of sovereignty. I'm guessing you talked to, you know, Dave Friend earlier about data sovereignty, right? We're a global company, right? So there's got to be probably, you know, wherever you are in the world, someplace that will satisfy your sovereignty requirements as well as your compliance requirements. We did talk about sovereignty, Drew. This is really, what's interesting to me, I'm a bit of an industry historian. When I look back to the early days of cloud, I remember the large storage companies, you know, their CEOs would say, we're going to have an answer for the cloud and they would go out and for instance, I know one bought a competitor of Carbonite and couldn't figure out what to do with it. They couldn't figure out how to compete with the cloud, in part because they were afraid it was going to cannibalize their existing business. I think another part is because they just didn't have that imagination to develop an architecture that in a business model that could scale to see that you guys have done that is I love it because it brings competition, it brings innovation and it helps lower clients cost and solve really nagging problems like, you know, ransomware, mutability and recovery. I'll give you the last word, Drew. Yeah, you're absolutely right. You know, the on-prem vendors, they're not going to go away anytime soon, right?  That requires sub millisecond type of processing. But once you've done that work, right? You want to store that data for a long, long time and you're going to want to also tap back into it later.  Yeah, don't believe the hype. You're still going to pay way more for that than you would with just a wasabi-like hot cloud storage system. And, you know, we don't compete with our partners, right? We compliment, you know, what they're bringing to market in terms of the software vendors, in terms of the hardware vendors, right? We're a beautiful component for that hybrid cloud architecture. And I think folks are gravitating towards that. I think the cloud is kind of hitting a new gear, if you will, in terms of adoption and recognition for the security that they can achieve with it. All right, Drew, thank you for that. Definitely, we see the momentum. In a moment, Drew and I will be back to get the customer perspective with Kevin Morenda, who's the director of information technology services at the Hotchkiss School. Keep it right there. Hey, I'm Nate, and we wrote this song about ransomware to educate people. People like Brenda. Oh, God, I'm so sorry. We know you are, but Brenda, you're not alone. This hasn't just happened to you. No! Colonial oil pipeline had a guy who didn't change his password.  His password leaked. The data was breached, and it cost his company four million bucks. A figure of date was sent to people working for the meat company, JBS. That's pretty clever. Instead of getting new features, they got hacked and had to pay the largest crypto ransom ever. And $20 billion, billion with a B, have been paid by companies in health care. If you wonder why your premium keeps going up, up, up, up, up, now you're aware.    It seems too good to be true. Like a reason to let it go. Just check with me first and I'll give you. It's a pretty sweet car. I'll keep it out there. You know, it's not very nice. To learn more about things. Do that. Because if we are... Drew and I are pleased to welcome Kevin Warrenda, who's the director of information technology services at the Hotchkiss School, a very prestigious and well-respected boarding school in the beautiful northwest corner of Connecticut. Hello, Kevin. Hello, it's nice to be here. Thanks for having me. Yeah, you bet. Hey, tell us a little bit more about the Hotchkiss School and your role. Sure, the Hotchkiss School is an independent boarding school, grades nine through 12, as you said, very prestigious and in an absolutely beautiful location on the deepest freshwater lake in Connecticut. We have 500 acre main campus and a 200 acre farm down the street. My role as the director of information technology services essentially to oversee all of the technology that supports the school, operations, academics, sports, everything we do on campus. Yeah, and you've had a very strong history in the educational field from that lens. What's the unique or not unique but depressing security challenge that's top of mind for you? I think that it's clear that educational institutions are a target these days, especially for ransomware. We have a lot of data that can be used by threat actors and schools are often underfunded in the area of IT security, IT in general sometimes. So I think threat actors often see us as easy targets or at least worthwhile to try to get into.  So there really are some, are there any specific data privacy concerns as well around student privacy or regulations that you can speak to? Certainly because of the fact that we're an independent boarding school, we operate things like even a health center. So data privacy regulations across the board in terms of just student data rights and FERPA, some of our students are under 18. So data privacy laws such as COPA apply, HIPAA can apply, we have PCI regulations with many of our financial transactions, whether it be fundraising through alumni development or even just accepting the revenue for tuition. So it's a unique place to be. Again, we operate very much like a college would, right? We have all the trappings of a private college in terms of all the operations we do. And that's what I love most about working education is that it's all the industries combined in many ways. Very cool. So let's talk about some of the defense strategies from a practitioner point of view. Then I want to bring in Drew to the conversation. So what are the best practice and the right strategies from your standpoint of defending your data? Well, we take a defense and depth approach. So we layer multiple technologies on top of each other to make sure that no single failure is the key to getting beyond those defenses. We also keep it simple. And I think there's some core things that all organizations need to do these days, including vulnerability scanning, patching, using multi-factor authentication and having really excellent backups in case something does happen. Drew, are you seeing any similar patterns across other industries or customers? I mean, I know we're talking about some uniqueness in the education market, but what can we learn from other adjacent industries? Yeah, you know, Kevin is spot on and I love hearing what he's doing. Going back to our prior conversation about zero trust, that defense and depth approach is beautifully aligned with the zero trust approach, especially things like multi-factor authentication always shocked at how few folks are applying that very, very simple technology and across the board. I mean, Kevin is referring to financial industry, healthcare industry, even the security and police, right? They need to make sure that the data that they're keeping evidence, right? Is secure and immutable, right? Because that's evidence. Kevin, paint a picture for us, if you would. So you were primarily on-prem, looking at potentially using more cloud. You were a VMware shop, but tell us, paint a picture of your environment, kind of the applications that you support and the kind of, I want to get to the before and the after wasabi, but start with kind of where you came from. Sure, well, I came to the Hatchett School about seven years ago and I had come most recently from public K-12 and municipal. So again, not a lot of funding for IT in general, security or infrastructure in general. So Nutanix was actually a hyper-converged solution that I implemented at my previous position. So when I came to Hatchett's and found mostly on-prem workloads, everything from the student information system to the card access system that students would use, financial systems, they were almost all on-premise, but there were some new SaaS solutions coming in play. We had also taken some time to do some business continuity planning, in the event of some kind of issue. I don't think we were thinking about the pandemic at the time, but certainly it helped prepare us for that. So as different workloads were moved off to hosted or cloud-based, we didn't really need as much of the on-premise compute and storage as we had and it was time to retire that cluster. And so I brought the experience I had with Nutanix with me and we consolidated all that into a hyper-converged platform running Nutanix AHV, which allowed us to get rid of all the cost of the VMware licensing as well.  Yeah, AHV is the Acropolis Hypervisor and so you migrated off of the VMware avoidant, the V-Tax avoidance, that's a common theme among Nutanix customers. Now, did you consider moving into AWS? What was the catalyst to consider Wasabi as part of your defense strategy? We were looking at cloud storage options and they were just all so expensive, especially in egress fees to get data back out. Wasabi came across our desks and it was such a low barrier to entry to sign up for a trial and get a terabyte for a month. And then it was $6 a month for a terabyte after that. So we can try this out in a very low stakes way to see how this works for us. And there was a couple of things we were trying to solve at the time, it wasn't just a place to put backup, but we also needed a place to have some files that might serve to some degree as a content delivery network. Some of our software applications that are deployed through our mobile device management needed a place that was accessible on the internet that they could be stored as well. So we were testing it for a couple of different scenarios and it worked great. Performance wise, fast, security wise, it has all the features of S3 compliance that works with Nutanix and anyone who's familiar with S3 permissions can apply them very easily. And then there was no egress fees. We can pull data down, put data up at will and it's not costing us any extra, which is excellent because especially in education we need fixed costs. We need to know what we're going to spend over a year before we spend it and not be hit with bills for egress or because our workload or our data storage footprint grew tremendously. We can't have the variability that the cloud providers would give us. So Kevin, you explained to your hypersensitive about security and privacy for obvious reasons that we discussed. Were you concerned about doing business with a company with a funny name? Was it the trial that got you through that not whole? How did you address those concerns as an IT practitioner? Yeah, anytime we adopt anything we go through a risk review. So we did our homework and we checked. The funny name really means nothing. There's lots of companies with funny names. I think we don't go based on the name necessarily, but we did go based on the history, understanding who started the company, where it came from and really looking into the technology and understanding that the value proposition, the ability to provide that lower cost is based specifically on the technology in which it lays down data. So having a legitimate reasonable excuse as to why it's cheap, we weren't thinking, well, you get what you pay for. It may be less expensive than alternative is, but it's not cheap. It's not, you know, it's reliable. That was really our concern. So we did our homework for sure before even starting the trial, but then the trial certainly confirmed everything that we had learned. Yeah, thank you for that. Drew, explain the whole egress charge. We hear a lot about that. What do people need to know? First of all, it's not a funny name. It's a memorable name, Dave. Just like the cube. Let's be very clear about that. Second of all, egress charges. So, you know, other storage providers charge you for every API call, right? Every get, every put, every list, everything, okay? It's part of their process. It's part of how they make money. It's part of how they cover the cost of all their other services. We don't do that. And I think, you know, as Kevin has pointed out, right? That's a huge differentiator because you're talking about a significant amount of money above and beyond what is the list price. In fact, I would tell you that most of the other storage providers, hyperscalers, you know, their list price, first of all, is far exceeding anything else in the industry, especially what we offer. And then, right, their additional costs, the egress costs, the API requests, can be two, three, 400% more on top of what you're paying per terabyte. So, you used a little coffee analogy earlier in our conversation. So, here's what I'm imagining. Like, I have a lot of stuff, right? And I had to clear up my bar and I put some stuff in storage, you know, right down the street. And I pay them monthly. I can't imagine having to pay them to go get my stuff. That's kind of the same thing here. Oh, that's a great metaphor, right? That storage locker, right? Yeah. Can you imagine every time you want to open the door to that storage locker and look inside having to pay a fee? No, no, that would be annoying. Or every time you pull into the yard and you want to put something in that storage locker, you have to pay an access fee to get to the yard, you have to pay a door opening fee, right?  It's your data, it's your storage, it's your locker. You've already paid the annual fee probably because they gave you a discount on that. So, why shouldn't you have unfettered access to your data? That's what Wasabi does. And I think, as Kevin pointed out, right? That's what sets us completely apart from everybody else. Okay, good, that's helpful. It helps us understand how Wasabi's different. Kevin, I'm always interested when I talk to practitioners like yourself in learning what you do, outside of the technology, what are you doing in terms of educating your community and making them more cyber-aware? Do you have training for students and faculty to learn about security and ransomware protection, for example? Yeah, cyber security awareness training is definitely one of the required things everyone should be doing in their organizations. And we do have a program that we use and we try to make it fun and engaging too. This is often the checking the box kind of activity, insurance companies require it, but we wanna make it something that people want to do and wanna engage with. So even last year, I think we did one around the holidays and kind of pointed out the kinds of scams they may expect in their personal life about shipping of orders and time for the holidays and things like that. So it wasn't just about protecting our school data. It's about the fact that protecting their information is something you do in all aspects of your life, especially now that the folks are working hybrid, often working from home with equipment from the school. The stakes are much higher. And people have a lot of our data at home. And so knowing how to protect that is important. So we definitely run those programs in a way that we want to be engaging and fun and memorable so that when they do encounter those things, especially email threats, they know how to handle them. So when you say fun, it's like you come up with an example that we can laugh at until of course we click on that bad link, but I'm sure you can come up with a lot of interesting and engaging examples. Is that what you're talking about, about having fun? Yeah, I mean, sometimes they are kind of choose your own adventure type stories. You know, they stop as they run. So they're telling a story and they stop and you have to answer questions along the way to keep going. So you're not just watching a video. You're engaged with the story of the topic. And that's what I think is memorable about it, but it's also, that's what makes it fun. It's not, you're not just watching some talking head saying, you know, to avoid shortened URLs or to talk to make sure you know the sender of the email. Now, you're engaged in a real life scenario of story that you're kind of following and making choices along the way and finding out, was that the right choice to make? Or maybe not. So that's what I think the learning comes in. Excellent. Okay, gentlemen, thanks so much. Appreciate your time. Kevin, Drew, awesome having you on theCUBE. My pleasure. Thank you. Yeah, great to be here. Thanks. Okay, at a moment, I'll give you some closing thoughts on the changing world of data protection and the evolution of cloud object storage. You're watching theCUBE, the leader in high tech enterprise coverage. Some things just don't make sense. Like showing up a little too early for the big game. How early are we? Couple months. Popcorn? On and off season, the Red Sox cover their bases with affordable, best in class cloud storage. These are pretty good seeds. Hey, have you guys seen a line for the bathroom? Wasabi hot cloud storage. It just makes sense. You don't think they make these in left hand, do you? We learned today how a serial entrepreneur along with his co-founder saw the opportunity to tap into the virtually limitless scale of the cloud and dramatically reduce the cost of storing data, while at the same time protecting against ransomware attacks and other data exposures with simple fast storage, immutability, air gaps, and solid operational processes. Let's not forget about that, okay? People and processes are critical. And if you can point your people at more strategic initiatives and tasks, rather than wrestling with infrastructure, you can accelerate your process redesign and support of digital transformations. Now, if you want to learn more about immutability and object block, click on the Wasabi resource button on this page or go to wasabi.com slash object block. Thanks for watching secure storage hot takes made possible by Wasabi. This is Dave Vellante for theCUBE, the leader in enterprise and emerging tech coverage. We'll see you next time.