is the second lecture now on Concernature Protocols. So now we're looking at the other class of Concernature Protocols called timestamp ordering. So just as a reminder for everyone, homework four is due a week from today on November 13th, and then project three is due next week on the 15th. Does anybody have any questions about project three? Yes. His question is, for project three, have we already covered everything that you need to know for project three? Yes. Everything would have been covered last week. You have to do two-phase locking as you build a lock manager and implement the internals of the lock manager. So when requests come in, you make a decision about whether to deny it or grant it, and then you have to do latch-crabbing on your B-plus tree. In the back, yes. So your question is, the wait-dive policy for the text book is vague? Can you be more specific? So let's take this offline. It should be straightforward, maybe I'm missing something. Yeah, I can ask later. Okay. For the upgrades, is that what's confusing about? I see what you're saying. Yeah. All right. Let's take this offline, and then we'll update the project classification. Okay. Then we should talk about lock upgrades as well. Okay. So, any other questions? All right. So again, to ground you with where we're at in the semester, we introduced Concurrency Control as the mechanism that the data system is going to use to allow multiple transactions to run at the same time, and to update and read from the database, but then we'll have the illusion that they are running in isolation and running by themselves. We said this before that this is when you're running at serializable isolation, the strongest guarantee you can provide. So, last week we talked about two-phase locking, where we said this is a pessimistic protocol where the transactions have to acquire the locks on the objects that they want to read or write to before the data system lets them actually do that. Then two phases were that you acquire the locks in the growing phase, and then as soon as you release one lock, now you're in the shrinking phase. Then we said there was two-phase locking where you just hold the locks to the very end. So, today we're talking about timestamp ordering. So, this is considered an optimistic protocol where we're not going to use any locks at all, although you can argue the timestamps that will maintain are essentially locks, but that's getting into semantics. But there's no locks at all, and you're going to assume that transactions aren't going to conflict, and you're going to let them just run, and you have to maintain some metadata to say what the transaction was doing while it runs, and then when you go to commit, you just check to see whether those requirements about the timestamp ordering was still satisfied. So, I'm going to talk about two types of, actually three types of a timestamp ordering protocol. I would say the first one I'm going to describe will be the standard textbook definition, and then we'll talk about the more complicated one, and then we'll talk about another variant that's actually used in some more recent systems. So, again, the high-level idea of timestamp ordering is that, again, we're going to use, we're going to assign transactions timestamps, and that's going to allow us to figure out what should be the order of their operations. When I say order, I mean that it's, again, it's the order that they're applied to the database. The database system is still allowed to interleave them in any way that it wants. Again, long as it produces a schedule that's equivalent to one of a schedule that produces a database that's equivalent to one that's generated by a serial ordering of our transactions. So, what we're going to introduce is this new timestamp function, TS, and for a given transaction handle or transaction ID, we will generate a unique timestamp for it. So, in our purposes, when we say T1, we'll say its timestamp is one, and there are a bunch of examples. So, the idea here is that the timestamp of the first transaction Ti, if that's less than the timestamp of transaction Tj, then the serial order would be one where Ti is executed first followed by Tj. Again, we can use these timestamps to figure out when these would occur. Again, high-level idea to think about this is that, we're assigning these timestamps on a per-transaction basis and a transaction can consist of one or more operations. So, we don't need to know about the individual timestamps of individual steps. It's really just about the transaction and all subsequent steps. So, the way you would assign a timestamp is through a couple of different ways. But a high-level, the key thing is that the timestamp always has to be unique for a transaction. We can't have transaction, the first transaction started at timestamp one, the second transaction started at timestamp two, or one as well, because those timestamps would be equal and then we can't make any decision about how to interleave their operations. So, they always has to be unique. It always has to be fixed for the transaction. Meaning, if we assign the transaction, a timestamp at that moment, later on the transaction keeps on running, it can't get a new timestamp. There are some variants of timestamp ordering protocols that where you can do that. But for our purposes in this lecture, for you guys to understand how timestamp ordering works, we'll assume that it's fixed. And then the other key thing is that the timestamp always has to be increasing. So, if we give a transaction timestamp 10, if there's some weird clock anomaly, where all of a sudden the clock drifts and we reset it with NTP or whatever, we can't now start assigning transactions timestamp two. Because that would sort of foul up this protocol. Now, I'm not gonna talk about this today, but I can talk about it in next lecture. There's obviously a problem of wraparound. If you have 32-bit timestamps, at some point, you're gonna have to come back to zero. Postgres hits this problem quite often. And in recent years, they've done a lot of good work and they can make it much better. But if, again, for our purposes, it's sort of keeping things simple. Just assume that the timestamp's always increasing up to infinity. So there's a bunch of different ways of how you can implement the timestamp allocator in your system. The most simplest way is to use the system clock. You can go to the OS and say, give me the current system time. And it can give you back that as milliseconds or maybe microseconds. There are, in the newer Intel CPUs, there's the RDTS or RDTC instruction that'll give you a sort of hardware timestamp. It doesn't map to actual the physical time, it's just the counter on the CPU. And again, same thing, it's always in increasing order. And that's actually much cheaper to do than get the system time, because that's a single instruction. I could also do this with a logical counter. So you could have a single global variable that's just a counter. And every time you need to allocate a new timestamp, you just add plus one to it over and over again. Or you can use a combination of these two. So in some systems, you can go say, give me the current time in milliseconds and then I'll have a global counter that you increase, the logical global counter that you increase for that millisecond tick. And this allows you to assign time stands for transactions, multiple time stands for transactions in the same millisecond. If you're executing transactions very quickly, then you would want something like this. The different systems do different things. The hybrid one is probably the best approach. We won't talk about this for this semester, but this will come up again in the advanced class. The problem with the system clock, obviously, is going to the kernel but the OS is an expensive call. The RDTC instruction makes this cheaper, but then you have to have the wraparound issue. We'll talk about next class. All right, so there's actually three protocols I want to talk about today. So there's basic timestamp ordering and then optimistic concurrency control, which is sort of an advanced version of timestamp ordering. And then the third one I forgot to put here is sort of partition-based timestamp ordering. That last protocol actually turns out to be the most simplest to understand. And in some cases it's actually the fastest, but OCC is usually the one that people implement. And when we see this, when we talk about multi-version concurrency control in the next class, most of the modern database systems that have come out in the last 10 years or so all use the OCC one, the middle guy, but in a multi-version setting. So one thing that's always problematic also too when we talk about timestamp ordering protocols is that the language gets reused, right? So the class of protocols we're talking about today are timestamp ordering protocols, of which there is a protocol called the basic timestamp ordering protocol. And all of these are considered optimistic schemes and technically are optimistic concurrency control schemes, but there's a protocol called optimistic concurrency control. So I'll try to be very clear as I go along. But for basic TO and OCC, these are specific implementations of algorithms and protocols that fall under the category of timestamp ordering, okay? Okay, so the basic timestamp ordering protocol is that the transactions are gonna be able to read and write objects in our database without locks. And then what's gonna happen is we're gonna maintain these timestamp tags for every single object in our database that corresponds to the last, the timestamp of the last transaction that read or wrote to that object. And then every time now a transaction wants to read an object, it has a timestamp and it compares its timestamp with either the read timestamp or the write timestamp of the object it's trying to access. And then that is used to determine whether it's allowed to do that or not, right? So again, when I said before that this is not using locks as we understood them under two-phase locking, but in some ways, setting the read and write timestamp is sort of like an implicit lock, like it prevents someone from doing something that you don't want them to do, but you don't have to release the lock in the same way you do in two-phase locking. So once you set the read timestamp, or the write timestamp, after you've completed your operation, you can keep on going and do whatever else you want you to do in your transaction, but anybody else that comes along has to check their timestamp against yours to see whether they're allowed to do that. All right, so when you read the basic idea is that if the timestamp of our transaction that we're trying to run, if their timestamp is less than the right timestamp of the last transaction that wrote to this object, then we're essentially trying to read into the future, which should not be allowed under serial ordering, right? So if this occurs, if your check fails this constraint, then the database system will have to abort you, and then when you restart, you'll come along with the same timestamp because now you'll be the older one and you can sort of keep on making forward progress. If you are less than that, then you're allowed to read the object, but then you're also updated read timestamp if your timestamp is greater than the current read timestamp. So what I mean by that is, again, these timestamps always have to be going in increasing order. So if I'm transaction one, I have timestamp one and your transaction two, if you read object X and you set the read timestamp to one, or sorry, to two, and I come along and I now try to set it to one, we don't want that to happen because now the reads are going back in time and as if the read from the second transaction, although it occurred first, it's as if it actually disappears and this causes problems in the protocol. Now one additional thing that we had to do under a basic timestamp ordering that you're not gonna do under two phase locking is that you're gonna have to copy the object that you read into a sort of private workspace or a local buffer for your transaction because that way if you go back and read that same object again, you'll be guaranteed that you'll get the same value. Now this is essentially a variant of multi-versioning which we'll talk about on Wednesday, but we need to do this because there's no locks on the object. I read the object once at timestamp one, then transaction two at timestamp two comes along and writes to it. When I come back and try to read it again, my old value got overwritten and so I would have to abort because now I'd be reading something that I shouldn't be able to read. So to avoid that, if you wanna have repeatable reads with less aborts, you just make a copy of the object you read and put it in your local buffer. Yes, in the back. So this point is if you're running at a lower isolation level, so this would be read committed. If you're running at recommitted or less, then you don't have to do this. You have to do this to maintain serializable isolation level. All right, now for writes, it's essentially the same thing, but we have to check both the read timestamp of our object and the write timestamp of the object as well. So the read timestamp basically says that if our transaction is trying to write to an object, but there's a transaction in the future that already read the current value that we wanna replace, then we have to abort ourselves because we would write something that the other transaction didn't see, which it should have if we were executing things in serial order. The same thing for the write timestamp. If we try to write to the object and are overwriting a modification made by a transaction in the future, then that again violates the serializable ordering or serial ordering because our write should occur before the other one, but in real time, the second write actually occurred first. So by checking both the read timestamp and the write timestamp, you prevent from overwriting things from future transactions and you prevent future transactions from not being able to read your writes. So for this, if you're allowed to do the write, you don't take the max of the write timestamp, you always set it to whatever you are, because we do that additional check to make sure that our write timestamp is greater than the current one, so that guarantees that the write timestamp of the object is always in increasing order, always increasing. And again, the same thing, if you want to ensure that you have repeatable reads, if you're running on the serializable isolation level, then you make the copy of the transactions, make the copy of the object in the transaction buffer. This idea, the private workspace, will be the key idea of OCC, which we'll talk about next. All right, so let's look at a really basic example here, right? So you have two transactions, T1, T2. T1 was a read A, sorry, read B then read A, and then transaction T2 wants to read B, write B, then read A and write A. So the first thing to point out is that under basic timestamp ordering, at the moment the transaction starts, the database system will assign it its timestamp. This is not true for all timestamp ordering protocols, but it's true for this one. So let's say now we have a single thread, and this little blue arrow represents the current location of where that thread is executing these two transactions. So T1 starts, it does a read on B, and then we go now in our database and see again we have the object. At this point we don't care what the values are, we just have the read timestamp and the write timestamp. And assume that this database system started and there was some transaction that inserted both these values and it had timestamp zero. So now to do this read, we'll do our check to make sure that our current timestamp for T1 is one is greater than the read timestamp of this and the write timestamp. And we go ahead and update the read timestamp to say now a transaction at timestamp one was able to read this object. Now we switch it over to transaction T2, T2 starts, it wants to do a read on B. And again, same thing, the read timestamp is in the past. So we know that there isn't a transaction in the future that read us. So we're allowed to go ahead and then update the read timestamp for this object to now be two. Then we do a write on B and again same to check our write timestamp is zero. So we can overwrite that and the read timestamp is two, which is us, so that's fine. So we go ahead and allow to write this object and then we update the write timestamp to two. Now we switch over here and transaction T1 wants to read A. Again, all the same checks as before, so that's fine. Then we switch back here and do a read on A and then a write on A and that's fine. And then there's no violations for either of these two transactions under the timestamp ordering protocol. So both these transactions are allowed to commit. And this is equivalent to one where T1 ran first, did read on B, read on A, followed by transaction T2. That would have been the equivalent serial ordering for this particular schedule. All right, so there's another example here. So now we have transaction T1 wants to do a read on A and write on A and transaction T2 wants to do a write on A. Again, the same thing, we start with T1. It does a read on A, we update the read timestamp for the object. Now we do a write on A and transaction T2. So now we said it's write timestamp to two. So now when we flip back over to T1 and it wants to do the write, we have a violation here because now the timestamp of our transaction that's trying to update this object is less than the timestamp of the transaction that less wrote to that object and that would violate the basic timestamp ordering protocol. So we know that we're gonna have to abort transaction T1 because it's not allowed to overwrite that value that T2 wrote to. Because that would not be equivalent to a serial ordering. Is this clear? All right, so what's a really simple optimization we could do here? You're back. Exactly, yes. So he said that in the case here, if we just ignored the write on A by T1 and just let the write from T2 persist, that's fine because nobody would ever see that write from T1 anyway because it would be needly overwritten by T2. So therefore we'd be clippant to a serial order or we'd be serializable. Yes. So his statement is if there's another transaction that slips through. Yes, and it reads that write that it shouldn't. If it should have read the write by T1 and not the write by T2. Actually no, even then that wouldn't happen because T3 would be in the future because it would get time stamp three. So therefore we'd never be able to read the write from T1 anyway. Because serial ordering would specify that you have to read from T2's right, not T1's right. Okay, so what essentially he described is actually has a name and it's called Thomas Wright Rule. So some guy in the late 1970s, I don't know if he's still alive, basically came up with this basic idea of ignoring these writes that would never be read anyway. So now we wanna do a write. We wanna first check to see whether our time stamp is less than the current read time stamp of the object. If so, then we know someone read in the future and would miss our write, so we have to kill ourselves on the board. If our write time stamp of the object we wanna modify is greater than our time stamp, then under the Thomas Wright Rule, we can ignore the write, just not apply it at all, let the transaction sort of think it did the right, even though it actually didn't, and we can let it keep on running and it'll be able to commit without any problems. And this obviously violates the protocol, the basic time stamp ordering protocol guarantees, but in our particular example here, this is okay. So we go back to this example, 404, do the same thing, we do the read on the A and transaction T1, flip over to T2, now does the write on A, then we come to T1 and tries to do the write on A and then we just not even do the update at all, right? If we're maintaining a private buffer workspace for the transaction, we'll apply the write there, so we try to read the same thing, it'll get it, but nobody else will see this write. Everyone will see the write made by transaction T2, yes. So our question is, if T2 did a read instead of a write, would it have to abort? Yes, so you say T2 got the wrong value in this example? All right, so her statement is, if I change this write in transaction T2, if I change that to a read, right, the way the protocol works is I do my read, I check to see the read time stamp, it's one at this point, all right, at this point in the schedule, it would be one here, so that's fine, I'll update the time stamp now to two because I read it and then I can commit, that's just fine. Transaction T1, now we'll try to do a write and it's time stamp is one, but the read time stamp is greater than him, so therefore it'd be trying to write to an object that a transaction in the future already read and saw the value before T1, so T1 has to abort because you can't go back and tell T2, go back and rerun this, like you read the wrong thing, right, because at this point when I commit, I'm done, assuming I can send back acknowledgement to the outside world, oh yeah, you read A, here it is, you can't go and retract that, so T1 will abort and it'll be just as if T1 never existed at all, okay. So the basic time stamp ordering protocol will guarantee that you can generate a schedule that is conflict-serilizable as long as you don't use the Thomas Wright rule. There are some corner cases where it'll get violated, I'm not saying Thomas Wright rule always achieves view-serilizable schedules, I have to check this, but I don't think it does, I think there's some corner cases that it can't handle. Again, the major difference between this protocol versus the two-phase locking protocols, at least in case when you're using deadlock detection, there can never be any deadlock because no transaction ever waits, right, it's essentially like the weight and die or wounding weight where you set the time out to zero, right, you check to see whether you can do something, you can't, oh, I'll just go ahead and abort myself, right. Now there's a problem where the, you could have starvation for transactions that try to reach a bunch of stuff and then other transactions keep modifying the database and they have time stamps that are in the future, so the longer transaction by the time it actually gets to do the objects, other objects once it's modified, some other transaction already sneaked in and already modified these things, so you'd have to keep aborting and restarting over and over again. Another issue that's important to understand for perpetual theory is that time stamp ordering the protocol that I've described here permits schedules that are considered non-recoverable. Now we'll cover this more when we talk about logging recovery in a few weeks, but the basic idea how to understand recoverability is that if you have a schedule where if you abort one transaction and then another transaction read that an aborted transaction changes, but that other transaction is already committed, then if you crash, you come back and try to replay what happened in the log, you won't be able to actually restore the database to the correct state because the transactions that got aborted will not actually not get replayed correctly. So we say a schedule is recoverable. If transactions only commit after all the changes that they read from other transactions were committed. So I think he in the back asked this question, I think in the last class or two classes ago, and the way this basically works is that internally the database system knows that I had the read set and the write set of my transaction and I know my transaction read an object that was modified by this other transaction over here, but this other transaction hasn't committed yet, so even though my transaction said I want to commit, you don't actually do the commit and announce to the outside world your transaction has finished until the transaction that it depends on goes ahead and commits, right? And we do this to guarantee that we're recoverable. So to use a quick example, say I have transaction T1, it does a write on A, transaction T2, does a read on A, then write on B, right? If T2 is allowed to read the changes from T1, then if T1 later aborts, this would, the abort would occur after T2 has committed, so T1 should be as if it never actually existed, but T2 was allowed to read those changes, right? So if at this point here when T2 commits, if we tell the outside world, yeah, we got what you wanted and here's the result, then this schedule is not recoverable because T1 shouldn't actually exist, right? Another way to think about this, we're violating the sterilizable guarantee that we would have no dirty reads, right? Because we're reading uncommitted data. So under basic time-saving ordering protocol, you can have non-recoverable schedules, right? Because all the decisions about whether transaction should abort or not occurs by the transaction that it would violate the protocol, right? Because there's no locks to say, oh, I need to wait until I can actually do something. I was allowed to go ahead and do it, and if some other transaction comes along and says, well, I can't write to this now because somebody else has already read to it, then I have to kill myself, but other transaction was allowed to run unscathed. All right? All right, so the performance issues of basic time-saving ordering, I would say also too that to the best of my knowledge, no data management system actually implements this as I've defined here. But again, this is just sort of as a setup for you to understand the more sophisticated protocols. So there is high overhead in implementing this because every single time you read or write to an object, you have to copy it into your workspace. The multi-versioning schemes that we'll talk about later, avoid this by being more smart about what they're actually copying when you actually read or write things. We also talk about how transactions can get starved if they're long running because the likelihood that some other transaction is gonna come along and modify something and they have a future time stamp then you would come along to try to read that thing, increases the long you stick around and do more reading right to the database. Now, this isn't so much an issue for disk-based citizens that we're talking about here, but it can be certainly the case in a sort of an in-memory data system that's the high performance can run on multiple cores and multiple CPUs. We actually done some research which shows that the allocating the time stamp can actually become a big bottleneck due to cache coherence traffic between the different CPUs. All right, good thing about it. If you have a single logical counter that you're doing atomic add over and over again, the atomic add is a single instruction that's fast but then invalidating the cache lines for any CPU core that has a copy of that counter can be really expensive. Okay, so one observation we can make is that if you assume that conflicts are rare between transactions, then forcing all the transactions to have to acquire locks under two-phase locking before you're allowed to do anything or always updating the read and write time stamps every single time you do anything, this can be sort of unnecessary work because the likelihood that transactions will conflict could be rare. Then if you also assume that transactions are short-lived, then again, it's sort of related to that, conflicts will be rare because most of the times transactions will finish in microseconds. So an argument can be made when designing a current control protocol is that rather than taking the two-phase locking approach where you have this pessimistic view that you assume transactions are going to conflict, so you make them acquire locks before they're allowed to do anything, even if there never will be a conflict, a better approach could possibly be to say I'm going to optimize for the most common case where I assume transactions are not going to have conflicts at all and I'll build a protocol that can be more efficient for that. So that's the motivation or the basic premise behind optimistic current control or in abbreviated form OCC. So this is actually a protocol that was actually developed here at Carnegie Mellon in the early 1980s, I think in 1981 is when this paper came out. And this was invented by H.T. Kung, who was not a database professor at all at CMU. He was actually a networking professor and somehow they dabbled in databases and it wrote one of the most well-cited papers in database literature of all time. He left CMU, I think, in the early 1990s to go to Harvard. It was described to me, they poached him out of CMU to go help build Harvard's CS department or CS program. Famously, Harvard also had another very famous database person, Phil Bernstein. A lot of this core fundamental theory and principles behind current control was actually developed by Phil Bernstein in the late 1970s, early 1980s. He famously got denied tenure at Harvard in like 83, 84. So Harvard CS wasn't that great and then H.T. Kung was hired to come and sort of rebuild it in the 90s. And this is a really great paper. It's really simplistic, but it lays out the basic ideas of this protocol very clearly. And as we'll see next class, a lot of the newer systems actually use this same protocol that we'll talk about here, but in the context of a multi-version system. All right, so the way OCC is gonna work is that the database system's gonna maintain a private workspace for each transaction. We saw this before when I talked about, in achieving, you know, repeatable reads on our basic times of ordering, I said there was a private workspace for the transaction, right? But if you want to run an over-isolation level that you don't need this, on our OCC you're always gonna have this private workspace. So anytime you read an object or write an object, you always make a copy of that object from the global database, meaning the database that everyone can see, and you copy it into your private workspace. And no other transaction can see the things that you write in your private workspace until they're actually applied to the global database. So then what's gonna happen now is when a transaction commits, the database system's gonna check to see whether the object you wrote to in your workspace conflict with all the transactions that are running at the same time, and if there's no conflicts, then you're allowed to apply those changes to the global database, and then all your changes become visible to any other transaction that comes after you. The other thing I'll add to this paper too is maybe hard to see in the back, is that maybe this is how they did it in the 80s, but they have Carnegie Mellon, they have a hyphen between Carnegie and Mellon, whereas now we don't do that at all. I always find that interesting every time I see that paper, sort of listed up there. So OCC is defined in terms of phases. And again, this part is always confusing for students because the name of the phase is kind of not intuitive. And I don't know whether this is because, again, it's coming from someone with a networking background. This is not how I would call things, but again, if you read the paper, this is how they describe it. So in the first phase, called the read phase, is where you read and write, not just reads, but write also, objects from the database into your private workspace. So again, any time you read a tuple, you make a copy of it, put it into your private workspace. Any time you write to a tuple, you make a copy of it first, put it into your private workspace. Any inserts you do, always go into your private workspace. Any deletes, you just mark the tuple as you want to delete them in your private workspace. So then now the transaction tells the data system, I want to commit.  And then you enter automatically into the validation phase. And here is where the data system's gonna check to see whether the changes you have in your private workspace conflict with other transactions that are running at the same time. And you can do this by poking or peeking into their private workspace and see what they've done and assess whether it's safe for you to commit. And if everything checks out, then you enter the write phase, where now you apply all the changes from your private workspace into the global database.  So another key difference is that the unlike basic timestamp ordering where transactions got the timestamp assigned to them immediately when they showed up in the system. Under OCC, you don't actually get a timestamp until the validation phase, right? So when you start, you don't have a timestamp, it's only when you say I want to go to commit, then the data system applies as your timestamp to you. All right, so let's walk through a quick example. So here now I'm annotating the schedule with the sort of the segments of the correspond to the phases, right? So again, just like under two phase locking, where you don't actually say lock this object, in different modes, although you provide hints in SQL, you're not explicitly gonna call, I'm in the read phase, I'm in the validate phase, right? This is something that the data system does for you automatically. So I'm showing these labels here just to make it clear what phase we're in at different parts of the schedule. So transaction T1 wants to do a read on A, write on A, and transaction T2 just wants to do a read on A. So transaction T1 starts, all right, the other thing too I'll also point out in the database, we no longer have to keep track of the read time stamps, we only need to know the write time stamps for our objects. So, and then we have all the labels here for the different phases. So transaction T1 starts and enters the read phase. And at this point here, the data system will allocate its private workspace, right? It's just some region of memory that it can read and write things to. Then it does a read on A, and what happens here is then we'll copy the object that they want to read from the global database into our private workspace. And this includes the write time stamp that we got when it came out. So now transaction T2 starts, same thing, it enters the read phase. So we create our private workspace, then it does a read on A as well, and we do the same thing, we copy down the same version of the, I'm gonna use that term, but the same version of the tuple that it existed, that exists in the global database. So now when transaction T2 says they wanna commit, so this part is always sort of confusing too, I'm having the commit be at the bottom, but essentially after that read on A, transaction T2 told the data system I wanna commit. So this, the commit I'm listing here is like the physical commit, when the data system actually applies the changes and the transaction is fully finished, truly finished. But when you enter the validate phase, that occurs again when the transaction tells the data system I wanna commit. So now at this point in the validate phase, we'll assign the transaction a timestamp. So now we're gonna say here, we'll just assume a simple logical counter, we'll always add in one to it. So transaction T2 is gonna get timestamp one. Even though T1 started before T2, T2 is entering the validation phase before T1, so it's gonna get timestamp one. In this case here, I'll describe the validation phase in a second, but it's sort of obvious here that both T1 and T2 have only read A, so there can't be any conflicts. So it'll complete the validation and go into the right phase and it can apply any changes that it made, which are none from its workspace to the global database. And then at point it actually truly commits, we tell the outside world you're done and we can blow away the T2's workspace. So then now we come back over to T1, it does the write on A. So again, we'll apply our change to the private workspace, but now we'll set its right timestamp to infinity because at this point, we're still in the read phase for T1, so it doesn't have a timestamp, so we just set it some non-obtainable value. So then it goes commits and we enter the validation phase, now we get our timestamp, so now transaction T1 is now assigned timestamp two. Now we look around and see are there any other transactions running at the same time as us in our validation phase that could conflict with us, in this case, no. So now when it goes and just the write phase, it's now to allow applied change up into the global database and here we set our write timestamp to be the timestamp that was allocated to us, which is two. Yes. So when you check whether there is some conflict before you meet your back to parent with the database yourself, what would that mean? Few more slides and you'll see this. I'm being very hand wavy about this, but I'll describe exactly what the validation protocol is in a second. So his question is, is there a guarantee that nothing will happen between the validation phase and the write phase of a transaction? As I'm going to describe it here, we'll be doing our validation in serial ordering. There is some, you could have, well, we'll do validation in serial ordering because that's the easiest way to understand this, but certainly in the write phase, you could have two transactions need to apply a bunch of changes to the database from the private workspace. At that point, we know their timestamp ordering, so we can make sure that the latest timestamp always gets written to the database first. For this, we're worried about correctness, so it is assumed there's a global mutex for applying these changes. But if you want to do parallel fast OCC validation, it's a bit more tricky.  In the back. Validation phase. Your question is, is validation only done by a transaction when it's in the validation phase? I think your question is, if one transaction is in the validation phase, do you do it, do you compare some of the other transactions that are also in the same validation phase? So, next slide, validation, well, let me walk through this and then if you're still confused we can go over it again. I'll also say too, I forgot to check on the, so another thing too is in the book, I think they call this validation protocols, so they have two phase locking protocols, timestamp ordering protocols as I described here, and then they also have a second sort of subchapter called validation protocols where they describe OCC. Technically that's not really true because it's a timestamp ordering protocol because we're using timestamps to figure out the ordering of things. So the big part of figuring out how OCC works is through the validation phase. I'll also say too that I'm gonna describe how to do validation in one direction. You actually can do it in both directions and for this it doesn't matter that you can do both directions. If you just understand the basic idea of what's going on, you can kinda see easily how it can be flipped in another direction and then we're also gonna do this in serial order which makes this way easier to understand. The parallel version were complicated. Okay, so in the validation phase is the key part of this is that we're gonna guarantee that we only have, we only generate schedules that interleave operations in serializable ordering. And so what we're gonna do is that we're gonna have our transaction that wants to commit. It's gonna check other transactions for read-write or write-write conflicts and then make sure that our conflicts only go in one direction. So that means that the order of transactions, we always wanna have order transactions conflict with, if there's a conflict between the order transaction and the newer transaction, we'll make a decision whether to abort a particular transaction based on the age of the time stamp that we're given to it. And we saw this before with deadlock prevention under wound to wait and wait and die. We have the protocol make decisions about whether one transaction is allowed to proceed or not, whether you should abort the other guy. And it always has to be in the same order, right? From oldest and youngest or youngest to oldest. Because if you mix them, then you can have, then you end up with incorrect orderings. And as I said before for all of this, that for simplicity, we're gonna assume that we're gonna do validation and the writing in serial ordering. And we'll just use a simple mutex or latch to protect this critical section of the system. During the read phase, you don't have to worry about this, right? Because you just write to your private workspace, right? And anything you read from the global database is you know this from a committed transaction. And you know you're doing this in proper serializable ordering. So you don't have to set in the latches or mutexes when you actually do any reads. All right, so the validation phase has three steps. And again, when a transaction enters a validation phase, we're always gonna assign it a time stamp. And the time stamp is always gonna be unique and it's always increasing. And then what we're gonna do is we're gonna check the order of our committing transactions with all other running transactions. So we're gonna look in their private workspace and see what's going on inside of them and see whether we're gonna conflict with them.  So this is what I was saying, you can do sort of forward validation or backwards validation. So what I mean by that is this is backwards validation. Sorry, this is forward validation because you're looking at transactions forward in the future to see whether if I commit, I'm gonna, if I commit, I would call this a conflict and therefore I should abort myself. In backwards validation, you check to see whether you've made changes that would conflict with transactions that I've already committed in the past. The basic idea is the same. For this one, we're going, again, we're always gonna go in one direction. So what we're gonna do is we're gonna check to see if our time stamp is less than any other time stamp that we can reading from from an object, then we have to check to see whether these conditions hold. Now remember I said before we set the time stamp of the object we wrote to in our work space to infinity. So that's why these time stamps are always gonna be larger than whatever the time stamp we are trying to commit with. And if that's true, then we know that we're actually trying to write to the same object that somebody read to and therefore we shouldn't be allowed to commit. So in the first phase, it's pretty simple. If the transaction that we're trying to commit finishes all three phases, read, validation, and write, before the other transaction we're preparing against even actually does anything, then we know that the validation will succeed. It's essentially just, do you have a serial ordering? So T1 starts as read, write, and validate, and then we know about T2, but T2 actually hasn't read anything yet. So technically it hasn't started the read phase, so they can never possibly conflict because this is sort of a sort of trivial serial ordering, so this is fine, right? We know that we're not gonna conflict with our transaction, so we can go ahead and commit T1. The next one is that if T1 completes before TJ starts his write phase and T, TI, excuse me, completes before TJ starts his write phase, and TI did not read anything, sorry, did not write anything that was read by TJ, then we know that there cannot be a rewrite conflict, so our TI is allowed to commit. And more importantly, you can write this, if the write set of TI, right, you look in the private workspace and say here's all the objects that I wrote to, if that write set, when intersected with the read set of another transaction, if that's null, meaning there's no overlap at all, then there can't possibly be a conflict because TI would have written something that TJ didn't ever read, so we're fine, right? So more simplistic like this, like, in this case here, we enter the validation phase for TI, sorry, T1, T1 read on A and wrote to A, and now we have in our workspace, we have the object A with our write timestamp set to affinity, so now in our validation, we would check, for T1 would go check inside of T2, and inside of T2 you would see, oh, why read to object A? So the read set of T1, sorry, the read set of T2 intersects with the write set of T1, so therefore T1 has to abort, right? So even though T2 will never actually write to anything, and this still would be a conflict here, because when T2 read A, it read the version that existed in the global database before T1 started executing, so therefore it's reading an object in the past, but then T1 wants to overwrite that with a newer version, but T2 won't see that because it wasn't applied to the global database yet, so this will violate our guarantee, right? So now do another example here, or is this the same one, sorry? Right, in this case here, sorry. This is exact opposite of what I just showed, sorry. Oh, oh, sorry, sorry, yeah, sorry. T1 does the read on A, write on A, and then T2 does the read on A, but then it does validation, and then that point it gets a timestamp of say two, or sorry, one, T1 then does its validation after T2 does its validation, and in this case here, it's gonna get timestamp two, so that's okay because T2 read the object as it existed at timestamp zero, and it has timestamp one, so it can't see anything that has a timestamp greater than one, which would be two, which is for T1, so therefore this is safe to commit because the validation of T1 occurred after the validation of T2, and therefore this is a correct serialized ordering, even though T1 started again before T2. Okay, so that's the second, the third step is that if TI completes its read phase for TJ, completes its read phase, and TI does not write to any object that was either read or written to by TJ, then we can go ahead and commit. This is essentially an extension of the previous one where now we're including the write set, and the idea is basically the same here, right? So if T1 reads an A, writes an A, T2 reads on B, and then reads A, if we get to here and we validate, then at this point, we don't see in the private workspace of T2 the object A because it hasn't been read yet, so this is fine, and T1's allowed to commit, right? And then when T2 does its validation, it's okay that it read the older version of B because T1 didn't write to it, but then they read the new version A. Yeah, sorry, sorry. At this point, the write phase of T1 gets installed in the database, right? So T1 does the write into A, and then updates the object in the database, now T2 does the read on A, and it'll get the version that T1 wrote into, right? And therefore that's okay, that's serializable, and it's validation, T1's not running anymore, so there's nothing to check, and let's go ahead and say for a lot of commit this. So even though again, the write on A occurred while T2 was already running, but by the time it got to the validation, the change got applied to the global database, and no one else has tried to read that object yet, so that's okay, so now when T2 read that object, it gets the correct version, and we guarantee that everything's serializable. All right, so an obvious question is, this is more sophisticated than the basic time sample ordering, when does OCC work well? When do we actually want to use this? And as I said in the beginning, this works well when the number conflicts are low, right? Because if you have a lot of conflicts with transactions, when you actually do the validation, you keep avoiding the transaction over and over again, and you would never actually, you barely ever complete anything, right? So this works well when the transactions are updating distinct things, and then go ahead and do validation, commit without any problems. This question is, if your transaction is read-only, do you still have to check the other transactions? In the example I showed here, if your transaction is read-only, you don't have to check anybody else, because all other transactions will have to check to see whether they've conflicted with you. Because in this case here, you've read a consistent snapshot of the database, because you're making sure you only read objects that are actually committed. You can still have phantoms, which I'm ignoring for this for now. The idea is that it's up to the other transactions that do the writes to see whether they would cause a problem with you, and then they abort themselves. If you're read-only, things are just fine. Again, so the main motivation for this was setting locks for things that never gonna have conflicts anyway is an expensive operation, because again, you're on the lock manager, you have to set a latch on that, then acquire the lock that you want, do whatever your transaction needs to do, and then go back and release it, which again, requires more latches. If you don't think there's gonna be conflicts, then you wouldn't wanna use OCC, because the only time we actually need to get a latch for anything is when we actually apply the changes to the global database, because I can copy things in my private workspace without worrying about any latches, because the transaction has exclusive access to it. So the problem with OCC though is that there's some overhead involved with copying data locally. Validation phase and the right phase can become a bottleneck unless you parallelize it, but even then, there's always gonna be some contention point where you have to make sure that one transaction is only allowed to apply the changes to the particular objects in the database. The other big issue is that in the case of OCC, you're making the decision about whether a transaction has to abort at the end of it, at the validation phase. Under two-phase locking, if I need to acquire a million locks, and I can't get the first one, then depending on what protocol I'm using, the data systems could decide right then and there, I'm not gonna, you can't acquire this lock, I'm gonna abort you and restart you. Under OCC, if I need to acquire updated million things, I'll update all one million things, and then I go check my validation phase. That's when I figure out, oh, I have a conflict, I need to abort myself. So I've just done a bunch of wasted work. So OCC does aborting at the end, in other words, in two-phase locking, you can abort at any moment in it. And the sooner the better, because then you're not wasting cycles doing useless work. And the same thing as we saw in basic timestamp ordering protocol in a highly concurrent multi-socket CPU environment, allocating these unique timestamps for transactions can become a bottleneck. Okay, so any questions about OCC? Yes, your question is, we're gonna assume there's nothing can occur in the database between the validation phase and the write phase. Yes, in this particular example, yes. Assume there's a serial order, there's a single latch you acquire to say, I'm now in the validation phase. Do your validation, if it passes, then you do all your updates, then you release that. If you want to do it in parallel, you have to, it's a bit more tricky. You can do things like, you can order the, you can sort the order of acquiring latches on objects that you want to write in sort of some electro-graphic ordering so that multiple threads running at the same time are always doing their validation and application of the changes in the same direction. That's a bit more you want to cover in this class, though. Yes. Why are a lot of wonders on the system, prematurely validate myself to see whether I should afford it? So his statement is, why can't you just do like a speculative validation? You do that, sure. But then it's like, then you have to, so you can do it specatively. And then, yeah, you can decide, oh, I think I'm gonna abort, so don't even bother acquiring the latch, then do their true validation. You could do that, but again, you're burning cycles. And if you assume that your conflicts are rare, which is what you want to assume if you're running an OCC, then you're doing, you're doubling the amount of work you have to do for validation. So there's trade-offs, right? There's no free lunch for this. So the overarching theme, that I'm glossing over of the way you implement these things, is there's all these internal data structures we have to maintain to figure out whether the transactions are doing the right thing. And this is not just true for time-safe ordering, it's also true for two-phase locking. Two-phase locking, you have a latch on the lock table, and internal things, the internal cues, you have latches on those. And time-safe ordering, you have to have a latch when you enter the validation phase or the right phase, right? And we have to do this because we have to check to see whether we have a conflict between another transaction that's running at the same time and avoid race conditions on our internal data structures. So if we have a lot of concurrent transactions, even if they don't conflict, then doing all of this extra work is expensive. And then we already talked about crabbing, latch crabbing in indexes last week. If you have all these transactions that have to update all these indexes, you have to take all the latches that when they go down don't apply your changes. So you're applying, you're applying latches for internal data structures as well as the indexes and whatever else is going on in the system. So this is really expensive. So an alternative is to split the database up in such a way where you'll allow transactions to run where you know that they're not gonna conflict with other transactions and therefore you don't have to check a bunch of transactions. You only have to check the ones that are sort of within your domain or region. And I think we already talked about partitioning before. The basic idea is we're gonna take our database or tables and we're gonna split it up into these disjoint subsets called partitions. And then now when we have transactions running at a partition, we only have to check our transactions running at that partition. We don't have to check all the other partitions. Under OCC, as I described, I have to check every transaction's workspace that are going on inside the system. But if I can limit the scope of what I have to check, then I have to check fewer things and I can run much faster. So in a lot of OTP applications, this is actually very easy to do because in a lot of OTP applications, the database is easily partitionable. So to give you an example, so this is the scheme of from the TPCC benchmark that some of you will be running for the extra credit. It's an old benchmark, it's from 1992, but in talking with people in industry, a lot of times a lot of applications actually look like this. So the application that's modeling is a sort of order processing system, think of like an Amazon store. So you have a warehouse, a warehouse has district, a district has customers and customers have orders and orders have order items. And so just to take these two tables, the warehouse and district table, there's sort of this natural dependency between these different tables. So in the warehouse, the warehouse has a warehouse ID and then a district, since it belongs to that warehouse, it has to have a warehouse ID as well. So it's gonna have a foreign key reference to the warehouse table. And this will occur all the way down from the customer because the customer has to belong to a district and therefore you have a warehouse ID for the customer because you have to know what district it's in and the same for orders and order items. Stock has a dependency on the warehouse table. And so what you can do is you can generate partitions where you basically take the warehouse ID of every single tuple and you use that to assign it to a particular partition. So for warehouse ID one, all the districts records, the customer records, the orders record and the order item records with that same warehouse ID goes one, we'll get put into one partition. The same thing for two, three and so on. The only sort of anomaly to this is the item table here because items are sort of global and they're not assigned to a particular warehouse. So what we're gonna end up doing is replicating that on every single partition, which I'll show in the next slide. So this is also very common in OTP applications. There's often these sort of small read-only or read-mostly tables that don't fit nicely into our partitioning scheme, but we wanna use in all different scenarios for all the transactions. So zip code is another great example. Zip code only has, I think 30,000, in the U.S. there's only 35,000 zip codes. The post office only updates them four times a year, but you need to use the zip code table in all sorts of transactions to update addresses and things like that. So in that case, you would replicate the zip code table everywhere. So let's see how we wanna assign this now, these are data to partitions. So for this, we'll assume we have four partitions and again, this is gonna be, we're basically doing horizontal partitioning, we're gonna slice up rows of tuples and assign them to different partitions so that all the tuples for a particular warehouse will appear in one partition and all the tuples for another warehouse will appear in another partition. So say that we split this up between four partitions, partition one goes here, partition two goes there and so forth. And then we're gonna replicate the item table at every single partition. So now what happens if I have a transaction that wants to update, place an order for a customer, you would look up and say, well, what warehouse does that customer in and then you could direct your transaction to just that one partition, do all your changes there and you don't have to check any of the other partitions because you know they can't conflict because they're not updating anything in your warehouse. So this is the basic idea of partition-based time sample ordering. And the particular variant I'm gonna show them, show you is specific to these couple systems here. So you could take OCC for example and run it in a partition-based cluster or partition-based deployment where you know if you assume transactions are not gonna touch each other or touch data from different partitions, when you do your validation, you don't have to check them. But what I'm describing here now is actually a very specific protocol that rigidly enforces these partition boundaries and will have a single-threaded execution engine assigned to each partition that can only execute one transaction at a time. And so now you don't need to do any validation, you don't need to do any private workspace, you don't need to do any crabbing or latching on any data structure because that thread is the only thing that's running for that particular partition. No other transaction is running at the same time, so you don't need any of all the protection mechanisms that we've been talking about so far. So another way to think about this is that the partition is protected by a coarse-grain single lock. And what happens is that lock is assigned to transactions in time sample ordering. So if you're the oldest transaction with the oldest time stamp in the system, then you're granted that lock. And then you're allowed to run to completion without worrying about any other transaction running at the same time, right? So this idea is pretty old, it's actually a paper from 1992 or so that describes that you could do this. In the 90s, there was a system called SmallBase which eventually became TimeStand out of Oracle, although they don't do this anymore, but the original version they did. We had the KDB guy at CMU two weeks ago, their system in the 90s, actually, and I think even now still does this, you have the single threaded engine. I was in grad school when I helped build a system called H-Door and H-Door sort of takes this to the extreme and does it on a distributed environment. So you have multiple partitions per machine and you can have transactions at each machine run in isolation of everyone else. And then H-Door was then forked and commercialized as VoltDB. So VoltDB is the commercial implementation of H-Door. And then earlier this year, there's a system called FaunaDB that came out. They don't do exactly what I'm describing here. They do sort of, he was proposing before where you do speculative execute transactions, figure out what they're gonna do ahead of time and then you assign a partition, you assign a lock to a partition based on a batch and you allow them to run completion without worrying about interleaving. But at a high level it's essentially doing the same thing. So to do reads, transaction, you can read anything you want at the partition as long as you have the lock for it. So if I only need to touch data at warehouse one, and as soon as I get the lock for that partition for warehouse one, I can read anything I want and I don't have to check anything. If I try to then access data at another partition, like I need to read something at warehouse two, then I have to abort my transaction, restart it and then when the transaction comes around the second time, you have to tell the data system, I need locks at partition one and partition two. You have to get all the locks you need before the transaction is allowed to run. And this is an okay assumption to make in a lot of old TV applications, because a lot of times that most transactions are only gonna need to touch data at a single partition if you partition it correctly. For writes, the same thing. We can apply our changes in place. We just overwrite whatever the current version of the tuple it is, because we know nobody else is running at the same time, nobody else can read our writes, so it doesn't matter that we apply our changes and now they're globally visible, because they're not. We still have to maintain an undo buffer in case we abort our transaction, because you need to be able to roll back those changes and put it back to the correct state. But again, we don't have to do any validation or make sure that there's any conflicts because we know only one transaction is running at the same time. And the same thing as before in the reads, if you try to write to a partition you don't have a lock for, then you have to abort, come back and acquire those locks before you're allowed to run. So this particular protocol, this partition-based time sample ordering is probably the fastest way you can execute transactions in a database system. Because there are no, there's no overhead of checking latches, checking for conflicts, making copies in your private workspace. You're sort of almost going at bare metal speed. It's essentially almost the same thing as just having a single thread for the entire database and having that do all the reads and writes. And obviously that would not be able to parallelize and take advantage of new cores and multiple machines. So this is sort of the next best thing. The other thing I didn't really talk about here is that this only really works if you execute every transaction as stored procedures. Because if a transaction acquires the lock for the partition but then you gotta go back and forth over the network through JDBC, say, what's the next query I wanna run? What's the next query I wanna run? Then that's gonna delay the amount of time that the partition is locked and it's gonna actually run worse than you would if you actually interleaved your operations. So this only really works if you have stored procedures. And it also only really works well if most of your transactions, if not all of them, are only associated at a single partition. Because if you have data at multiple partitions, what will happen is the same thing as the network problem, you acquire the lock for the other partition, you do most of your changes to your one partition and then the other partition is idle waiting for the next query to run. Now you can get around this with spec of execution and other things, but that's, again, beyond the scope of what we wanna talk about here. So a system like VoltDB works really, really well and probably is the fastest system you possibly have and I realize it's based on HR, sounds like I'm biased, but the studies show it is actually the fastest system you can have if all your transactions are single partitioned. I don't know what it is for and their numbers, but if it gets around 10% of your transactions that touch multiple partitions, then you're better off using a OCC or time stamp ordering protocol, sorry, OCC or two phase locking protocol that we've talked about so far. This is sort of a specialized case but if most of your transactions can be single partitioned, this is the best way to go. Otherwise, you use these other protocols. So any questions about this? I know that there's one group for the extra credit that has signed up to do the Ron experiment for extra credit for VoltDB and be interested to see how they compare against some of the other systems. Okay, so just to reiterate, every concurrential protocol that's out there can be essentially broken down into sort of fundamental operations into either two phase locking or the time stamp ordering protocol that I've shown here in the last two lectures. There really isn't another way to schedule or interleave operations. It's either pessimistic where you prevent them from doing it before they're allowed to do it or optimistic where you check after they've done it or that they were allowed to do it. And you use locks for the first one and time stamps for the second one. Yes? Given the trade-offs with all these concurrency control protocols based on different workloads, if you have a database that has varying workloads, can you mix and match protocols or do you have to decide on priority, I'm gonna use locking or I'm gonna use time stamps? So his question is, can you mix these protocols inside of the same system for transactions that, keep it simple, say you have a single database but I have sort of two applications accessing the same database at the same time. Can I have some of the application transactions run at two phase locking? Some of them run on our time stamp ordering. As far as I know, no commercial data system does that. And part of the reason is that it's actually really hard to implement these things. So you'd have to implement two phase locking correctly. Then you got to also implement OCC or time stamp ordering correctly, right? And so from an engineering standpoint, that's really hard to do. So nobody does that. From a correctness standpoint, yes, but in very limited cases. So MySQL sort of does this. MySQL does two phase locking but what they do is you get a time stamp when the transaction starts and you're allowed to read older versions without having to acquire the lock, right? And they're doing that through time stamps. There are some research systems that can figure out things like, all right, well, I'll observe the application and I'll run it under OCC when there's no conflicts but if my conflict rate goes up, then I'll flip a switch and internally I switch over to two phase locking. I don't know whether they can intermix things. I would say in practice it's hard. It's only for special cases, I think you can do this. It's a good question though. So another thing I'm not showing in any of these lectures is that I'm not showing any benchmark results and I'm doing this because at this point, I care about correctness. I've mentioned a little bit about where the performance bottlenecks are but if I showed you any graph and say like, here's TPCC running on these different protocols, right?  We don't want to do that. We want to use OCC, right? Because I can show you benchmark numbers where OCC actually does better than two phase locking and I can show you other benchmark numbers where two phase locking does better. So in the advanced class, we'll talk a little bit about understanding at a more fine-grained level what could be the bottlenecks or limitations of these different protocols but in practice, there's not one that's better than another, right? For all possible applications. The other thing I'll say too is that at its lowest level, if you have extreme conflict, meaning every single transaction that executes wants to write to the same object, then these two phase locking and timestamp ordering basically digress to be exactly the same thing, right? You end up just executing transactions in serial order. So, again, if your application has a low conflict rate, OCC might be better. Or if your transactions want to run for longer then two phase locking might be better. There's no one thing I can point to you say, if you don't remember, if you forget everything that's class, just remember this, two phase locking's the best, right? There's nothing I can say that would, there's nothing I can show you that could justify such a statement. Different database systems do different things. In addition to just sort of the underlying protocol, whether it's two phase locking or timestamp ordering, there's a bunch of really low level details of how you actually implement these things that can make a big difference, right? And we'll see this when we talk about MVCC in next class, but it's the same high level protocol, but whether you use a hash table or tree structure for your locks, right? All these things can affect performance. All right, so with that, again, so on Wednesday, we'll start talking about multi-version concurrency control. And again, there's another aspect of database systems and the literature where this is confusing. So there is no protocol called multi-version concurrency control in the sense of it's not separate from timestamp ordering or two phase locking. So you can do a multi-version concurrency protocol that uses a timestamp ordering protocol or that uses a two phase locking protocol. So multi-versioning is also another way to sort of architect the system of how you retain information about what's going on inside of the system, but at the high level, again, it's still just either two phase locking or timestamp ordering. So that's what we'll discuss on next class. And as a preview, I'll say that this is essentially what I'll describe next class is pretty much what every single database system that was written in the last 10 years actually uses. The only exception to this would be faunaDB and voltDB. Everything else that you can possibly name is going to be using a variant of MPCC. And we can discuss why that's actually the case or not. Okay? Yes. So his statement is, and again, I would agree with this, a concurrency control protocol has to be either two phase locking or timestamp ordering. It has to be either pessimistic, you think transactions are a conflict, so you prevent them from doing the thing that would cause conflict before they're allowed to do it.  So it's either one or the other. And how you actually implement them can vary a lot. You can use a queue for your locks, you can use a tree, you can do validations with a graph. So it's saying what would be representative, what would be a locking scheme that's not two phase locking? It has to be two phase locking. If you want to guarantee conflict serializable schedules, it has to be two phase locking. The Jim Gray paper from 1976 proves, and some of Phil Bernstein's later work proves that it has to be two phase locking. Because otherwise you hit the issues that we showed before. And it's intuitive, right? Like if I release locks, and try to come back and try to acquire more locks, someone else could have updated the database before in between them, and would violate serializable ordering. Okay, any other questions? Thanks guys, and then we'll check about the Project 3 for the weight die that's missing after the class. All right, thanks, see ya.


Live from Las Vegas, it's theCUBE. Cover EMC World 2016, brought to you by EMC. Now, here are your hosts, John Furrier and Dave Vellante. Okay, welcome back everyone. We are live in Las Vegas for EMC World 2016. This is SiliconANGLE Media's theCUBE, our flagship program, we go out to the events, extract the signal from the noise. I'm John Furrier, my co-host Dave Vellante. Our next guest is Michelle Allen Approach, who is the Senior EVP and CEO of North American Operators at those SC, welcome to theCUBE. Thank you very much. So you have been working with EMC now for about four years. What is this relationship like? Talk about the philosophy around what you guys are doing together and as a customer, what's going on? Right, right. So in year 2012, we founded this Enhanced Alliance with EMC Federation and it was really based on a joint view that our customer wants now to consume technology as a service, so first point. Second point, they want to have a full OPEX model and third point, they expect a full end-to-end and turnkey solution. So three major things really. And we were sharing these three views and that's why we created in year 2012 this Enhanced Alliance. We've seen that we have invested into a broad range of technology with our friends at EMC. To tell you the truth, in year 2012, we really began by deploying very massively VBlock everywhere, okay, that was quite traditional. Then we moved on to Enterprise Private Cloud, then after to the EHC, the Enterprise Hybrid Cloud and now we're working with EMC and VMware on the software design data center which we are deploying to our customer and particularly to our first largest customer, Siemens. I know Siemens was testifying this morning about smart storage and behind this smart storage is something we've built together through this Enhanced Alliance. So I wonder if we can pick up on that. So your philosophy is as a service, it's got to be an OPEX model and it's got to have end-to-end capabilities. Exactly, so that's really the three things we were sharing with the Federation which are the basis of this alliance. Which on the two of the three are sort of new for EMC, right? They're typically not as a service, typically not an OPEX model, but of course they have end-to-end capabilities. So talk about the value that ATOS brings to that relationship and what's the value that EMC brings. Yeah, so basically EMC is bringing first and foremost innovation. Innovation to the customer, faster and more innovation. We are bringing end-to-end commitment and we are bringing a single accountability to the customer and that's very important because we are integrating EMC and VMware building blocks into our solution under one single hat facing the customer. And that's the most important value. The last point which really has developed in the last, maybe 18 months is that on top of everything I have said, now customers are beyond anything what they want is business outcome. They don't want you to sell a new technology even if it's a cutting edge strategy. To tell you the truth, they don't really care about the technology. They care about what the technology is going to make them unable to do. So is it speed? Is it agility? Is it cost takeout? But that's a business outcome I'm talking about. So we hear a lot about digital transformation. You hear that everywhere. So how does that theme of digital transformation, what's the outcome that customers are looking for in that context? Well, so at ATOS we have named our digital transformation strategy Digital Edge. So that stands for engineer, design, gather and ends, that's the edge. We've built this digital strategy with the capabilities of our partners. As I said, the PC, the DDC, and another partner service now in order to fully give an orchestrated and automated service to our customer. Not only at the level of the infrastructure, but at the level of the infrastructure, the application, and the security gathering, all of that. So you're set service now? Yeah, I did. Okay, so they're embedded in that. Exactly. In this Digital Edge offer we have, it's a full holistic offer which is orchestrated by ServiceNow and in which we have the different building blocks. One is obviously the legacy infrastructure because our clients are complex and still have legacy infrastructure. EPC with EMC and the software design data center. Then on top of that, we're linking it with Public Cloud, mostly with AWS and Azure. So everything as a service, that's kind of the service now. Well, everything being into one hat. And you were asking about what's the value at the end to the customer. I mean, you really have now two type of companies. You have the brand new companies like the Booking and RBNB. The infrastructure and the application landscape is actually quite simple because it's brand new. So they have all the bright and shiny thing that have been issued in the last two years. Now, when you talk about Siemens, you talk General Electric, you talk whoever. They have a layer of infrastructure and application that they've been building for years. This is what our strategy is proposing, managing under one single ad, the old world and the new world, both at the level of the infrastructure and the application. Now, you made an acquisition, Xerox ITO, right? Can you talk about that? How does that fit into the strategy? Sure, so, I mean, before diving into it, ATOS today is making, it will take 13 billion, worldwide, 100,000 people, out of which, after Xerox ITO acquisition, here in the US, we're making 2.5 billion, 13,000 people. So really what it brought us here in the US was a jump from a situation in which we were subcritical in terms of size, footprint, capabilities to the first ATOS operation in the world, largest, larger than Germany, UK or France. So US is our first operation. What it was really was bringing the capabilities of ATOS and its partners to the Xerox ITO portfolio of clients because with 40 clients, you had most of the revenue that Xerox ITO was making in the US. And it was the first step of the strategy was materializing the synergies between the two. That's what we've been doing for the last 12 months because we closed the acquisition pretty much 12 months ago. The second step is obviously a very aggressively gaining market share and this digital edge offered that I was referring to just a moment ago. We just sold it to the largest agrochemical company here based in the US, 230 million on seven years. So it works. It's something that our customer are expecting which is bringing value. And obviously the third step is to carry on our inorganic development. ATOS has always been a very acquisitive company. We've been built through acquisition in our last 10 years and we're going to carry on doing so in the world but particularly here in the US. So where do you see the partnership with EMC going? And maybe you could describe, give us the journey if you will of the partnership. Well, you know, we've just launched some months ago the second time of this partnership which we called Enhanced Alliance 2.0. So it's more innovation, more engineers being certified on your technology. Now we have 5,000 engineers throughout the world certified on new technologies and we make a really, really big bet on the software design data centers. With both your technology and now with Dell too at the level of the blades. And I would add that we, on top of that, we are, we had this rare situation in which this alliance is, at the same time, more and more and more large, larger, okay, on a footprint, on a geographical footprint and deeper in term of offers. So that's what we're going to carry on doing. What are some of the, you know, sort of high profile wins that you had or initiatives? I know, well, you've got something going on in the Olympics, right? Is it Brazil? What are some of the things you can talk about? Maybe some examples there. Sure, so the Olympics, every two years it's a great thing to be the worldwide IT partner of the Olympics. We've been the worldwide IT partner of the IOC for now 15 years. We still have, we've got our contract goes until year 2024. So right now we're working obviously very intensively on Rio and on Pyongyang because we are beginning working four years in advance before the games. In the games, what we are doing is that we are operating 80 application, 80 critical application. I'm not going to list all of them but just to give you an example, you know, for example, we are operating the system that beams the results throughout the world in less than one second. We are operating the system that is recruiting and managing the 70,000 volunteers working on a game. We are operating the system which is issuing and managing 300,000 accreditations to athletes, journalists, public that has a status actually of a visa. You can enter in Brazil with this document, it's as good as a visa. And finally, and maybe the most impressive to me is that we've built a security operation center based on data analytics technologies which is managing 200 IT security event by second. Meaning that by second on a game, London, Beijing, Sochi, there is 200 IT event coming in so you don't know if it's malware, spyware, crypto lockers, whatever. And this without any impact on the game. You know, at the end of the day, we'll be in front of our PC, television or mobile on the 14th of August for the 100 meter final. I'm pretty sure Usain Bolt will be in the starting block. So imagine 60,000 people watching in the stadium, four billion people watching the game on their TV or mobile, even more and more mobile by the way, tablets and so on. You can't have an outage. At the end of the day, you can't. I mean, it's simply not an option. So it's maybe the most demanding service contract that we're operating and it's a great, it's a great great experience. So now we've been introducing EMC into the Olympics. We're delivering the Olympics with EMC in Rio and what really Rio is changing is that we used to build the infrastructure of the Olympics each time another one at each Olympic if you want, okay? Each time a new one. Now with what we are introducing at Rio, it will be for the IOC, a build and reuse. And that's completely new. Now at the next Olympics, Pyongyang, we will then take that entirely to the cloud. So it's, you see, it's a progression. Yeah, progression and it shows that the most secure cloud is private or hybrid cloud. That's why it, because look, the Olympics with 280 security event, that's the choice they have made. And you need agile. You need to have also low latency. Exactly. So you have high performance environment. Exactly. That massive scale. Massive scale. What do you, what's the, what's the choice product there? Is it using some of the DSSD stuff, using the, the rack scale stuff? What are some of the products that you know? It's pretty much all of all of the above. We've been, we've been operating with the, with the full range of the federation. And I said, and I said, what is interesting is that it really evolved years after years. Anyway, that's, can you get the cube there? And we can get a spot to broadcast live at the Olympics in Rio. You will be my personal guest. Thank you. Okay, the cube is going to Rio. No, demanding is really understatement. That is such a massive contract. But in general, you have customers, they want as a service, they have mobile, they're consuming. What would you share your advice to others that are looking to do what you're doing? You have a lot of experience over the years, working with the EMC, that's great. But now that you're on the front edge, you're in the bleeding edge, but you got to ship, scale, and performance.  Well, I would, I mean, I would begin with, as I was saying, with business outcome. What is it that you want to achieve? What is it that the technology you are bringing to the table, not take, I would not take an engineer position which is so proud of its technology, but really takes the other way around and say, hey, Mr. Customer, you know, what is it that you want? Do you want speed? Do you want agility? Do you want first that I take out cost? And what is it that I need to go through in order to achieve such a business outcome? That, to me, is the first important point and then design, you know, then design the solution that actually fit with this business outcome and which encompass, I'm really, really insisting on that, both the new world and the new world is all around us and the old world, because don't forget that there is still AS400 and mainframe all over the place and you still need to, for CIO, have a complete view and integrated view of the entire infrastructure. Michel Allen, take a minute to close the segment and share with the audience how they should engage with you, how do you engage with your customers and how do people take advantage of the greatness of ATOS? Well, we've built, I think, in the world and in the US, a very easy way to work with us. We are a company which is at the service of our customer. We are understanding humble, it's important, humility is very important, humble to our customer and we want to make it work. So, I mean, at the end of the day, beyond the technology, is really what is it that you do for your customer, even on a personal basis. Awesome. Well, thank you so much for coming on theCUBE, sharing your insights here. Thank you for welcoming me. EMC World. We are here, going to the Olympics. We heard personal guests. We're going to sit-up shop in the Olympics. Rio, great place. Looking forward to a great time here at EMC World. Wrapping up day three coming up, three days of wall-to-wall coverage. I'm John Furrier with Dave Vellante. You're watching theCUBE. Looking back at the history of the world,


So we can transport energy by either moving objects or by sending waves within an object. Now, in some ways, waves and objects or waves and particles can behave very similarly. And in some ways they're very different. So let's look at some of the similarities and differences between waves and particles. Okay, so they can both carry energy. And in fact they can both also carry momentum. You can send momentum with a wave or you can send momentum with particles. However, in particles you're sending the actual matter and in waves you're not. So the medium stays where it is. So particles can go in straight lines. Can waves go in straight lines? Well waves in a string certainly go in a line because they just travel along the line. If I have a water wave coming into a beach it can be very straight. Those wave fronts can be very straight as they come in. So they can go straight and they can go in lines. But if I put ripples in water then they do tend to spread out. Is there some way of making a wave in say water that goes in a nice narrow straight line? An answer is actually yes. If you have very high frequency waves then you can make them go very much like little pulses of particles. They will still be blurring and spreading out at the very edges but the scale of that spreading out is typically on the order of the wavelength. And the wavelength can be very small for certain kinds of waves. And when we go outside that regime we see a very stark difference between particles and waves. So particles exist in a particular place. And they also have a particular velocity. Whereas waves are spread out. A water wave can be anywhere on the entire surface of the water and indeed all over the surface of the water. And it can be going not just in one direction but a different direction and different parts of the wave can be going at different speeds. As an extension of that if you have one wave, one oscillation of the medium and one small displacement coming in contact with another travelling wave when they come in contact with each other the two displacements just add up. And so if you have a displacement up then they add up constructively and you have a bigger wave but if they're displacing in opposite directions then they cancel out and you get no displacement at that point. And that's called superposition. So if they add up like that so in other words they're coming in and maybe this one might be going one way and the other one might be going the other way when they're just like that then you can see that this point's going up and this point's going up at the same time. And that means they're going to add up to be a bigger thing and so those two things have what's called constructive interference where they add together. On the other hand if you have one like this then the other one is going the other way at that moment in time then when you add those two together there's nothing, no displacement. And that's called destructive interference. So let's look at two simple waves interfering. You can see that as they're moving they're spread out but when they pass through each other there's constructive and destructive interference and then when they're passed through they're just themselves again. Because the waves are going up and down at different points when they pass through each other they don't have complete destructive interference or complete constructive interference. It happens differently at different points in space. And in fact if you look at the energy the momentum carried in that wave it doesn't change while they're passing through each other and so it has a certain amount of energy beforehand a certain amount of energy all the way through and then that is conserved to the end as well. One thing that particles can do that waves can't do is travel at different speeds. The only reason a wave travels at all is because of the interactions between different parts of the medium. So the speed of a wave is a property of the medium. So for example supposing I was trying to communicate by throwing rubber balls with messages written on them I could throw those at any speed I liked but if I was trying to communicate by sound then I'm limited by the speed of sound. If I shout louder all I'm doing is changing the amplitude of the sound wave I'm not actually changing how fast it travels and the fact that the speed of a wave in a medium is basically only dependent on the medium actually provides a limit to how fast you can make a boat because boats tend to make a wave as they move and then they tend to sit in that wave as they travel. So the boat drives harder and if it doesn't make the wave go faster it just makes it have a higher amplitude and so that really limits how fast it can go and that's called the hull speed of a boat.