Who's actually still awake now? We'll try it what we'll do I think we'll use is we'll do the session and then as you all fall in asleep We'll do the time travel back to the beginning and just do it over and over and over again and get stuck one in one of these little time loops I think that sounds like fun No, okay, so let's do the actual session instead As Gabrielle said my name is Magnus Hagander In postgres I work on the core team and I'm one of the committers I also do a lot of work for postgres Europe. I see a couple of people in here at least we're over from that side of the pond When I don't work on postgres community things I work for a company called red pill in pro Which is an open source services company and postgres is one of our focus products I'm the you know principle guy on all things database for some strange reason Our company is entirely based in Scandinavia, which is why I'm betting many of you have never heard of it I'm based out of Stockholm in Sweden myself And again doing sort of all things postgres Consulting support training and things like that And this talk is actually coming out of more or less directly out of a customer project that we did last year So the talk is all about say a TARDIS for your ORM So who in here is actually let's do this or who in here considers themselves an application developer Okay, quite a few or DBA Okay, and anything else Okay, there's least a couple you don't count So probably everybody knows what an ORM is right It's the object relational mapper. It's how we you know application developers today to a lot of their database access And presumably who who knows what a TARDIS is? I'm almost disappointed every version of this is obviously the TARDIS right As we know it's a time travel machine it does a lot of interesting things other than just time traveling But we're not going to focus on exactly on the TARDIS part We're going to focus more specifically on just the time travel part and in this case we're going to look at application level time travel I think there is actually another talk right at this conference talking about temporal support in the database Which is sort of deeper down in the database layer This is implementing something similar at a higher level in the system Where we can do it on things that we have today So it's application level time travel, but it's still implemented in and on the database So in order to get going I'm going to start with a step back and just sort of look at what are the requirements? What were the requirements when we got involved in this project? So actually initially I had a bunch of people from a different department who do a lot of Java development In our company who got into the project and then they read through the requirements a couple of times and then they came to me Saying can we do this in Postgres so that we don't have to try to solve this problem? The first parts are fairly simple. So One of the requirements was there is an existing data model and we'd like to prefer to change exactly nothing Now we do have to make some small changes But there is a general ever high requirement on minimizing the changes to the data model because this is an existing application This is not a green level application It's an existing application that we want to add this functionality to so we need to minimize the changes Now what's contained in this data model is basically both detailed and statistical data So there is a lot of data collected over many years The customer is a government agency This statistics contains Very very sensitive personal data which leads to some of the more interesting requirements Unfortunately, I can't tell you what type of data it is But let's just leave it with the fact that it's really really sensitive and it contains data both at the very detailed level about individual people and Then you know step-by-step greater levels of aggregation where we in the end have a lot of reporting and things like that Now data gets into the system Mostly by being loaded in batches, which is true of most of this kind of system, right? I think there's like a four-hourly batch that loads one type of data a nightly batch that loads another type of data So we don't have a lot of individual inserts. It's larger chunks of data goes into the system But there is also people who are working on manual corrections all the time as These are entirely manual. They're done by the staff or at this agency. It's not a huge amount of them It's not The in particular there's no concurrency in this they have like people who are working with the data who just noticed that This data is wrong. Let me go fix that kind of level Or they pull out a report and it doesn't make sense and they have to go back in because the origin of the data Some of it is very historical So obviously data entry wasn't necessarily correct Some of it is manually added even going into the systems that then feed it into this now So there's a lot of manual work attached to the data as it comes in so it's incorrect So we need to fix it. So we have this one part of the system that's Getting the data in loading and batches and manual corrections And then of course we look at the data. We use it for something without that. It would be sort of pointless So there's a lot of things running like larger aggregate reports these things that can run for a few hours and dump report at the end But there are also quite a bit of like zoom in work where you're looking at I don't think they ever actually look at individual records I don't think they're allowed to do that But they look at like smaller groups of people and smaller or other statistical Selections that leads to a small group of people. I do actually think they're not allowed to look at a single person That's against the rules. Obviously. It's in the database if you have sequel level access, you can do it So this all sounds like pretty standard requirements, right? I'm sure you've all run into similar kinds of applications But there are a few requirements here that made it a little bit harder When we looked at it purely from an application level perspective One of them being that there is a requirement to reproduce incorrect reports Okay, so we're correcting the data we need to be able to reproduce the report before the correction And we also need to run the report with the correct data And at some point we may need to compare those two actually surprisingly enough that was not in the requirements Being able to compare what actually changed in the report because of a correction It was just the requirement to go back and say this report that I ran Back in October last year How did it look? from the source data Which at the core means we can't actually ever delete any data and Trying to do this without changing the data model from the application side becomes really hard There are some other challenges For example, we need to be able to go back and identify which reports Contain data about a specific individual Somewhere in the past. So which reports ran back in August last year that returned any kind of data about Steven Luckily, this is not a performance sensitive thing. It's actually okay if answering that question takes two weeks No, he's not Steven is never okay So these are the things that are like database level challenges, right? We need to reproduce the incorrect data which kind of feels wrong like there's a reason we corrected it And we need to be able to run this identification of what actually ended up in return now Luckily, we're not running, you know tens of thousands of reports every day It's a fairly small group of people who are running the reports against a large amount of data Now the other big challenge is we need to do this while maintaining application flexibility So we need this not to be present at every point in the application. It's a fairly large Java application that has you know reporting from a whole lot of different places And it contains something that's interesting as a manual query interface Because these are statisticians right they can input queries Now the input the queries they don't actually put them in SQL But they can partially put them in I think it's called HQL for those of you know hibernate. It's the That's exactly my reaction as a few people here said like I don't like HQL, but you know, they're Java people And they also have a simple UI where you know drag and drop fields and joins and things And we need to deliver these Requirements for those queries as well It's reasonably simple ish as long as you actually know exactly which queries run, but we don't We can basically run arbitrary queries and we still need to be able to do this And then as I mentioned whole preferably we will do this with zero changes to the application. Obviously, that's impossible Right, it's not going to be zero But we need to minimize it and we need to minimize it as much as possible because also we have you know these Multiple different teams working on the different parts of the application and we all know how well You know cooperation of those things can work in particularly in government organizations. I'm not biased at all So the tool box that we have here is basically we have J boss and hibernate. That's the part that we can't touch Because you know the application already exists Now it helps us that they at least choose an open source framework to build it up And actually a fairly flexible one as far as ORMs go hibernate is not that bad There are many examples of ORMs that are much worse than hibernate Hibernate lets you do some interesting things now, but you can do this I mean the solution is actually independent of our M. You can do this in most ORMs But the one that we used was J boss and it was hibernate and we got the option to use postgres Originally the system wasn't running on postgres, but when we got involved they were already thinking of moving it to postgres So we got to use that which gives us the flexibility of using the whole set of tools that we have in postgres Now if we look at the actual database schema of this existing application, it's a fairly simple schema because you know it's generated by an ORM Those schemas tends to be fairly simple It's at least an ORM sophisticated enough to know about the concepts of foreign keys So we do have foreign keys in the system. We do have these things. So it's an ORM generated. It has quite a few tables But there's really nothing difficult in the schema itself Because you know hibernate is designed to work on any database So the original schema doesn't use anything postgres specific, you know, it's integer. It's strings It's you know date times things like that and some foreign keys, but fairly simple Now it gives us a couple of things that hibernate does at least by its default that will make this solution easier for us All tables live in the public schema They don't use multiple schemas and they don't use fully qualified schema names in their queries We're gonna exploit that and All tables have an ID column With the serial attached to it This is enforced by hibernate. Actually, I think you can set up tables without that in hibernate, but it does it by default That also makes our life much easier and The way it's been set up the application has very few schema changes over its lifetime So there's a lot of changes to the data, but new versions of their application itself Generally doesn't change the schema much It changes the application layer. The scheme is fairly fixed. There are changes to it But that made us gave us the option to say that we actually can turn off and we will not be using hibernate's ability to automate schema migrations Just say any schema migrations will just require a DBA to extract the script from hibernate Then run it and potentially adjust it to work with our solution So we automated the initial setup of our solution But we didn't automate the incremental steps. We probably could have done that. It just wasn't a requirement So we got a little got off a little bit easy So based on that, how do we actually solve a problem like this? I'm sure most of you who work with databases for a while can figure out step one, right? Step one, we need to keep our old data around. We can't actually go delete it How do we typically do that? Well, we stored in the history table somewhere But what we need to do which we don't necessarily need in in like your typical auditing table Which is how we usually do this is we store when did somebody change something Now instead of storing that we're gonna treat keep a track of for every individual row Throughout which time frame is this version of the row? Existence so it's sort of like we have the multi-version concurrency control in Postgres It's sort of that but at an application layer. We say this version of the row existed from this period in time until this period in time and Then if you updated it you get a new version that has a different validity period and That way we can reconstruct the history of this row So yeah, everybody who does not know how our history table works Okay, good at least your database people right who does not know what a range type is Okay, that's more impressive like one or two. This is clearly a Postgres crowd I Ask that when you go to like Say a JBos conference Like who knows what a range type is you get maybe one person who knows it So what we do is basically we do a traditional history table and We put a validity period on it using a range type Query wise this is just like this So for a table called table one that would then based on the previous requirements always reside in the public schema We just create a history table one always with the same name That is exactly the same as public one as public table one But we add the column we just call it underscore valid range to make sure it doesn't conflict in case there happens to be Column named valid range in a table somewhere That is a tstz range. That's the timestamp with time zone range Now in fairness, I think there is also a not null on that that I forgot to get on the slide But that doesn't entirely matter in this case So who knew about the construct that you could say create table like and then also add more columns Okay, that's quite a few It's good to know it saves you a hell of a lot of typing and It makes automating this much easier because I can just loop over all the tables and run this Whereas if I didn't do that, I could do the same thing by you know Inspecting PG attribute and get all the different columns out and like recreate them on their other side This is obviously much easier But what like does is it copies all the columns as the definition is now? I've heard people who think it's different in that if you go after you've done this and you go back and add Column to public the table one. It does not appear in the like table in in any way So it's a snapshot of the table as it looks now plus one call And that's the part where we have to go in and manually do this if we do a schema migration And we add a column to the old one. Well, now we need to add it to the history table as well And in that case, we're also gonna have to add it with any not null constraints removed Otherwise our historical rows just won't work So we create these for all of them Okay, for those of you who may not have used Range types very much. This is just an example output from one of my test databases of this underscore valid range column I took just the valid range column Obviously, you'd need more than that to make it useful But this will show two versions of I believe it's the same row. Yep We can see on the first row. That's the validity starts at February 17th at, you know, 1449 52 or whatever and it's valid until 14506 and Then there is a second version of this row Which is valid from 14506 until infinity There's also the important thing to note that the start of the range type here has a square bracket and the end of it Has a regular bracket Now that defines whether it's inclusive or exclusive Right is this exact timestamp because that's exactly the same timestamp as we have down there Is that actually included in the range type and in order to create our proper indexes and our constraints? We need to have one of them being inclusive, which is this one here and one of them being exclusive, which is this one here If we didn't do that, we'd have if we were to look at the way the data or the data looked at this exact timestamp We'd get both versions and That's not good That would give another interesting type of error in our reports getting two versions of everything So this is basically when we insert a row like this is a row that's the current version always has a validity into the infinity Because we insert it or in this case. We updated it at this timestamp And we've read it now and don't say anything else then that's the version that we're gonna look at And then of course to make this to make sure that we get this right we need an exclusion constraint Which is again using this syntax we end up having this is where we can rely on the fact that there is an ID column in every table So we're just for each table We're going through and saying for this history of table one added constraints called table one exclusion in this case Exclude using gist, which is our standard syntax for creating an exclusion constraint With ID sorry ID with equals underscore valid range with double ampersand Says for each ID value so for each entity so to speak from the Java side There cannot be any overlapping ranges of validity That's what guarantees that when we look at this whichever sort of a valid point in time We're looking at we'll only get one version back And if we had created this one with a square bracket on both sides that constraint would not create Because it would not complain that we have an overlap between these two timestamps So this will guarantee that and it will also give us an index By which we can make fast Searches if we include the ID field and a valid range with with a valid operator And as it turns out, you know, that's how hibernate issues most of its queries It will look them up by ID So if we can just specify get away of specifying it into the valid range and setting the ID It's going to work now to make this part here work You need to install a contrived module called be tree jest But it's part of the standard postgres packaging It just needs to be you know create extension Steven it would be nice to fix that we should just include that in core, but we don't right now So right now you have to say create extension be tree jest. It's not very hard It became easier once we got extensions in 911 Okay, so now we have a history table and we have a main table or actually we have a couple of thousand history tables Then of course we need an update trigger This is our simple we say create a trigger We do it before insert or update or delete because we need to trap all these operations On the table for each row execute procedure and we just decided to put the procedure in the history scheme as well Just log table trigger Means for every row that ever gets updated or inserted or deleted. We're gonna run this code So what we basically have here Is our public schema contains all the current data at all times and The history table will contain all the historic data So basically we need to track every operation on the current row and somehow create an entry in the history table To reflect the changes and to then to tag this with the time that we're at right now So I'm not gonna show you the actual create function statement because it's too long But if we look through the actual code, this is a PLPG sequel function. That is this history trigger And the first part is pretty simple the insert trigger right? We'll say if tg op equals insert So if this is an insert Then basically as we insert it into the main table. We're also going to insert it into the history table And we just say insert into history dot and we copy the name of the relation tg role name And it's select dollar one point star We're dollar one. It's gonna come from new down here new being the version of the row that we just inserted And then we just add a comma so because this will contain all the columns from the original table Which we copied into the history table using the like syntax and then we're gonna add the validity period Column where we just say then the tstz range is a function that will take three parameters and return a range type for timestamp with time zone The first is the starting point When does this row first become valid? Well, that's gonna be now That's right now How long is it gonna be valid for it's gonna be valid until infinity because we just inserted it into the database It's the only version that exists But if we start looking at the database the way it looked an hour ago, it didn't exist, right? So from right now until infinity and Then we add the final parameter is this inclusive and an exclusive part of the range type So we're saying the the left-hand side the start side is inclusive. It's from and including now Until you know, well, I don't think it makes much sense when it's infinity at the other end saying whether it's including or excluding of infinity itself Because just before infinity is still infinity And we just insert that using new and then we return it new if we forget to do the return new Well, actually if we forget it completely it won't go in anywhere if we do a return null If we forget it completely we get an error if we return null it will go only into the history table Now we did actually look at that initially to store the data entirely in the history table, but it turns out there are still Unsurprisingly most reports most things still run on what's the data like right now? So we keep the data in the main table as well as basically a performance optimization To making sure that you know, we can fast access the current version of the data Now it gets a little bit more interesting when we start doing updates Because well, so the first one is pretty simple right if trigger op is update And what we're doing then is here is you can open a cursor selecting the valid range From this history table For the same ID This is the second part where we rely on the fact that every row has a unique ID column That makes life so much easier for these things So where for this ID and we order it by the valid range descending So basically this will get us the latest version That's in the history table If there's a hundred versions we get the latest one if we just inserted the row we get the only version that's there And then we do a limit one Steven no, we cannot check for just where it's infinity because the row might have been deleted concurrently So it we could almost do that Anyway, so we had the limit one for update so that we also at this point look the row and Then again using new ID because we run this once for every row And then we just fetch that into a temporary variable then we create a TSTZ range so create a new range saying the new validity period is from Lower TT so that's the start the previous starting point Until now With the same reasons and if that is empty What that means is that somebody else already updated this row with exactly the same time More or less or in for the within the restrictions that we have that means we already updated this row in the same transaction If that happens, we're just going to overwrite that So what we do in this case is we delete the history entry Because we're going to be writing a new history entry soon Whereas if this thing is not empty That means okay, we now actually have a new validity period for this row So what we do now is we go back to this current latest version of the row in the history and we update it and We change the valid range To be again a new timestamp range Starting at so dollar one going by which is the lower of the previous one meaning Okay, we don't change the starting point of the validity if it was inserted an hour ago We're going to change it. It's still going to be inserted an hour ago But we change the end time To be now So basically we're saying this version that's already in there. It used to be valid until infinity It's no valid until now Because we changed it Not if it's a new no, not if it's a new transaction Yes, you could so if you this is where we get into the point. We don't have a high concurrency on the updates so Yeah, no, we don't we don't have to worry about that part now We can actually and we have in and we do have a slightly more complicated version of this that also checks the TX ID So you can do that I explicitly excluded that version from the slides because it's longer But we do have a secondary check in here actually based on TX ID But we because we initially thought it was fine and it was actually fine in production, but it wasn't fine in testing We broke it in the unit tests Yes, you mean this row I Had to cut this proceed. Yeah, I had to cut the procedure into two slides So once we have updated the old row and set the validity until now We also need to copy in the new version of the row With an insert so we get a second row in the history table for the same thing Where we insert all the data and the new valid range starting from now until infinity Because yes, we do need one of the old versions of one of the new ones and again We need to specify this one here so that the previous one ends on the regular bracket The new one starts With a square bracket so that we get no gaps Where am I missing the single quote? Yeah, oh, yes, that is correct Okay, so yes, there should be a single quote over there because execute takes a string Thank you. I will tell you that typo is only in the slides Because it wouldn't actually work otherwise you wouldn't be able to do create function otherwise Yes, we did There were arguments for and against In particular in being able to update the function and how and getting this done with people who are very uncomfortable using database interfaces aka Java developers I mean no offense to any Java developers in the audience if you are here You're probably not one of the Java developers who don't like databases But there are certainly those and then in the end we do return you so that we actually let this change Happen on the master table as well so the update gets tracked on the master And of course there's one operation left Delete is very similar of course to an update What we do here as well if the operation is delete again within open cursor this this is exactly the same query So we get the latest version That's in there Again the interesting case here is if this timestamp range which is again the same one as before Do we already have an update at this exact time? That means that somebody did first did an update and then be the delete For those of you who are not used to ORM applications. You'd be surprised how often this happened I've seen cases where they does like 10 updates and then deletes the row all in one transaction Which is kind of pointless But you know, it's what they do So we do the same thing here if it's already been updated, but it's not been deleted We're going to delete this updated row because you know transactionally It's never going to have existed at the end because we committed everything at once and If we actually need it or if we actually did the delete it's a new one All we do is we found this last row and We update the last row and we set the valid range Well, it's the starting point is still dollar one lower TT, but it's not valid until now So it's no longer valid until infinity because we delete it. So it's valid until right now So basically a newly inserted row will have a validity from insert time until infinity When we do an update The new row the new version will have an availability from now until infinity and we change the old row to be from infinity to right now and As we delete them we just say well, it's not valid anymore We don't like this guy anymore and then we do return old And That's one of those things that tend to confuse some people like yes on a delete trigger We have to return old because there is no new version of the row because we deleted it Again, we could return null and then we're just going to block the whole operation But we want the current version of the data to be materialized in the table itself So now we have the data now. We're actually keeping our historical data Yes, Jeff Yes So yeah, you we could have a version which had just the latest version would only exist in the current schema and All the previous versions alone would be in the history schema with this one We keep both the current and the history in the history The main reason for that is it's much easier for the application To query because now it doesn't actually you can input a timestamp whether it's within the current or not The application needs to know less if we do that or our view creation can be simpler You could but they become a lot more complicated No, well Not easily because the history has multiple versions in it I guess you could have implemented that on top of a history like one partition for all the old and one for the new But then you'd still need a view in in that hides the the actual time validity Look at both tables. Yeah, I Don't know we didn't look into that in detail because we looked at how it would affect the application side And that was enough to say that we're not going to do that I am not sure that we would know it might well be a real performance hog So accessing this data Fairly simple anyone who's worked with the range types know the syntax right if I want to look at this You know, I just specify the time I see I select you know ID in this example. I have columns A, B and C I don't know why it highlighted that guy From history where ID equals 42 and valid range and then we have the act greater than the sign Which says, you know, give me the version where this timestamp with time zone is inside the valid range and The exclusion constraint that we created promises that this can be zero or one rows, right? It can never be more than one row It can be zero if we ended up after the row was deleted or before the row was inserted But we'll always be able to do that Now this got us I mean we can just put this in the application right and we're basically done Unfortunately, that doesn't really fall under minimum modifications Modifying every single query that's done anywhere in the application That is not very minimum It is sort of what their developers had considered doing Before we got involved and got starting You know seeing what you really do because in particular when we start doing this and doing joins on top of it You have to like qualify all these validity checks against or each individual table It does work, right? The query just massively explodes and if you're putting that in your database, nobody's gonna be happy So we're gonna fix that at the database level and we're just gonna do you know create schema time travel Because for some reason they didn't like us calling that schema TARDIS in the actual production release So we're gonna create another so we have a public schema that has the current data And we have history schema that has the whole history So we're just gonna create a time travel schema that contains nothing but views And each view is gonna look like this Create time travel dot table one which is the same name of the table as select And we do actually expand the names of the columns here because we don't want to include the valid range column Because this view is gonna look exactly like the table not the history table But we selected from the history table and we just say where valid range app greater than and To get the timestamp and we use a custom guck So we just create we invent a parameter that is called history dot timestamp Current setting will just read the current setting in our session of history dot timestamp and Cast it to a timestamp with time zone if there is no setting we don't have that guck set This is just can give us an error So basically this view requires the application to set the value of history dot timestamp But once you've done that this one controls the time in which you are looking at the database and You can join these views and you'll get this sort of the the exploding query with many many validity checks But it all gets auto-generated And what we then do is we set the schema search order To control whether we're looking at public or history This is where we initially at some point we ran everything through this time travel point and realized We're paying quite a bit of performance overhead for the case when we're actually looking at the current version of the data So that's why we also made the application actually talk to the main schema So when we just connect and run a query we'll get the current version because it's in public that's what our ORM is doing and To enable time travel what we really need to do is we need to do set the search path to time travel and Then set history dot timestamp to this value and do a select star and that's the way this table looked at you know the 7th of March at 1432 And to continue on that the same results, but if I then just change it to 1429 So three minutes earlier you can tell I was not running this on the production database When I was building the slides and we can see it's the same table But the row is looking different Because we had an update obviously happen in between this and that's sort of the core of it is the application needs to switch It schema and set the timestamp and when it's done you're going to reset all and you're back to looking at things the way They're looking right now Which is course the reset all don't forget to you know configure your connection pool to actually do that Interesting things can happen if you don't because another session gets your connection back Which is like time travel back to August last year and you know, that's an interesting result So what we basically get is automatic trying travel right? Once we inject these variables we do need to do that now exactly how we do that is of course going to depend on the framework You can do it driver level we tried a bunch of different things I'm not going to tell you those parts in detail because I don't really know hibernate that well And I don't really want to know hibernate that well But we did some different examples you can inject it at the driver basically at one point created our own driver And injected it there. We can wrap individual queries. We can wrap it in a function call We tried a few different ways so just an example of how we did the driver injection part is basically For those of you love Java right we this one is well that used to be the company name Postgres dot driver and we just created our own driver that extends the original driver we override the connect and We call a little function called inject time travel which will run these set commands on it Of course, yeah, this is simplified right that's not quite enough code, but you don't need to do much You just need to trap those two things and then using annotations or whatever to actually set the time In some cases just passing instead of calling, you know execute SQL or whatever it might be called you much as they execute SQL time travel include a timestamp So there needs to be an interface for it, but the advantage is the main OLTP app that does all the corrections needed zero changes and The reporting app basically needed to add a field where you could input the time and then we need to capture that and Inject it in and then we could run exactly the same Java code of Iran before and we'll automatically get injected for the Hibernate language queries as well So it did all those now there again. There are a few things to consider about that, you know, don't forget to reset it weird things happen And query the public scheme of recurrent data Once you get a bunch of versions and a lot of data, that's a very efficient Optimization like don't pay the overhead of going into the history scheme if you don't have to It was when we're looking at that's when we decided that yeah, we're actually going to materialize the current row in two different places Purely as a performance optimization because it does make quite a bit of difference And of course, you know, it depends also the one thing that this the automated system does not set up for us Here is indexes Right. It only set up the index on the ID Any searches on anything else would actually by default only use the index on time and nothing else Now for a number of reports that was fine because we didn't have the same level of performance requirements on the reporting For some you could just go in and manually create that index or well manually would go version controlled on all that stuff But you do need it. We didn't try to auto generate like time travel indexes for every single Index in the system that just seemed unnecessarily complicated Of course, this covers all except that one last requirement Identify which reports contained a person Now this is the part where we don't have quite as nice a solution I think the solution can best be described as brute force So to do that we log all queries all select queries are locked And then we rerun the reports To verify which data so basically if the question is which queries that ran through August Actually return data about Steven, which is just going to rerun every query that we ran in August and See if they include Steven So this is slow. There are some heuristics for some of the reports We know how to like short track some things and make it more efficient But that's sort of at the very base of it and that's actually acceptable because this is something that Frankly, they expect to never have to use but if they do Having this take a couple of weeks is perfectly acceptable Yes, so it's likely unrelated to the time travel part, but no you don't well you need to Probably if you want optimization You need to run it manual because you might be able to say like this whole set of queries cannot be there But as long as you rerun it with the same parameters, it's gonna return the same data So yeah, the brute force method is just rerun everything. Yes. Yeah, so Agree if you have like people they have the depends on which level the aggregate is at So yes, you're gonna have to adjust some of the things Because some of the real thing the reports that actually return, you know aggregates over a million people are Never gonna have this requirement Which nobody cares if Steven was included in that the problem comes if Steven is included in a report that returns like ten rows Because he's probably identifiable within those ten rows So yeah, that's there there is manual stuff involved in when you get to this point Like you need to do some some thinking about what what's actually represented But the main thing is we needed to keep all the required data And really the only way to know who looked at something is to record all the queries Well, the no the other way would be to record all the responses and That would have been much more data as in much more data Materialize every single row ever returned from the system just doesn't really scale So one other word of warnings around this is or I'm level caching Hibernate does caching at two different levels Query level or entity level if you actually have that enabled Very interesting things can happen because you can get back different rows from different times Basically, you need to just either turn that off, which is what we did Or you need to make it aware of the time travel and like key it including the timestamp But you can't leave it on in default mode that will give you really really funky results So I think the first conclusion may be the best one here like anyone is not used range types You you probably have used cases for them. You just haven't figured them out yet Of course, we could have done this without range type with just regular start and end times But it would have been so much more complicated Yes, much of the code is auto-generated, but it would still have been so much more complicated range types are great and We can actually you know trick ORMs in a lot of cases so that they're sim they're so simplistic that they're easy to trick Like if the ORM actually made it easier to spread your data out across multiple schemas, this would have been much harder Now we knew everything was there the ORM gave us the requirement that there will be an ID column everywhere Okay, so let's exploit that and then trick the ORM because hibernate itself can run Like perfectly exactly normal as it did before having no idea that this happened once you manage to inject these things And I think it's a good example of like the flexibility of Postgres that lets you really take this problem that trying to do this at the application level I mean you could do it, but it would not have been minimally invasive It would have been many many many thousand lines of code This is a fairly simple way to solve what's actually a fairly complicated problem So if you haven't already look into range types, that's good fun, and you know, it's fun to trick the application developers Especially the ones who aren't here. Okay Well, I'll thank you very much, and I think we have time for a couple of questions. Yeah, yes So statistics on inserts and updates. We are there were some impact. It was surprisingly low In in the initial benchmarks people didn't think it was working Because the thing is we're adding this we're not adding any more commits It's kind of shows like because the problem is they were limited by number of commits in the benchmarks and it doesn't add any more commits It adds more data So actually the insert and update speed were barely affected at all. It was less than 5% Now query speed when you go into time travel will drop obviously because we're looking at much more data and we've got more complicated Queries but insert and update were surprisingly low. I would have expected more I think we had I don't remember what we had the requirement but the requirement allowed for quite a bit of slowdown like 30 plus percent But it was nowhere near that requirement Yes transparency to the application Complete transparency in that the views look exactly like the tables And that was one of our original requirements and making it simple and like making the application devs More or less not have to care and in particular those statisticians who actually input the queries need not to know at all That was the main reason yes no, actually party we Given that we know we're not gonna have a huge amounts of updates through them We're not currently in a position where we even had to start thinking about partition Partitioning them. I guess you could probably partition based on validity time But again, I'm not entirely sure how smart the optimizer is in figuring out Sending the queries to the right place, but it also depends on why you're partitioning if you are actually partitioning for the access Or if you're okay scanning five partitions every time of when it comes to all data You just need partitions for you know vacuum or something like that You could also partition based on ID Because almost all of these have I mean they have a uniformly increasing ID where some rows are deleted But in the history table, they're never deleted. They're always gonna be there Okay anyone else Slides are not available yet, but slides will be available on that website The code itself is not open source unfortunately The parts that are on the slides obviously are and actually just the other one is basically It's just a loop that runs the create table and create trigger statement on every Statement the the only part that would be useful that is not either in these slides or trivial is the injection code in the Java side and that's unfortunately not our property at all. So that part we can't help you with yes Yes, if there was significant schema updates, there would definitely be a significant uptake in DBA things if we were looking at a lot of schema updates We you would have to script that side It depends on what they're I mean adding a column to this is very simple You just need to remember to add it to the other table and then alter the view Things like changing data types of a column would be really really hard because then then the whole thing would sort of fall back You'd have to switch to a different Type so that would be very hard, but just adding columns is fairly simple Okay, without I am out of time So thank you all very much


I didn't plan that out really, I'm very passionate about both, I'm very passionate about start-ups and each idea has its own impact in different ways and yes it happened to be that way but it just happened to be that way, I didn't think that element through, I think I genuinely focus on the vision that each company has and then of course as a human being I have a larger, I feel like I have a larger purpose which I want to take forward. It's a B2C and then we have got a B2B sort of line as well, we've got a huge digital marketing plan that we have in place, our social media handles are on, we are meeting people like amazing people like Yol, which Yol are the mediators to really getting out there to our consumers, we have outdoor marketing that we're going to do as well in terms of adverts, there's a lot going, eventually we will, but within time, yeah within time. In 2016 when the first one started it was called Piper where we were using the Raspberry Pi, I don't know whether you've heard of it, it was invented in Cambridge to help, to put together a laptop for children, for young children, so it's not like it didn't work out, the team was planning to move to Silicon Valley and I took a conscious decision to stay back in India because I wanted to do these various things, so Piper is still going on in Silicon Valley, I am on the advisory sort of panel, so it didn't fail, I'm just not part of it and then at 17 I started Swatantra. Everyone knows that, I think it's out there in the public, I don't think I need to say too much about that, but we're doing things very differently like you may have heard in the press conference as well, so we're sort of following an inventory based model, we have a B2B as well, we're going to be a global platform not just restricted to India, so many different things that we're doing very differently, which hopefully will set us apart and will change the dynamics as well, so we're not a typical typical sort of, we're not following that typical marketplace, we're not an aggregator, so we have our own label as well, so we're different, we're different, that's it. You know typically it boils down to me as a daughter and my parents and then my friends and family and my team and that's for any individual, so I think I feel very blessed that I have all these people, genuine people around me who give me so much love and who understand me and who support me in everything that I do, so that's the way I think about it, of course coming from a legacy, I feel very privileged and proud and I think, yeah, I'm proud in a very good way and blessed and hopefully I'll be able to do justice to my parents and my team and my best friends. I don't think so, you know, I think, so for example, so everyone goes through different sort of pressures, right, it could be someone who needs to get a student loan who wants to do an MBA, so they have so much pressure, I think everyone has pressure and this is my sort of journey, so everyone has pressure, you sort of have to get accepted and move forward, so I don't think so, you know, my parents have also sort of brought me up in a way that I follow my dreams and my passion and you know, just sort of that philosophy that try your best to need the rest to God, so when you have that attitude to let go after a point and detach yourself, I think things get much easier.


Hello and welcome to Dane Fern and inside the Stone River Estate, an estate that's just a stone throw away from Dane Fern Shot, Pincente as well as the schools of Stain City and Dane Fern College and a short drive away from St. St. City itself. This is an area that is both tranquil as well as up and coming with home prices that are about to skyrocket in the near future. My name is Chad Rivera. This is the private property home shopper shown. I'd like you to come with me as we explore this 3.5 million four-bedroom two and a half bathroom townhouse in Stone River Estate. Come along.


Okay, so here we are then, welcome to this presentation. And now as you asked, I work as a software engineer for Microsoft, I work part of the QIN4 team there, so we did, you know, in-spec project and other projects using the BDVF. Today's presentation is about how to use BDVF to generate some security policies for our cluster. So yeah, in today's presentation, I will show you a quick introduction of different security policies, probably this is something you already know, but as in case we are already in the first, we are going to show how can we generate those policies by using some of the BDVF tools, then we'll show you how to audit those policies to others around the cluster, and then I will present the different implications of this approach. Yeah, I will show you a quick introduction to something. This talk is to be together with Alban, but Alban is not here, so Alban will be here and I will show you some of the most traditional of these presentations. Okay, so let's get started with the security policies concept. We are known that for the software is the very kind of security policies that we can use to make our cluster more secure. There are so many of them, but in this case, we concentrate on ZCON, both capabilities and network policies. They are changed by open policy engine and other security related things that we are going to cover on this presentation today. Well, the problem that we have with the security policies and core netizens that they are difficult to configure. So in order to configure those security policies, we have to understand what our application is doing. And this is something that we don't understand all the time. For instance, if we want to configure a second profile, we have to know what our assistant codes are, are secured by our application. The same applies for our latest capabilities and in the case of network policies, we have to understand how the different network services that we are running there communicate to each other. Also, yeah, this is something that unfortunately is true many of the times the person that is defining those policies is not the person that is developing the applications. Maybe there is one person that develops the application and then there's another person on the team or maybe a different company also that has to configure the security policies for a different application. So this is, for this person, this is very difficult to understand what are the security policies that has to be built. How to configure those security policies because many times that person doesn't have a lot of knowledge about the application. So yeah, in today's presentation, I want to present you a different way to generate those, or a different way to define those policies. So what about if we observe the application, if we use some tools of certain application based on the behavior, based on the activity that application is doing, we can define the security policies that we want to use for our application. So the idea is actually very simple. There are two steps. The first one is we have to observe the application. We have to understand what the application is doing. There are different technologies for that. In this case, for sure, we are going to use EVPF because it has a very low overhead. So when we are observing the application, we are not affecting too much the performance of that. And also EVPF is very, very flexible. So we are able to do a lot of things that we need to do there. So yeah, that's the first step. We observe the application. We capture the activity of the application. We know what the application is doing. And then by using that data, we generate a policy. This is something that sometimes is really easy to do. For instance, for a second profile, this is just to capture the system calls and to put a list of system calls. For both capabilities is the same, but in other cases, like network policies, this is really difficult because we have to generate the network policies in a way that are easier to understand for the developer or the operator of the platform. So yeah, depending on the kind of network of policies that we are generating, it could be very easy or it could be really, really difficult. So yeah, let's go ahead and start with the first kind of security policies that I want to cover today. This is second. So this is a mechanism on the kernel to limit the system calls that a given process can make. And yeah, how it works is that we define a second profile, we put a list of system calls there and we can tell the kernel, okay, if a given process executes this call, perform this action, and the action that we have are for instance, just get the process. We can send a signal to the calling process. We can deny the execution of the system call and return an error. Or we can also execute the system call and lock to the system. Actually, yeah, there are other actions that are support, but my intention here is not to go into the detail of those ones. This is what a second profile looks like. So we have a default action. This is what happened by default when our, when a given system call is not a secure it. Or better, when a given system call is a secure it and it is not on the list that I have defined, then I can define different list of system calls and put an action for that. So in this case, I'm allowing the few tests on the red front system call. Yeah, actually in second profile, I can even define different list of system calls with different actions. So I can have a sort of mix, allow, and deny this together. In Kubernetes, we can define those profiles in the pod security contest. This front coordinate is 1.19. Actually, before it was also possible to do, but it was defined by using an annotation in the container pod, but now this is the precary, so I'm not going to talk about that today. We can configure the second profile at the pod or at the container level. So it means we can have the same second profile for all the containers on the pod or we can even have different second profiles for each container in the pod. So this is what it looks like. You have the security contest. We have the name of our security profile and that by default is looking in this bar lift cubelet path. So yeah, actually handling security second profiles in Kubernetes by Han is a little bit difficult because you have to be sure that the second profile that you have is installing all the different nodes so you have to keep that aligned on the different nodes. So this is sometimes difficult to do. So for this reason, there is the security profiles operator. So in this case, you can define the second profile as a custom resource and this project, this operator will take care of synchronizing the second profile on all the nodes in your cluster. This is just to show you an example of how we define a second profile. So here we have defined the second profile and yeah, in this case, this is a very simple one. We are only going to log all the decision calls that are the security. And this is how we associate a path to that second profile. Actually, the SPO project has different ways to do that but this is not so relevant for this presentation today. So yeah, let's get into the interesting part of this talk. So how do we define a second profile for our application? Of course, and a low list is the preferred approach. We only want to allow the decision calls that we know that are safe. There are times that new decision calls are added to the kernel. So if we are using a block list, maybe one of those dangerous decision calls is implemented, we are not going to block that. So yeah, in this case, this is better to have an allow list to be in the more secure side. But the problem with this approach is that we have to be sure that we include all the decision calls that our application needs on the profile that we are defining. And then the question is, okay, so what are the decision calls that my application needs? Well, how can we know that? The first option is, yeah, if we are developing an open source application, if we have access to the code of the application, we can check and we can try to read the code to understand what are the decision calls that the application is using. But actually, this is really difficult if you are using any standard library, maybe from Go, LAN, C, anything, you are doing a call to the standard library but you don't know what is the decision call that a standard library is doing to the kernel. For instance, in some cases, you can just call open but the standard library but it implements an open app call to the kernel. So there is no like a one-to-one mapping there. So yeah, it's a difficult approach. Another idea is to use a trace. We could run an application with a trace. A trace is a tool that brings all the decision calls that are executed by a given application. But this is like a more in-box approach because we have to... We need a trace to be available in our container. So if you are running a disk release image, you are not going to have a trace there. So yeah, it will need some more work to make that useful. Another option is, yeah, we can define a simple set-com profile by using the login action and then we can check the audit log of the kernel. But actually understanding what is on that audit log is not so simple because for instance there, we don't even have the name of the decision calls that are executed. We only have the number of the decision call. That number depends on the architecture. So I will have to map those numbers to the name of the decision call. So again, even if that's possible, that's not so easy to do. So the thing that I want to propose here that we want to propose here is we can use a tool that records all the decision calls that are executed by my application and based on that, we can generate the second profile. As far as I understand, there are two different projects that allow you to do that. The first one is the security profiles operator that has an EVP recorder and the second one is the second advisor from Inspector Gadget. So let's look at the demonstration of the first approach by using the security profiles operator. Hello, I will show you the security profile operator on how its EVP recorder can monitor system calls and generate the second profile. So first, I need to enable this feature. So I'm checking the enable EVP recorder feature is set to two, that's cool. Then I need to enable in the feature in the namespace how we do my testing. It has this level set to two, that's cool too. Then I need to create a profile recording resource. Here I have a profile recording resource in the namespace of itals and it has a podselector with a selected level I want to test. Okay, so I apply this profile recording. Now it should be installed on my cluster. And now I deploy my pod. This pod will have a level that match the level selector of the profile recording. Okay. So now let's see my pod, it is deployed. I will generate some network activity on it. For this, I use port forward to enable it to my app and I use some calls command to connect to my app. So this is working fine. Now I can stop the port forward and let's delete the problem. Okay. So let's see if my second profile was generated. Yes, it is installed. Let's see what is inside. On here, I see generated a second profile with a default action of returning an error except for the system calls which are not. Thank you. Okay, so that's the first way to generate those security second profiles. And the other one is by using this inspector gadget second advisor. Yeah, actually the mechanism that this uses is very similar. This is something that we implemented almost in parallel, two different, two very similar features implemented by two different teams of parallel. So yeah, just let me show you the second one. So in this demo, I will use the second advisor to monitor my workload to see which system calls it is using. So let's start the gadget with advice second profile, ask it to monitor the second demo on this face on the flow pattern pod. I get a trace ID that I can use later to stop it. And now I can deploy my pod. This pod is unconfined, so it doesn't have any second profile. I see it is running already. And I will not generate some network traffic to it to exercise different code paths. I use path forward, so I can use code directly to it a bit later. So now I run a couple of codes command. I can see the result with the webpage. And now I can stop that. I will stop the path forward and then get the result from this gadget. So I reference the trace ID and ask it to return the system calls in this file. So if I look at this file, it will simply be the list of system calls in the JSON format. So you see there are different system calls for socket TCP connections and so on. I love this. I have this file in the ML format as well. This is a second profile from the security profile operator. And here by default, it returns an error for any unknown system call or it allow the list of system call it has detected before. So I can deploy this second profile that I generated. Now it should be installed on my cluster. And I will deploy a pod that reference this second profile. So now it's no longer unconfined, it just reference the second profile I generated before. So let's deploy it and that's it. Okay, so now we understood how to generate this second profile. But yeah, once we have those second profiles in place we have running our applications with those security profiles. How can we know if a given second profile will break my application or better? How can I know if a given second profile is blocking some system calls that my application is trying to execute? Again, one option is to use this option lock. Sure, in this case it will not block the system call but will give you some understanding that the system calls are being executed. Then we can check this is locked but again as before this is difficult to understand. So an alternative that we have is to use this second audit gadget that provides you, that tells you when a given second profile blocks a system call from a pod. So let's look at another demo. Hello, this time I will show the second audit gadget. I will use a different version of my second profile. This second profile contains the same list as the whole system called us before that are alone. But the default action will no longer be to return an error but to log them. So the audit log will be able to display them. So let's deploy this second profile. Now it should be deployed on my cluster. Okay, before deploying my workload I will start the inspector gadget. So you can see the stream of system calls being executed. So let's start this command inspector gadget. I will monitor the second demo name space and display the system calls on the return code and so on. Okay, so far there is nothing because I don't have another workload. So let's see the workload. The workload reference my second profile as I showed before. And let's deploy this and let's see if it is running correctly. It is already running but so far the only system calls that are executed are the one I wrote. So there is still nothing to display here. I will try to generate different network activity to see if something happened. So like before I will use port forward on some curl command. So port forward is running. So I can run curl and see if anything happened.  Okay, so now I can stop the network activity and I will try something different. I will exact into the port and see if something different happened. I will execute bash and bash should execute different kind of system call that were not done before. So now the audit can notice them. And if I use this command, M cannot, that should, although on this M cannot system call that is, that were not in the all list. So I see it in the audit log. Thank you. Okay, so that's it about set com. So let's move to another kind of security profiles. Those are Linus capabilities. And yeah, science Linus 2.2 privilege process doesn't have all the capabilities of the system. So those privilege operations were divided into different blocks, if I can say that way. So the idea there is, if there is a, if our process gets compromised, if a privilege process is compromised, it is only able to do some things on the system. So yeah, for instance, if we want to change, if a process needs to change the owner of a file, it has to be running under this caption capability. If it has to be buying at poor, less than 2,000 24, it has to have this net buying service capability and so on. So the idea there is to have, to limit the capabilities that are all process have. Of course, the list of capabilities in Linus is very long and actually some new are in the different releases of the kernel. In Kubernetes, the container runtime give us some capabilities by default and we can add or we can drop the capabilities that we want to give to our container by using this security contest. So this is an example in this case we are dropping all the capabilities, but we are only adding the capabilities that we need for our container. So yeah, the problem here is very similar to the previous one, how do we know the different capabilities that a given pod needs to run? And actually in this case it's actually more complicated because this is not only about system call but the capabilities are like an internal concept on the kernel. So sometimes those are used, sometimes those are not used depending on the kernel version that you are running on different configuration parameters of the kernel. So let me show you how you can generate, how you can understand the capabilities that a given application is using on a Kubernetes cluster. Hello, today I will deploy it to RSI and see what kind of capability it use. So first look at the original version of RSI in the specification, you see it use all the privileges and so far I have not installed it. So let me install it first. I will deploy the inspector gadget capability to RSI gadgets. I deploy it on the RSI system and M space, looking at the specific pod for RSI and looking at all those columns and I don't want any duplicate capabilities and so I specify the unique flag. Okay, so now let's start RSI and it will start the reason. Okay, it's already started and you can see at the bottom the capability it has exercised, setPK, Ptrace, and this admin, this resource. Okay, so now we can delete it and let's make some space. And now we'll have a prepare the new version of RSI. Instead of using privilege equal to, it use the following capabilities and you can see it will work in the same way as before. And so this gadgets allows me to find which kind of capabilities I can have in my pod spec. Okay, thank you. Okay, so yeah, that's it about the Linux capabilities and the last one that I want to cover today are network policies. This is just a clear view, I will go very fast on that. So network policies are a coordinated mechanism to limit how the pod can communicate with a network entity. Network entity is another pod, coordinated service or an external endpoint. Coordinated network policies operate above the IP and the port level, that's layer three, layer four. And we can use that to restrate the ingress or the egress traffic from a pod. This is an example of what it looks like a network policy resource. So we have the pod selector, which means the pods that the policy should apply to. This is where we define the policy. So here we are saying, okay, our pods should be able to communicate to other pods, other name spaces, other pods and name spaces or IP blocks if those are external endpoints. And in the part we define the layer four the layer four of our network policy that's the protocol and the part. So yeah, it was just a quick reminder of network policies. So how do we define coordinated network policies? Well, usually those should be created when we are designing our architecture. But yeah, as I was telling you at the beginning, this is not what happened in many cases. So in some cases you have to define the network policies after your application is already running. And yeah, in those cases using a policy advisor, using a tool that suggests you send network policies to be applied can be a motion. And yeah, let me show you very quickly how we can generate those by using the a network policy advisor from Inspector Gadget. Hello, I will demo the network policy advisor. So I will deploy a workload and observe the network traffic done by this workload and then generate network policies thanks to the network policy advisor from Inspector Gadget. So first I create this demo namespace and so far there is nothing in it. And I will deploy this Kubernetes manifest.tml. Before that, let's start the network policy advisor. Start it with this command. I use a kubectl-gadget-advice command to monitor the network activity in the demo namespace and redirect the log in this file. Okay, so far there is no recording because there is no network traffic. But once I deploy the manifest, at some point it will create some network activities. It will take some time to prepare all the deployment to be ready. So let's wait some minutes. After some minutes, my deployment is successfully deployed. So now I can stop this. And then let's ask Inspector Gadget to generate the network policies from this log. And then the network policy will be saved into this file. Then we can have a look at this file. And we see the first network policy generated is for the add service. We noticed some equals traffic to DNS. So power 53 UDP to the kub system, kub DNS service. And some increase service from the front end to some TCP port. And you can see other network policies for other services as well. Okay, and finally, how we do audit the network policies. So how we can know if our network policies open the packets from our application. So yeah, depending on the CNI provider that you are using, maybe you have that support implemented there. If you are using Celium, they have this audit mode through or this flat better. And when you enable that, the Celium doesn't force the network policy, but it will tell you when it eventually drop the packets there. If you are not using Celium, there are other tools that kind of the same at the IP table level. But yeah, in this case, you have to find a way to correlate the IP table rule with the network policy that is dropping the packets. So yeah, just to finish here, I want to discuss the limitation of this approach. So I will say the biggest limitation here is that our application has to generate all the events that it needs to work when we are monitoring that. So if there is, for instance, a given system call that our application needs, but is not used, why we record that to generate the second profile, the second profile is going to block that system call. So when the application is, we'll try to use that specific system call. When running, for instance, a production environment, that system call will be denied and our application won't work. So how can we ensure that our application is generating all the events that it requires in the real scenario? Well, maybe one option is to deploy our application in a production environment, send real traffic to that and monitor that there. Of course, there is some risk on running an application in production without security policies. Another option is to run some integration tests to try to go through the different parts of the code. So those are one of the options that we have, but actually this is like the biggest limitation and it's difficult to do. So there is no like, this is the way to do that. There are so many different options. And the second limitation, this is, I will say, limitation warning is that when we use this technique, we are assuming that our application is trust. So we are monitoring the application if there is any malicious activity going there, we are going to include that into our security profiles. So basically what it means is that we are going to allow the malicious activity in the security profiles that we are generating. So yeah, we have to be sure, I don't know how to do that to be honest, I don't have an answer, how to be sure that our application is trust, but we should be sure that this is why we use this approach. And yeah, just to finish, there was a, I will say similar talk this Monday in the Cloud EVPF negative day. That is also how to generate this security profile by using EVPF. But in this case, it was by the ISO Valentine. They were using Tetra for that. So if you are interested on that, maybe you can check the recording of that. It should be available on YouTube in some weeks. And yeah, that's it. So happy to take any questions that you can have. Thank you. Thanks for the great talk and not just great but comprehensive overview. My question is, it's all in the limitation about during the recording phase, you can create policies only what you saw in the recording phase, right? So what you haven't seen, you cannot define. Have you been looked into or researched how could you decide when to stop recording? I mean when to, or how to signal to end recording and you say, I saw enough of the application activity that now I can turn this into a policy? Yeah, that's a very good question. We haven't done that. So far, when to start, when to stop recording is totally controlled by the user. So you as an operator, you say, okay, I want to deploy my application, I will start recording, then you, I don't know, you wait for one hour, 24 hours depending on an application, and you say, let's stop. But yeah, maybe that will be something interesting to explore how to start and stop recording in an automated, smart way. Thank you. Thank you. When, if SETCOMP stops an application because it breaks security policy, what is the experience of either the APTA or the operations team? What do they see when that happens? Oh, sorry, I can't understand. Will you mind removing your mask for a bit, please? I can't understand what you're asking. Yeah, when SETCOMP takes some action on an application that's running, it breaks a policy. What is the experience for the operations team or the application that created that? Is that just an error, or what do they see to know that this app broke one of the policies? Yeah, so when the second profile is violated, depending on that, it could kill your application. So what you see is a pod that is crashing off because the application is being killed there. Does it say, like pods crash sometimes frequently? So does it say a particular reason why the pod crashed? You have to check the logs from the application. I mean, if you are using the audit second that I was showing you before, that will tell you what is going on. So that will tell you that a given second profile sent a kill signal to a given process, and it will give you information about the container, the pod, and so on. If you are not using that, the only information that you have available is the log of the application. So you will have something like the process was killed and that's it. Probably there is also something in the audit log from the operating system about that, but in that case, you will have to correlate the PID with the pod. So in that case, if you are running a lot of pods in a node, this is not easy to understand what is the pod that was killed. Right, yeah, exactly. Okay, this is not great. Okay, cool. Thank you. Thank you for the talk, it was really helpful. It looks like the tools that you showed are really helpful for developing the initial policies that you wanna enforce. My question is, as applications change over time, they may have requirements for new links capabilities. They may need to make new system calls. Do you have any thoughts on how you can maintain these policies over time such that it's not blocking application developers from making progress? Yeah, I will say that this is a rather difficult question. So the idea that we have with these tools is, okay, you don't have any policies. We suggest you what could be an initial version of those policies. So the idea is not to use these tools automatically. So the idea is you, as a developer, you use the tools and then you have to understand the output of those tools and you have to be sure that the things that are in the security profile make sense. I'm not totally sure that it will make sense to update those profiles automatically when the application changes, because what about if there is a malicious activity there and you are updating those profiles automatically to allow the malicious activity that is going there? How can I understand? Okay, yes, I updated the application, so I have to update the profile because my application is doing something similar. How can you be sure that that's the case and that this is not about that you are having some malicious activity there? So yeah, I mean, this is very complex to understand if you can update that automatically without human supervision there. I was pretty interested in the network policy generation and I noticed the network policies that get generated there, using pod selectors with labels to clearly identify the pod that is involved in the network policy. How intelligent is that network policy generation? Is it trying to create like the simplest possible network policy or how does it decide what labels goes into that? You have a point, very good question. Yeah, so the general logic that we have is that we capture the traffic. So the traffic that we capture has the, I mean source destination IP addresses and we enrich the data with Kubernetes related information. So we do a transformation like from IP to pod name, service and so on. And at that point we capture the labels of those pods, services and so on. And then we try to consolidate that to generate the network policies. Actually what we are doing right now is like try to be as granular as possible. So we are generating very small network policies. So actually if you are running so many different services you are going to have a very big generating network policy and this is what I was talking at the beginning. I mean, we have a limitation there because how can we consolidate those network policies and less network policies with more labels? This is something that we don't know yet how to do. So probably more work has to be done there in order to provide network policies that are consolidated and that are easier to understand for the users. So actually one very simple case is when there is a pod that is reaching an external host, I mean by IP. So we only generate a slash 32 rule. But what about if actually the pod is trying to access a app signlet. So how can we consolidate those into app signlet? This is something that we don't know how to do that and this is something that we require more research on that. Thank you. Thank you. Last one. The output that is used to create the network policy, that log file, is it human readable? Like, could it be used to audit the frequency of traffic and create a profile so that if you're trying to create a secure network policy based off of that output, you have information to use. Like, obviously as a security sys admin, you have more information about the expected traffic between services. So if you can use that, not actually the generated policy, but the log file to inform the network policy that you're creating. Like, is that possible? Is that a use case that's been considered? So do you mean only to capture the data about the network packets and then use that for? Yeah, like you've got that log file that is being used to create the network policy rather than just creating the network policy, use that log file to create a traffic profile that you can then use to inform the network policy? Yeah, actually, those are two tasks. I mean, capturing the traffic and reaching the traffic we coordinate this information is like the easy task. The difficult task is, okay, how do we convert that network traffic that we have captured into a network policy? Currently we have a very simple algorithm there that just tries to consolidate that. But yeah, there are many options that we haven't explored yet. Yeah, it just seems like it'd be very interesting to output a profile rather than a policy. So something you could run that advisor several times and compare the traffic over a period of time to see like, oh, we're getting hit by this request at this frequency, which is not something we expect and maybe is a malicious actor that we should look into. Yeah, that could be also an option. Also, thank you. Thank you, everybody. Thank you.


Gunsmoke. Brought to you by L&M, the modern cigarette that lets you get full exciting flavor through the modern miracle of the pure white miracle tip. Live modern. Smoke L&M. Around Dodge City and in the territory on west, there's just one way to handle the killers and the spoilers, and that's with the U.S. Marshall and the smell of Gunsmoke. Starring William Conrad, the transcribed story of the violence that moved west with young America, and the story of a man who moved with it. I'm that man, Matt Dillon, United States Marshall, the first man they look for and the last they want to meet. It's a chancey job, and it makes a man watchful, and a little lonely. The old doc ain't too far wrong. Well, he claims winter's coming early this year. He claims we'll have snow for the end of the month. He's just passing along Indian talk, just so they're all clean. Miss Cobb? Evening, ma'am. Do you declare Mr. Dillon that that ain't the most dilapidated old buggy I've ever seen any worse? Ah, Jezra believes in getting full value out of things. Make him do, as he calls it. Yeah, let's get something to eat, huh? Well, now, eating is something I get full value out of. I know. Well, let's sit over there for the winter. That team of Jezras could do with some eating. The way their bones are sticking out. Ah, Jezra figures fat on a horse is a sign that grains be in wasted. Well, you're short of me, Marshall. Okay. I figure the same thing about women. Miss Cobb's bones are sticking out something, too. He probably works her half to death. Has to run a farm that size without any hard help. I sure do keep to their selves. Ah, people just don't take to Jezra much. He's got a pretty cool way about him. That is hard to say what she's like. That poor lady don't even open her mouth less than he tells her she can't. Plain mouth. See, that's what she's like. What do you figure to eat, gents? Well, what do you got? A stewed beef and boiled greens. Well, I guess that's what we'll eat. All right, I'll bring it right out. Boiled greens. Boiled gyms and we'll eat more. What? That was up in the street. Yeah, come on, Chester. Sit, Matt, it's spooky. Lots of trouble, Chester. Marshall, you've allowed this town to become a stink of iniquity. Now, why did it supple her? How's that so? A town of painted Jezebel scoundrels and murdering assassins. The names should be changed to Sodom or Gomorrah. You all right, Miss Cotton? Somebody tried to kill us, Marshall. They were standing right over there in the shadows. Minerva, you'll kind of remember a woman's place. Marshall asked. Silence. I'm sorry, Jezebel. Did you get a look at him? We did not. The coward struck in the darkness. It was only a miracle we escaped. Any of the rest of you see who fired those shots? You're the law here, Marshall. I demand an accountant for this outrageous assault. Jezebel, do you know anybody who might think they got a reason to kill you? I never had an enemy in my life. Well, it looks like you got one now. Tell yourself of old-fashioned ideas.  Smoke an L&M. Why are more people changing to L&M than to any other cigarette? Because only L&M lets you enjoy full, exciting flavor through the pure white miracle tip. L&M draws easy. Taste richer. Smokes cleaner. So, free up. Freshen up your taste. Get full, exciting flavor. Live modern. Smoke L&M. Make your day your big red letter day and start to live the modern way. Live, live, live modern. It's America's fastest growing cigarette. You've got to make this prescription look real authentic. Oh, what do you mean? These are fooling pills. I've got nothing in them but sugar and some chalk and a little gum arabi to bind them. So that's the kind of medicine folks get from you. Oh, no. It's the kind of medicine Mrs. Prudlin gets. As long as she figures they help her. That's all that's necessary, at least they don't do any harm. Nothing wrong with her, anyhow, except in her mind. Oh, there you are. Well, let that sit a while and dry. Oh, so you found out yet who fired those shots at old Jezrecombe? No, Doc. Not a hint. That's a funny thing, man. I don't think I know a soul and dodge that likes old Jezrecombe. Yeah, no, Doc. Nobody really hates him either. They just stay clear of him as far as they can. Yeah, that's just it. He just isn't the kind of a man to rouse up strong feelings in anybody. Good or bad. He must have roused some and whoever tried to kill him or kill her, I don't even know for sure which one they were after, him or his wife. Are you busy, Doc? Huh? Well, come right in, Kitty. Hello, Kitty. Matt, what are you doing here? Well, he's hiding from Jezrecombe. What? Don't mind, Doc. I'll step outside, Kitty. Oh, no, no, don't go, Matt. I just wanted to get something for a headache, Doc. Oh, it's too bad. Dip yourself some water there. Here are some pills I just made up, Kitty. Oh, not, Doc. Formulate him straight from Boston. Well, then take care of this headache. They're medical pills. That's exactly what they are. Here you are. You swallow a couple of those. Thanks, Doc. I personally guarantee those to stop the world's worst headache in one minute flat. Yeah, they've done wonders for Mrs. Prudlin. Mrs. Prudlin? Pay no attention to him, Kitty. He's all upset over that shooting. Well, I could name a few people who aren't upset over it. Oh, which people? Some of the girls at the long branch. He drops in about three times a year. Jezrecombe at the long branch? Old self-righteous Jezre. And every time it means trouble, he always drinks too much. He bothers everybody. What bothers him, huh? I try and to reform him. And it's the girls who always get the worst of it. He... now, who'd have thought it? He calls them painted Jezrebells. He says he means the curing of their transgressions. Of course, the only cure, he seems to know, is to grab a cane and beat the daylights out of any of them he can get his hands on. I didn't know Jezre was that bad. You asked some of the girls, Matt. Daisy or Billy Bell. Yeah, I will, Kitty. Doc, could I have some of those pills to take with me? Oh, look, Kitty, Doc, would just point... They're real good, Doc. My headache's all gone. What'd you start to say, Matt? Uh, nothing, Kitty. Nothing. The eight hours, Marshall, and you've accomplished nothing. I demand legal action. You haven't helped things, any, by lying to me, Jezre. Lying to you?  Ha-ha, dance, old girl. Maybe soon. But I sure hate to have them looking at me over a set of gun sights. Abandoned, painting their faces, devoting in public to the devil's music.  Wasn't a woman who fired them shots? A woman wouldn't have to fire them, Jezre. She could get a man to do it for her.  I don't have any proof of anybody plotting anything. You're in league with the adversary, Marshall. You're aiding and abetting the forces of evil. Let me tell you something, Jezre. I'll aiding abet anybody's right to live his own life according to his own lights. As long as he's within the law. Yeah, what is it? Could I see you a minute? Yeah. Yeah, what's the matter, Chester? Well, it's Miss Kitty. She wants you over at the long branch, right away. Oh, what for? Well, see, there's a fellow there that's been drinking real heavy, Miss Dillon, talking to my... Oh?  Self of old-fashioned ideas, freshen up your taste. Smoke and L&M! Today, all over the country, more people are changing to L&M than to any other cigarette. And it's all because only L&M gives you full, exciting flavor through the pure white miracle tip. L&M draws these tastes richer. Smokes cleaner. So, free up, freshen up your taste. Live modern, change to L&M. Make today your big red letter day and start to live the modern way. Live, live, live modern. It's America's fastest growing cigarette. He's playing poker over here with a house dealer and a couple of tall boys. Have you been in here before? Yeah, just during the last week. He's a drifter, I guess. Goes by the name of Puggy Rado. There. That's him, Matt, on the far side of the table. It looks like a saddle bump. He was making out real braggy for a while. Fastest gun in Texas, that kind of talk. But I guess he knew he'd gone too far when he said that about Jezre Cobb.  All right, Kitty, I'll try to get him away from that table. Wait, Matt, I think he's coming over here. I reckon you're the marshal. That's right, yeah. I reckon she heard what I said. Sent for you. That's just what I figured she was doing. Don't you follow me! Hold it, hold it, marshal. And move toward that gun. And I'll put a bullet right in her back.  Check it easy, Kitty. You tell her, marshal. Tell her me and her leave and now. And if anybody lays a hand on a gun, there's gonna be a pretty corpse on the floor. I told you to let go!  Drop that gun, Redo! Not a chance! I'm all right. Thanks, Matt.  There was no time for anything else. Marshal, was that the man? Yeah, it looks that way, Jezre.  Well, that's the fella who... Who what, Jezre? Well, he stopped in at my place last week, and he gave me a hand out. Did you say $300? That's right. Well, I've got to be getting on home, marshal. I've got to stop to attend to it. I've got a lot of things to attend to. Well, for evermore, come over here, Mr. Sudden. I don't know, Jezre, but I think I can guess. There's a light on them now. I knew you were such a poor built house, ain't it? Yeah, it sure is, Miss Cobb. All right. I do, Marshal. Chester. Miss Cobb. Your husband got home yet, Miss Cobb? He's out there in the barn.  Yes, we will, thank you. As a matter of fact, Miss Cobb, you're the one we wrote out here to see. That's right, kind of you, Marshal. Now, you two sat you down there at the table. Just this minute took a wild plum pie out of the oven. Well, no, no, thank you, Miss Cobb. I don't often get to feed collars. You ain't going to deprive me of the chance. Maybe we could just taste it a little bad, Mr. Jones. I always had the impression that you were opposed to visitors. Miss Cobb? Oh, I love to have folks come. Jezra ain't never been one to encourage it. Oh, I see. The righteous must turn their faces from the world. For the world is the cradle of sin. That's what Jezra always said. There you are, piping hot. Well, gracious, that smells good. Well, eat hearty. There's plenty more. It'll just go to waste. Jezra's never cared too much for plum pie. Miss Cobb, how long have you and Jezra been married? Twenty-seven years, Marshal. Twenty-seven years. Uh-huh. I know with those years, Miss Cobb, how many times did he beat you? Hundreds of times. For my transgressions, he told me. He used to read me from the good book that a husband's got a right to do that. I ain't never learned to read myself. But last month, I asked Reverend Blouse, and he said there weren't nothing like that in the good book. Jezra's got his own version. He lied to me. That's what he done. And if he'd lie about that, then, well, I reckon you know, don't you, Marshal? Yes, ma'am.  Yes, ma'am, I guessed it. Jezra did, too. Well, I figured he did the way he acted in town. He'd come home and told me about you having to shoot the man. Then he asked me for our savings. And I got it for him. He sat down here and counted it. And when he seen it was the same as that fella had been talking about. $300. Then he knowed for sure. What did he say? Nothing much. He just sat here a while, smiling at me kind of in that cold way of his. His glory smile, I always called it. Then he got up and went out to the barn. And, of course, I knowed what he was going for. What do you mean? He was aiming to fetch a hickory stave. He always keeps some out there to mend fences. I didn't clear it, Mr. Dillon.  I think, ma'am, maybe I better go have a little talk with Jezra. Won't be no use, Marshal. Just won't be no use. I think it will, Marshal. Listen to me, Marshal. You're wasting your time. I'm trying to tell you. When he went to fetch that stave, I knowed what he was aiming to do. And I followed him out to the barn. Yes. I stood real close, Marshal, so I wouldn't miss. And I pulled the trigger four times. I put the gun there in the cupboard. I figured you'd be wanted. Jezra ain't never gonna beat me no more. And you sit right back down there and it's your pie. In a moment, our star, William Conrad. If your daily routine never varies, chances are you're heading for a great big case of monotony. Everybody needs a break once in a while, and sometimes a vacation just once or twice a year isn't enough. But a break once a week would more than fill the bill. What could you do with that leisure time? Here's the answer to that. You can vary your routine with a fascinating and vital pursuit, spotting planes for America's Ground Observer Corps. It's exciting, it's interesting, and just two hours a week of your spare time is all that's needed to keep up the 24-hour schedule of the GOC. Men and women from Teenage Up can help the Air Corps cover the blind spots in our radar system by volunteering for the Ground Observer Corps, a civilian component of the Air Defense Command. You'll be trained and supervised by officers and airmen of the Air Defense Command to spot planes in your area. Find out from your local civilian defense office how you can be a civilian Ground Observer. This has been a public service message by CBS Radio. And now, William Conrad. You know, when a Batman riding a good horse came into Dodge looking for trouble, a likely as not, he got a bullet hole placed in him. And the horse went to the man who did it. And that was the West. Gun Smoke. Produced and directed by Norman McDonald, stars William Conrad as Matt Dillon, U.S. Marshal. The script was specially written for Gun Smoke by Les Crutchfield, with editorial supervision by John Meston. The music was composed and conducted by Rex Corey. Sound patterns by Ray Kemper and Bill James. Featured in the cast were Virginia Gregg, Ralph Moody, Don Diamond and John Daener. Harley Bear is Chester, Howard McNeer is Doc, and Georgia Ellis is Kitty. Join us again next week for another specially transcribed story on Gun Smoke.


The daily Iquot today ended house arrest of Mr. Gautam Nalakha, one of the five men who were languishing in house arrest since October 28, so the daily Iquot has set aside the CMM Quotes order Gautam is with us to talk on the issue of Gautam and just to give you a bite tarik today but yes I'm happy personally that I've been free but it's a long battle ahead because my fellow folk used to be in jail so my happiness is not would be full only when everybody else gets their freedom how was this period of house arrest we'll talk at length later but in short it was put to good use I caught up on many things Sabha and I on caught up on many things that we have not been able to do from listening to music that we long which was some time ago that we hadn't heard read books saw a lot of films read a lot and chatted and had long conversations with each other so it was put to good use I have no