Cool. Hi everyone, thanks for joining me this late in the day for my talk. My name is Niels Bentelen and I'll be talking to you about enforcing data quality in data processing and machine learning pipelines with Flight and Pandera. Before I hop in, just a little bit about myself. My background is in biology, dance and public health, but I kind of did the little pivot and now I'm a machine learning engineer at Union AI, which is a Seattle-based start-up. One of my, as part of my roles at the company, I'm one of the FlightKit maintainers. FlightKit is an open source package that's part of the flight ecosystem. I'll get into that in a little bit. I'm also the author and maintainer of Pandera, which is also the subject of this talk. It's an independent project that I've been developing over the past two, three years. In my free time, I like to build machine learning and data science tools and also training whimsical models just for fun. The outline of this talk is as follows. I'll start with a sort of a problem statement phrased more as a question. First out of curiosity, who here is just a show of hands familiar with data frames as a general kind of data structure? Cool.  I'll then get into Flight as a type safe data pipelining tool and Pandera as a statistical data testing tool. In combination, I'll show you kind of a high-level architecture for an example pipeline for training a model, putting both of the two tools together, show a little demo of this example pipeline, and give you some takeaways and some next steps both for you and me. So what's in a data frame? Since everyone here knows what it is, it's basically a table. Generally it's two-dimensional, but it can really be n-dimensional, and it's a table generally with human-readable or maybe sometimes non-human-readable labels for the axes of the data structure. And the intersection of, in this case, columns and rows gives you a value, each value depending on the library. It could be columnar or row-wise. You can think of this as a table that you're all familiar with. So just to ground this conversation, my background is in data science and machine learning, and I do most of that in Python. And so in this talk, I'll be mostly focusing on Pandas data frames as this in-memory data structure with a lot of little niceties to make it really easy to do statistics, information, plotting, all that stuff, given that you have kind of data that fits on whatever machine you're using. And in recent years, there have been several projects that have been trying to scale up data frames. And these projects are ones that try to emulate or at least reproduce the Pandas API just to make it easy for new users to onboard and, you know, minimally change their code to scale to multiple machines, maybe GPUs. So these projects are, as you can see, Dask, Moden, Koalas, Polars, Rapids, and there are probably many more that I'm not mentioning here. There are also certain projects that don't conform to the Pandas API, but are nevertheless offer a data frame API where, you know, you can do similar transformations at scale. So just to sketch out the problem statement a little bit more, just take a look at this innocent piece of transformation code. Super simple. It's a function called transform data. It takes in this poorly named thing called DF, data frame. And since this is Python land, you know, anything can be anything. And so here we're assuming, though, that DF is a data frame, and the transformation is super simple. You know, you're assigning column C to the sum of A and B, so that's an element-wise operation. And then you're doubling C to give you D, and then you're returning the data frame. So the contents of this data frame are integers, you know, the add and multiply operations are defined, and we get, you know, the expected output. So C is literally A plus B, and D is literally the double of C. And all is well. This is like the intended, you know, thing that I wrote this code for. But again, in Python land, if the contents of data frame are strings, well, we don't get any runtime errors, you know, the code happily executes, and this is because the addition and multiplication operators are also defined for strings. And this might be fine, but, you know, in this toy example, I didn't intend for strings to be transformed in this way, but this is a completely legal thing to do, at least in pandas. Now if the contents of A are integers and B are strings, then we run into a type error at runtime. So we execute it, and the addition operation is not defined for operating on, you know, ints on the left-hand side and strings on the right-hand side. So here we run into trouble, and the middle case really kind of highlights the things that I've been burned by when doing, you know, ad hoc data analyses or even code that I write that might be going into production at some point. The middle indicates, you know, errors that run silently, and I don't know until way after, like, let's say, training a model or doing a statistical analysis, that the data is not what I expected it to be. So the central question of this talk is how might we enforce the quality of data as they flow through data processing pipelines, especially when they involve data frames? And data frames, again, in the pandas case, they're kind of schemaless. The dynamic nature of Python and pandas makes it such that I can assign and kind of mutate things at will, and this can really bite you, you know, at some point if you've ever used these libraries. So the first part of the kind of the solution or the way that we can sort of avoid these types of silent errors is through flight. I'll get into Pandare a little bit later, but as I said, flight is a data processing and ML orchestration platform. You can see here, you can pip install it. Flight kit is the Python SDK for flight. I won't get into all the nuances and all the features of flight, but I'll just say that flight is like the stratosphere. It has many layers. It can mean many things to many different people. So you know, if you're more on the infrastructure side, you might think of it as a Kubernetes first workflow orchestration engine. Maybe if you're like a data engineer, you might think of it as an automated data lineage tracking platform, which it certainly supports. But as a machine learning engineer who trains models and creates the data sets that might supply the training data for those models, I think of it as a type safe data pipeline language. And I think this is very powerful because you can avoid many kinds of errors that one of which I highlighted earlier. So just a quick tour of flight, I'll be referring to these simple toy examples, the transform data function here that's just A plus B and then C times 2. And then here I'm just introducing another function that takes the output of that and then aggregates the data by getting the mean of each of those columns. But the first thing you do with flight is you can write tasks. And tasks are sort of the main workhorse of the flight platform where you can write isolated pieces of data processing. Under the hood there's a lot of stuff going on, but you can kind of think of everything under each app task decorated function as isolated within a container, a Docker container that is. The second thing you can do is compose workflows. And workflows are tasks connected together in sequence or with some conditional logic. I won't get into all of the things you can do with flight and composing workflows. But in this simple example, I'm calling transform data on some input data frame to give you transformed DF and then aggregating that and returning some output. So you can see here that I'm kind of using Python's type annotation here to just kind of indicate what the input and output types are. But that's not really, I mean, it's informative in that, you know, you expect to input and output a data frame, but it doesn't really give you more than that. I'll get to this later. But for now, I'll explain more a little bit about what flight provides. So with flight, you can develop locally. So all the code that you saw earlier, you can run locally and it executes as regular Python for the most part. Then you can deploy to scale. And as I said earlier, I won't get into the deployment and the back end aspects of flight. I'll really just focus on what the local user experience is before you try to scale things. So as you see here, I'm calling pipeline this workflow that I defined earlier. And now I get this result. I get ABCD with the main, median, and standard deviation of each of those columns.  So flight locally and in the back end caches task outputs. As I said earlier, it does track data lineage. So you can see as your data passes through your workflow and tasks, what the input state of the inputs and outputs are. You can also recover and resume partially completed workflows and integrates with an ecosystem of data tools that I won't really get into too much. And just to highlight again, what I really value it for is that I can build type safe DAGs. And so what I'm doing as an author, as a developer, as a data scientist is this tool helps me to think about the types so that in the future, I don't forget what the types are or my collaborators can see exactly what the intent of the code is. And that's very valuable. There's kind of maybe this trend to adopt typing and type annotations in Python more. And as that happens, I think many people in my data science machine learning circles will see the value in this. And really returning back to the rationale and the reasoning is high-quality and clean data is critical for data analysis and modeling. So if you get certain, if your expectations about your training data or your analysis data are broken and it passes through silently like a numerical column say having negative values when only positive values are valid. Then if you're making business decisions or you're deploying a model off of training data with these invalid things, then you're kind of wasting time training models. And it's also challenging to isolate and trace the origin of bugs in these kinds of analyses and models. You have to kind of go backtrack all the way potentially to the very beginning of your pipeline. So again, flight is a strongly typed data pipeline framework. So here I'm trying to define a flight task, but I've ignored to provide a type hint for the DF data type here. Oops. And here flight will complain. So type empty is not supported in FlightKit. So you have to provide something. It could be just pd.dataframe, but at the very least you're saying what the expected thing is of what DF is. And again, composing tasks with mismatching type signatures, which is something that has also burned me many times. So here I'm taking transform data, which has no return signature. And I'm trying to hook it up to aggregate data. And here type flight will complain. So it says cannot pass output from task transform data that produces no outputs to a downstream task. And this is another thing that it's great, this is a great feature to have because now that you have type information, you basically have function types for your functions. And your functions now can be analyzed to see what is a valid set of operations. So you can assess your workflow for validity just on the basis of the allowed input and output types. So when I said pd.dataframe, that's great. And everything, it doesn't really give you more information. So flight does have a built-in type for a data frame. And this is called flight schema that I'm importing here in this code snippet. And you can express the columns and then the types of those columns. So here you can just read for yourself, a equals int, b equals int. And then the aggregate output type is, or they're all floats since we're collecting aggregate statistics, the ints will become floats when you aggregate them. But what if I want to validate data frame properties beyond data types? And this is where I'm going to get into Pandera, which is a statistical typing and data testing tool. As you can see, just install Pandera and you can get started. Pandera adds guardrails to your pipeline. So if your pipeline is this suspension bridge that looks kind of dangerous, and your data frames are these people kind of traversing this dangerous thing, Pandera are the guardrails that help you know when someone's fallen off, hopefully that doesn't happen. But it helps you keep your data frames as you expect them as they flow through the pipeline. So I mentioned two terms, statistical typing and data testing, and you might be familiar with these, but I just want to kind of flesh them out for you. Statistical typing, it's something, I don't want to say I coined it, but it's something that I've seen other terms for, but I think statistical typing is a good phrase for it because it captures the concept really well, and it's specifying the properties of collections of data points. And so as opposed to if you have a single data point, say it's a row on a table and it has certain fields, you can do a bunch of really useful checks on it, and this is very valuable still. You can check the data types, value ranges, if you know them ahead of time, allowable values for free text fields or string fields, you can do some kind of regic string matching, you can check whether the nullability assumption of the field is upheld or not. But you can do more when you have a collection of data points. First of all, if you have a tool like Pandas or Dask, you can apply atomic checks at scale. All the checks I mentioned in the previous slide, you can, you know, if you have some vectorized implementations of them in some underlying framework, you can apply all of those on potentially very large data sets. But then you can also do things that you can't do with atomic validations. So you can check for uniqueness in a column. You can check for monotonicity, you know, is a value increasing or decreasing all the time. And you can check for statistical descriptive statistics and check, you know, maybe they're within some certain range that you expect. Maybe a column is a normal distribution, a standardized normal distribution, so you'd expect the mean to be roughly around zero. You can do other statistical distributions and also like fractional checks of those atomic validations, like you say 90% of my data are not null. So the second term I mentioned that Pandera offers is data testing. And this is a term I've also kind of seen in various data science and machine learning blogs. I'm not sure how widely used the term is, but I kind of see it as validating not only the actual data but also the functions that produce that data. So maybe in production in the real world, you're getting, you're grabbing data from the real world, some raw data, you're transforming it, and then you have some transform data that you want to use for some downstream process, and you want to apply validations on that, that transform data. But then in continuous integration tests, you might have some test cases to see, you know, what are some valid code paths for this transform function, what are some invalid code paths for this transformation, transform function. And essentially by providing these test cases and applying validations on the output, you're making assertions about the function and what it is doing to that those test cases. So maybe it raises an error, you know, so you can maybe try testing that. So Pandera in practice involves encoding assumptions about data frames as schema types. You can extend schema types with custom validation rules really easily. Easily integrate data frame types into your pipelines. Then get informative errors if something goes wrong. And unit testing pipelines, you can unit test pipelines with hypothesis strategies. I'll go into each of these aspects in a little bit more detail. So as you can see here, we have the types that you saw with the schema type. The syntax is slightly different, but if you're familiar with data classes or Pydantic or that type of syntax for declaring Python types, this might look a little familiar to you. So the transform input has a column A and B. Both of them are integers. You kind of have to use these generic types. The syntax will change as the library evolves, but right now you have to specify the series of integers. A few things to note here is that Python's inheritance semantics work out of the box. So output inherits from transform input, as you can see. And so all the properties you define for transform input also apply to transform output. This is convenient to reduce redundancy, kind of make your type definitions a little bit more concise. Since Pandera primarily is a pandas data frame validation tool, the scope is slowly expanding as the adoption expands as well. But pandas has this notion of indexes, which are basically keys to the rows of your pandas data frame. And finally, it supports regex column matching. So here I define this property called columns in aggregate output. But then I can provide an alias. And usually this alias points to the actual column name. But if I wanted to turn it into a regex pattern, I then provide regex equals true. And this validation rule, which just says these columns should be floats, are applied to whatever matches the alias. Extending Pandera's built-in validation checks is super easy. So it's literally just a method. And you just reference the property that you want to validate. And the method signature, it's a class method. So the first argument is class, CLS. And the second argument is series, which is the column that you're trying to validate, in this case, A.  Since this uses Pandera as the underlying validation engine, so to speak, you can go crazy. I mean, as long as you have the series, there's a global check as well. That's a data frame check. And so then you have access to the entire data frame. And so that's kind of like a way to encode assumptions. Maybe you want to condition a rule on some other column. So this gives you a really flexible way to define validation checks that are custom to your case. Just to highlight, they're built-in checks. So this checks whether each of these columns are greater or equal to 0. And then, as I said, custom checks are just methods. You can easily integrate with your pipeline via a function decorator. So this works both with methods and just regular functions. If you decorate your function with check types decorator, at runtime, it will apply the validation rules to both the inputs and outputs that are annotated with the syntax of this data frame generic supplied with the schema type. So one of the things I really wanted to get right with this library is to get informative inputs, because I had used similar libraries in the past. And when there's an error, it might not be super informative or maybe it's a data structure that is custom to that library that's like a list of dictionaries or something like that. So in this case, I wanted to design something at the birth of this library to just be like you're working with pandas. So the errors in the failure cases should be really easy to identify. And when you can, give the failure cases as a pandas data structure so you can further manipulate and inspect it. So in this case, if a column has unexpected data types, it will complain. So in this case, the validation rule applies to the entire column. So in this case, it just says series A is expected to be an int, but I got an object. It'll also complain if a column is not present in the data frame, an obvious feature to have. And if any of the assumptions in your specified schema are violated, then in this case, the presence of negative numbers and odd numbers are in this data frame, then, as I said earlier, you basically can inspect the error and the dot failure cases property gives you access to all the failure cases in detail. So it gives you the specific failure case. It gives you the check that caused the error. It gives you the index in the data frame. So quite neat. Cool. And then one of the really kind of unique parts of pandera is its integration with the hypothesis library. So hypothesis in other languages might be called quick check, but it's kind of the Google term is property based testing. And so instead of manually hand crafting test cases to test unit test your functions, property based testing allows you to here specify the types in your test case. And the hypothesis will generate data according to that type specification, basically the contract. And then you can feed that to your function, in this case, the add function. And then you can make assertions about the result. So in this case, a really simple example, adding two numbers, and then you just test whether the result is as you expect. So just to reiterate, the problem of manually written test cases are subject to happy path bias. So I don't know about you. But for me, I like testing. But I can get lazy and just kind of test the happy path and maybe test a few of the error cases. But here, hypothesis will really scour the entire domain of the contract that you've specified in this at given decorator. And it'll easily find bugs for you. And so the way hypothesis puts it is it finds cases that falsify your assumptions about the test. So this helps you further refine both your tests and then the implementation that you're testing. So Pandera hooks into this functionality. And so here, instead of hand crafting data frames, which can be quite tedious if you've ever done it, I don't recommend it. It's probably not good for your health. But here you just call this .strategy method with the size that you want it to produce, the size of the data frame you want it to produce.  So in this code snippet, I've deleted the function body and I'm just returning the original data in my transform data function. And here, obviously, it immediately finds a failure case. Like probably the first thing it generated didn't have columns C and D, which it expected. So here you can find the error. You can see what was wrong. And then you can go back to your source code and fix it. One thing I wanted to call out here in the Pandera project is the limitation is that it operates on Pandas data frames. And so that comes with the limitations of Pandas itself. And so if you want to validate very large data frames or data sets, it's currently really not possible. You might come up with a solution that distributes it with some other framework. So for example, fugue allows for this. So you can apply the schema to the partitions of your distributed data frame and scale that way. Coming in at the next release of Pandera is a support for Moden and Koalas. That'll be, it's a work in progress. We'll see where that goes. But this is a way to apply exactly the same schema specification to these other data frame libraries. OK, so we'll put it all together. And to build robust data pipelines, I'll just show you the only meme of my presentation. This is like the epic handshake meme. Flight in Pandera. And to do that, I'll train a very simple model with sklearn on the UCI heart disease data set. This data set contains 13 predictor variables and then one target variable. The target variable being one if there's disease, observed disease, and zero if there's none. And you can see the other attributes and their various distributions. I did a little bit of homework before this presentation, obviously. So I came up with a schema that I'll show you in a second. High-level architecture is pretty much like a cookie cutter machine learning training pipeline. You have a process that fetches raw data from some source. It might cache it or it might store it locally somewhere. So you don't have to keep fetching it. Then you parse the data. You do some kind of processing to get it machine learning ready. And then you split it up into a training and test set. You train the model on the training set. You get a model back. Then you evaluate the model in the test set. And you get some metrics that you care about. OK, so now I'll show you Pandera in flight. If you're interested, there's a Google collab notebook here I'm going to pop in. Hopefully this doesn't break the presentation. So I'll just kind of go step by step through this. I won't go through every single aspect of the code. But just to give you a sense, hopefully the demo gods will smile on me today. But traditionally, you might document your features or your variables just in some kind of markdown or some kind of thing where you write down each variable and then what they mean if it's some kind of non-human readable code. But in this case, I can sort of encode what the data is expected to be straight on my schema type, Pandera schema model. And we're working to make this even better, so to add a title or a description to each of these fields. But for now, you can get a sense of what the raw data is expected to be. And this is a kind of artifact that you can use to enforce the assumptions about the data. So just looking at age, being an integer, and providing some kind of reasonable bounds on it, and doing the same for the rest of these variables. Typically, what you want to do is you kind of want to mess around with the data, visualize some stuff about it. But for the sake of time, I'm just going to go through and define this fetch raw data function. So what it does is it grabs the data using Pandas read CSV and fetches it from this URL of somewhat clean, but still needs to be preprocessed data. I have a note here seeing a demo. Remove these lines to see what happens. So this is actually kind of a case study in data that isn't as you expected. And you see I've annotated the output here with the raw data data frame. Let's see what happens. Okay, so invalid literal for int with base 10. So there's a string here, 0.0, that Python cannot coerce into an integer. So okay, this literally happened as I was writing this demo. So first I was like, okay, let me convert these two columns. So this CA column, but also this Thal column had this problem. So I'll just tell you that. Oh, okay. So there's this question mark in what is it? One of these columns. Oh, yeah, the error message is not super informative, but just trust me when I say that the question marks are in these CA and Thal columns. So let me do this. Okay, so I'm trying to convert a null value to int, which in pandas is not allowed. So I'm just gonna drop those rows since I can't really train on those data points anyway. And okay, cool. So based on what I know about what is defined as valid data, this function passes. And this kind of was kind of like a clean, very clean narrative just to show you here, but it's a lot more iterative than what I just showed. Typically, you want, like as a data engineer or data scientist, you kind of build domain knowledge about the valid domain and ranges of your various variables in your dataset. So moving on, let's look at the parse raw data implementation. So in this case, all I'm doing, if you notice, this target has values that are not just zero and one. So actually, the raw data had some more granularity to it. So actually, it had values from zero to four. And those have some meaning in the original context, but in this case, one through four were some varying degrees of heart disease onset. But in the case of this machine learning task, we're just binarizing it. So now we express, we write down this parse data that inherits from raw data, but overrides the target to be just between zero and one. And just to scroll back up here before it was between zero and four. So I'm overriding this value. I'm here transforming the data so that the target is actually binary. I'm not gonna do what this demo prompt here. Suggests, I mean, okay, fine, maybe I will. So if I just return this, again, as you saw earlier, it's gonna complain. And it here shows a bunch of the first couple of failure cases, and that makes it really clear. Oh, there are values that are not zero and one in the target column. Excellent. Just a little aside, for machine learning, you kind of want the labels to be balanced. In this case, they're balanced-ish, so I'm not gonna do anything else on the modeling side, except for training the model. The next thing we do is we split the data. I'm just gonna run this first. So all we're doing here is splitting the training data into a training set and a test set, specifying some test size here. And here you'll notice a feature of Flight, which is since you need to express all the types of your inputs and outputs, you can use the name tuple if you want to output multiple things. In this case, I'm outputting, I'm creating this name tuple of the training set, which is in test set, which both have the same type. So since they're just splits, you could define different schemas for the training and test set, but in this case, that's not really necessary. And the second to the last step is to actually train the model. We have a helper function here that just grabs features and then targets separately. Notice here within that the task, like this get features and target is not a flight task. And so here it's just Python code. And so you don't necessarily have to type annotate it. And here I'm just training a random forest classifier on the data. I have this really nice convenient job lib serialized file type. And so I'm gonna train this model. And it outputs a model file and under the hood flight determines where this output is gonna go, but just to prove to you it's a model, I'm just gonna load it up here again and you can see that it's a random forest classifier. And the last step is to evaluate the model. So the input is the model we just trained and then the second input is a test set. And again, here you'll notice I'm just executing this Python code here just on this collab notebook interactively. Okay, cool, we're just looking at accuracy. It's probably not the best metric to use in this case, but it was just a demo. So accuracy on the test set is 82%. I don't know whether that's good or bad.  And then finally, here's the pipeline that composes all of these tasks that we just defined and puts them all together. The inputs of this pipeline are some random states, random seeds for both the data set splitting and the model initialization. And this will run the entire pipeline. You can see that it's deterministic because of the seeds. So I'm getting the same accuracy each time. If I were to change it, let's see. I get a different accuracy because it's a different model. It's a different train test split which results in a different model and also a different test split. And then finally, I'm crossing my fingers to hope this works, but this is sort of like a mock. So you can imagine this code snippet being in your PyTest or unit test suite. Here I'm creating a positive examples and negative example schema. This is a little bit hacky, but this is a way to ensure that there's a data set with at least kind of balanced labels. In the future, there might be ways of using a single schema with a validation rule that says the labels must be within some distribution. But all this does is this train test, I test my training model. So here this, before I would have to like handcraft a data set with all the columns, but in this case, all I do is use hypothesis and it'll generate the data for me. And I can at least test that my train model function will run and I can generate predictions from it that are within the domain that I expect which is predictions of zero and one. Okay, so let's run this. Really the best demo would be to show you an error, but this is gonna take some time since I'm also testing different random states. So I'm looking at different, the hypothesis routine is looking through different random states, it's looking through different data frame types and hopefully maybe we'll return to this. But tests will complete. Okay, are you done? No, it's fine, trust me, it'll work. So that's really it. Using both these tools, you can have type safe data pipelines and be more confident in the correctness of your code. So to summarize, flight is a flexible type safe DAG, but it's more framework for orchestrating data processing and machine learning pipelines. Pandera provides an intuitive interface for specifying complex data frame types which includes statistical tests. And used together, data scientists and ML practitioners can be more confident about their code and when data quality checks are violated at runtime. Next steps for you, if you're interested, you can give flight and Pandera a try today. Next steps for the flight union and team and myself. On the flight kit side is, continue streamlining the user experience, add more integrations for libraries that you might wanna use in your data processing ML workflows, supporting additional data frame types like Dask, Moden and Koala's data frames, perhaps. Improving the flight kit Pandera plugin, so that's the thing that kinda glues them both together to support improved error reporting and visualization. And on the Pandera roadmap, as I mentioned earlier, to scale up data validation. And in the process of doing that, making it an extensible API for validating non-Pandas compliant data frame libraries. So there are many out there where the abstractions are still not quite there, but in theory, you could separate the schema specification from the validation engine, the thing that actually does the compute. And so that, once that's fleshed out, you can support even like lists of dictionaries of JSON objects.  And then finally, a pedantic and fast API integration. These are two kind of popular libraries that would, I think, benefit from Pandera integration. Cool. So there are gonna be two more talks at this conference relating to flight. The first one, you can just see the information here, an integration between flight and feast, and then integration between flight, Spark and HauerVot. And that's my talk, thank you for your attention. I'll be around for questions. Just to call out the flight and Pandera repos are over here. We are doing Hacktoberfest 2021, so if you're interested in participating, you should head over to the flight slash, or flight dash or dot slack, Hacktoberfest 2021 channel and we'll help onboard you and see if you're interested in contributing. Thanks so much.


Well this is very timely of course, and under the current EU President, I think Ireland has a very important role to play in this area. You know that already, and also of course because the UK's decision to exercise blocked out, blocked off out on justice and home affairs issues will certainly have implications for Ireland, a'r dynig, i ffwrdd, fel y cyfnodigwyr geografiol, lle oherwydd, dwi'n fwrdd ar y cwrdd y sydd o'r gweithio rhan o'r pethau o'r cyfnodigwyr ymlaenon ymlaenol yma yn y UED, a'r dynig o'r dynig o gyfnodigwyr polisiol. Yn y cyfnodigwyr ymlaenol ymlaenol, yw ein cyfnodigwyr eu cyfrifoedd ymlaenol o'r cyfnodigwyr sydd, ac yn y dynig ymlaenol yw Ysgrifennu Bryddem yn y Llywodraeth yn amser i'r ťdod yn y ffordd. Pan fydden ni'n dda chi i gilydd y 20ol nationale ofri oherwydd unrhyw drwy'r pan知lwch o'r rhaglen i'r gaelirdau newyddol yn y teimlo i **** yn y gynhyrchu. Mae'r rhaglen i'r gaelirdau oherwydd iddyn nhw sy'n gristafol i'r ein lleol i'w gyfeirio i'r llunios i eu polisiadau. Nid yn fan, mae'n neud i'r Rhaglen i'r llunios i'r unionid yn y Llunaidd. Nid yn fan, mae'n gweld bod y Llunaidd yn y Llyfr gan unrhyw gynhyrch in the field of police operation in Europe at the moment. I will stress the decision to exercise the bloc opt out, I think is at odds with the pragmatic approach of the law enforcement communities in Ireland and the UK both, actually towards European police cooperation. I will conclude that the opt out therefore will make it more difficult for UK law enforcement authorities to protect British citizens. Indeed it should remain very much at the heart of international police cooperation maesias, not least in Europe. As most of you know, the creation and strengthening of a European area of freedom, security and justice is one of the main objectives of the European Union and I believe it's also one of the best examples of the EU adding value to the lives of European citizens. You can't say that perhaps for every part of the EU but in this area from what I see I have a very good view of an everyday basis. I think the EU does add real value in the fight against serious crime. So the decisions made in this field by the UK, and by Ireland, and the European partners on issues ranging from organised crime, drug trafficking, terrorism, all the way through to migration and border security, of course these directly affect millions of citizens in those countries. y bwysig, y brif Weinidog, y brifyddyn yma, a'r rhai i'r Uchael yw'r rhaid, sy'n ystafell ar y dyfu Llywodraeth ac'r Yngrifedig. Mae gennym ni'n ddechrau i'r bwysigon mewn cyfnod mewn cyfnodol, sy'n mynd i ddim byddai ysgolwr yn Llywodraeth, ac mae'n dweud i gael ei weld o'r ysgolwyd ymlaen yw Llywodraeth. Yngyrch yn mynd i'r byd i'r pwyllwyr yn cael ei ffordd o'r cyfnodol ..on fundamental rights and civil liberties. So, the Lisbon Treaty has tried to change that and brought substantial changes into the way in which these justice and home affairs measures are adopted, implemented and reviewed. Most importantly, the Court of Justice of the EU will see its jurisdiction extended over the range of these measures, making the rights of EU citizens arguably better protected as a result. Secondly, the Lisbon Treaty has also increased the role of the European Parliament. They will now be fully involved in the adoption of these measures and this has increased democratic control as a result. Lastly, the European Commission enjoys its full right of initiative and will pursue those member states which fail to comply with EU legislation. Like for the internal market, which is of course so dear to the heart of many in the UK, a consistent application of crime and policing measures in the EU means less legal uncertainty and I guarantee that British citizens will benefit from a fair judicial and police treatment when travelling in the EU. This is a good thing, not a bad thing. In the EU, we're talking about 27 member states with more than 200, maybe more than 300 separate law enforcement authorities and 30 different legal systems, so it's quite a fragmented community on police cooperation alone. This fragmentation, lack of cohesion, lack of common application of systems and processes benefits only the criminals. So we need common rules and instruments with respect of course all the while our traditional national judicial systems but also allows police officers and judges to investigate and prosecute crimes that are committed in more than one country. Now some member states have anticipated the potentially far reaching implications of this super national legislation and have included a number of safeguards in this Lisbon Treaty that we see today therefore. We have the so-called emergency break procedure which allows any member states considering that a draft legislation would affect fundamental aspects of his criminal justice system. It can suspend the legislative process and refer the matter to the European Council. Article 67 of the Lisbon Treaty provides that in establishing an area of freedom, security and justice the union shall respect fundamental rights and the different legal systems and traditions of member states. In other words when adopting rules to facilitate police cooperation in criminal matters the legislators must take into account the differences between legal traditions and systems of the member states. But in addition to these common safeguards the UK and Ireland to common law countries of course have obtained that their former opt-out arrangement in the field of immigration, asylum and civil law be extended to the entire area of justice and home affairs. This is the substance of so-called protocol 21 that decisions in this area will not apply to UK and Ireland unless they decide to opt in. But when it comes to the 130 or so measures adopted before the entry into force of the Lisbon Treaty the new prerogatives of the Commission and the Court of Justice will apply automatically after the end of the five year transition period and therefore from the 1st of December 2014. This is why the former UK Labour government decided to negotiate the right to a block opt-out of all justice and home affairs legislations adopted before December 2009. So this is what we're talking about this is the so-called protocol 36 opt-out. And Ireland didn't seek to have this option. On the contrary Ireland insisted to include a political declaration underlining Ireland's firm commitment to participate as much as possible in measures in the area of freedom, security and justice. And that is the fundamental difference in the attitude of the two countries. Both are close partners in the fight against serious crime and terrorism. Both share the same concerns that their common law traditions risk being undermined by the European Court of Justice but only one has sought the possibility of a mass opt-out. Last October the UK Home Secretary announced that Britain would exercise the right to opt-out even if the formal decision has yet to be made following a vote in the British Parliament. It has until June 2014 to decide finally in this issue. Now today or indeed any other day for that matter I'm not going to give any advice to the British government or parliament on how we should decide in this matter. I think personally there are rational arguments on both sides and in the end it's a political decision of course. I am bound by my profession as a public servant and I know that I should refrain from straying directly into political affairs. But I am the Europol director and I do have an interest in this matter because I have a responsibility to ensure the strongest possible cooperation between member states in the area in which Europol is competent. This means in this context demonstrating from my side that proactive and constructive engagement with EU partners is the best way to protect national interests and the safety of citizens. I don't want any of our members to leave whichever country it is. It's really important to the success of our agency in the future that we retain the full membership and active membership of every member state in the European Union. We will be weaker if we are fewer. So my motives in this debate certainly not driven by politics or any sense of establishing a great European grand design. But just by the mandate as I said given to Europol and indeed to my function to help member states fight organised crime and terrorism. Now it's obviously for the UK to decide its future in the EU also in relation to Europol but it's my firm belief that it would miss something at least by no longer being an active part of our agency and indeed many other EU anti-crime measures. I also think that the UK's decision will have a long lasting impact on the country's ability to respond to security threats and to influence the EU's internal security policies. And it will have important consequences for the future of European policing. It will affect the EU as a whole, not just the United Kingdom. So allow me to detail what I think some of those consequences will be. Firstly that the opt out that the UK may decide will affect the future of EU policing because the EU is one of the biggest players in the EU internal security arena. At a strategic level for example and despite this rather peculiar position of opt out that it's maintained for some years, the EU often sets the agenda and security related issues in the EU. A couple of examples immediately following 9-11 EU ministers urgently considered how to respond with an overall of the shengen information system for example as a priority. And it was the UK that provided the lion's share of the ideas, even the practical requirements of shengen information system at the time based on the UK's model of the police national computer and its own border protection system. Another example, during the British presidency in 2005 the UK pushed for the adoption of council conclusions on the concept of intelligence led policing and as a European intelligence model. Seven years later this concept has become the cornerstone of the serious organised crime threat assessment carried out by Europol and the cornerstone of the new EU policy cycle in this area. And a more recent example is the current proposal to establish a European passenger names record system. Initially the commission proposal excluded internal EU flights but after an EUK initiative during summer 2011 the proposal has now been amended to include them. So for a country that's not a shengen member has an opt out of most JHA common rules and possesses a minority legal system that's quite an achievement. It shows that when British civil servants engage proactively with their counterparts in Brussels they carry a considerable influence. The same is true by the way for Ireland assets recovery is a good example as many of you know of just that. It's committed engagement with these European partners and institutions was instrumental in sharing the successful Irish model of the criminal assets bureau. And indeed Ireland was at the forefront of the creation of the so-called Karen network whose first meeting took place here in Dublin in 2002. A primary focus of that network which now includes 61 members with a secretariat based at Europol was to encourage member states to establish asset recovery offices. And after the adoption of a framework decision in 2007 all member states have now established such national offices. 18 of which are directly linked by the way to Europol's main information exchange system. So to conclude on this point if the UK does indeed use the blocked opt out it will reduce its own influence certainly on the development of EUJHA policies. And I think that will be bad for the UK but also for the rest of the EU because the EU has benefited from the UK's ideas and involvement in this debate on key dossiers over the years. And I believe that this reduced influence would come at a very bad time on the horizon of very important measures in this sector. Reform of eurojust, the creation of a European public prosecutor, a possible EU terrorist financing system and of course above all in my book at least the reform of Europol. But also at the operational level the UK law enforcement community has played a leading role in very very many major transnational criminal investigations. The UK is certainly one of the most active and best supporting member states at Europol and I think derives increasing significant benefits from engagement in this area. Organised crime and terrorism are certainly becoming more transnational in nature as the director of Europol and former international director of SOC I've seen enough evidence of that. Organised crime is becoming increasing diverse in this methods and structures. The groups themselves are becoming more mobile and flexible operating in multiple jurisdictions. These new trends therefore provide a new rationale for the EU to strive for more effective coordinated action at EU level and certainly not less. So a more globalised threat surely demands a more joined up response between governments and few people doubt this. In fact the UK law enforcement community itself is acutely aware of this and traditionally therefore been at the forefront of EU police cooperation. As I said it's one of the most active and committed members of Europol sharing large amounts of criminal intelligence and cooperating in hundreds of cross border investigations each year. In the last 12 months alone for example UK law enforcement authorities have been involved in more than 300 of the 600 most important operations supported by Europol across Europe. Half of all our most important work in the whole year bears a strong British footprint. With Europol this means therefore that the UK plays an increasingly progressive role in extracting operational value from Europol's unique capabilities. The UK and indeed many other member states I should say recognise we offer something that they haven't got already and that includes for example a set of intelligence databases and organised crime and terrorism which are unique in their pan-European scope and detail. Increasingly powerful in identifying criminal connections across borders in this more globalised world of criminality that we've identified. Soccer for example now routinely interrogates these databases with all its new cases. Every new case that is established at the national level is routinely checked with our database with a hit rate somewhere between 10 and 20%. So 10% and 20% of major cases in the UK are getting a response from Europol telling that investigators something that they don't already know that their investigation of a serious criminal in the UK has links to Italy, Portugal, Germany or somewhere else. And that's a major benefit for the investigator in such cases. Also we offer a breadth of expertise in every form of crime and terrorist activity drawn from the diversity of our workforce. Eugene mentioned that we just recruited as the head of our cyber operations an outstanding investigator here from Angada Shacona. We have 800 people like him now in our building that between themselves therefore operate, offer a breadth and diversity of experience that is found nowhere else in Europe drawn from every major law enforcement agency on our continent. Our staff base includes 140 liaison officers from those countries. Again this is a unique arrangement found nowhere else in the world. They carry full operational authority of their home country co-located with each other but also with 100 or so operational analysts that Europol employs. They use a purpose made secure information exchange system that we've developed and rolled out to every agency in Europe. And they use that capability and our analysts to exchange information and to coordinate around 14,000 major cross border operations every year. So it's an operational hub that is truly different in its scale and impact to anything else that you'll find in Europe. The UK recognises the value of these unique arrangements and has greatly increased its use of them in recent years alongside nearly every other member state. Unlike so many others around Europe the UK does this for pragmatic reasons because the UK strategy to fight organised crime and terrorism is founded on a belief that international cooperation must play an essential part. Also that the other reasons for this progressive attitude by the UK is of security professionals that believe that Europol is a more cost effective way of doing international police cooperation. I think the same could be said for Ireland as well. I'm aware that the Guarda has made a strategic decision to invest more in Europol and we've seen certainly the benefits of that in the last few years. In fact several member states including for example Denmark, the UK and Belgium have now closed a number of their bilateral police liaison posts around Europe in order to invest much more in the Europol network. We offer a cheaper way of doing police business in Europe. We also offer at a stroke access for one investigator from one national agency access to a pool of 140 officers from almost 40 countries. We know that this is taken up by several member states not least Ireland as well involved in the last few months in 30 operations or international investigations. 20 of these are still ongoing. It includes for example operation Oakleaf initiated approximately two years ago by the Guarda to tackle a mobile organised crime group known as the Rathkeel Rovers. This consisted of several close knit clans based in Ireland in the United Kingdom mainly involved in the distribution of counterfeit products, organised robbery, money laundering, drug trafficking right across Europe. It attracted some media attention not for those very serious criminal activities but because of its particular specialisation in the theft and illegal trade of rhino horns and their activities were being reported in North and South America, in China, in Australia because of that novelty. But that was just one relatively small part of their overall criminal activity. This group established companies all over Europe to legitimise and cover their illegal activities and a coordinated approach was certainly necessary to tackle their activities. Ireland recognised this and invited Europol to be the principal repository for intelligence and analysis in this investigation. Several meetings were held between different investigators at Europol, analysts helped to identify several intelligence opportunities and in the end more than 30 people have been arrested in more than eight different countries in this operation. So law enforcement community both in the UK and Ireland is making the best I think of European police cooperation instruments. As I said for pragmatic reasons cooperation between member states in these areas has brought many benefits to the member states themselves. Now when we look at the list of measures that the UK must decide whether not to opt out, we notice that many measures ensure that all member states codes of criminal law provide minimum penalties. For example in the field of terrorism, drug trafficking, cybercrime and money laundering. The UK derives obvious benefit in that these measures ensure protection to British citizens when they are elsewhere in Europe. It also deters the commission of such acts in the first place. For instance the framework decision on combating trafficking in human beings and the sexual exploitation of children. They have helped to tackle these terrible activities in a consistent way in the EU and reduced the possibilities for perpetrators to escape justice and to hop between different jurisdictions. In this case there is no additional burden on the UK because these minimum penalties were already in place within the framework of EU domestic legislation. There is no real risk of compliance proceedings at the European Court of Justice. Instruments such as the European arrest warrant however are much more controversial in the United Kingdom. Having said that, I think in this area there are real benefits derived by the police. Very often the real value of this instrument is missed in the media reporting currently in the United Kingdom. Certainly the warrant is used to track down some serious British criminals overseas. But actually the real public safety interest in this warrant is the ability for police authorities in the United Kingdom to track down and to forcibly remove from the United Kingdom several hundred serious criminals every year. In the most effective and cheapest and simplest extradition arrangements that the UK has ever benefited from. So there are amongst the list of criminals that the European arrest warrant deals with some real issues of proportionality which I think should be the heart of any reform of the instrument. But also there are as I said several hundred very serious criminals, child sex offenders, drug traffickers, even murderers, even suspected terrorists that without the availability of the European arrest warrant would otherwise remain on the streets of Britain. Without the ability for the police and the justice system to track them down quickly and remove them from the UK. So that is a very large public safety interest to forego if the UK decides to opt out. So these are just some of the examples that the UK must decide over. Opting back in again is of course an option which I think the UK is minded to negotiate over. It might be difficult because the commission and member states would negotiate with the United Kingdom in that case and may decide to add their own conditions to such an opt-in. But also I think it makes the whole environment that much more complicated because a partial opt-in, opt-in to some, not an opt-in to others dilutes perhaps the overall effectiveness of the suite of measures many of which are only efficient if they are used together and as a whole. So let me just conclude with some remarks. I'm certainly not here or anywhere else to make a political comment. I'm very sensitive to that point, that's why I've said it twice already. And I'm certainly not in the position to substitute myself for the British Prime Minister or the Home Secretary. But my experience as a practitioner in this area is that in order to increase efficiency in the fight against crime and terrorism, another time for example when cybercrime is growing exponentially, we need more European cooperation in police injustice. Of course it has to be effective and balanced and proportionate, but we certainly need more and not less. I don't call for more legislation by the way. I think we have enough to be getting on with. The real challenge for all of us in this community is to make sure that we make maximum use of what is already on the statute book. This assessment I think is widely shared by the law enforcement community across Europe. I think it is in Ireland, I think it is in the UK as well, where law enforcement authorities are leading the trend in terms of international cooperation. And they do it because they're pragmatic and because Europol support delivers real benefits in terms of disrupting serious criminal gangs. And I guess they know as much as I do the value of the UK remaining closely involved in European police cooperation in order to add value in fighting crime and terrorism. And they're not doing that, they're not in this game to please the European Commission or even the British director of Europol. They're doing it simply to help themselves in their daily job of protecting the British citizen from organised crime and terrorism. I think the same logic applies to the UK's role in policy making in the field of justice and home affairs. The UK is a respected player at a strategic level. Its influence of the development of an area of freedom, security and justice has been remarkable over the last 20 years. And British citizens are safer and better protected therefore when the UK is fully engaged in policy making because EU legislation is better as a result. The announcement that the UK will opt out of all JHA measures adopted before 2009 therefore may be at odds with the pragmatic attitude of security practitioners in the UK. The UK is an essential part of the EU internal security regime. The British and EU citizens are safer also when the UK is fully engaged in operational cooperation. There's an important point here that when the UK decides on its level of participation in operational terms in Europe then it affects also other member states, not just its own authorities. If over half of our case work carries a British footprint then in every one of those cases there is at least one other European partner whose ability to cooperate with its own major international partner will be diminished in the event of a UK opt-out. So to a certain extent the British decision has major European dimensions as well. But I believe in the end that the opt-out will severely hinder the future capability of the UK to operate effectively on the international stage in combating crime and terrorism. So I think that whatever the UK decides it must remain at the heart of the decision making process in the same way I think that Ireland is. For Europe we look forward to the UK and Ireland as member states remaining very strongly involved in all our activities. Thank you.


Today, I will be talking about x86 PIE, make kernel images which are just flexible. I hope this topic will not be boring for you. So let's start our presentation. Here is the outline for today's presentation. Firstly, I will introduce some basic knowledge about compiling and kernel address space layout. Secondly, I will show how to build the kernel as a position-independent executable and talk about some challenges we have encountered. Next, I will explain how to locate and move the kernel downwards. Then I will discuss the impactor of PIE on the size and performance. Finally, I will show the current status of our PIE patch set and its usage. And we will also talk about our future plans later. This is the first part. The building process involves compiling and linking. Firstly, each source file is combined into an object file, and then the link could combine one object file into a single F file. This is a simplified description, as the detail could be more complicated. So what is PIE? Our position-independent executable is an F file format and a link-in technique. It aims to enhance program security and aroundness. For no PIE program, when it is loaded into the memory, its memory layout is looked like this. Its starter address is usually fixed. However, for PIE program, its starter address can be randomized. So for each loading, it can have different load base. And when the kernel-advanced-adjust-space-layout randomization ASLR is enabled, the memory layout is also randomized. That is, the stack area and the heap area and the map area can be also randomized. Next, about the addressing model. The addressing model refers to how to calculate the effective address within an instruction. Different architectures have different address models. Today, we will only talk about two common types. The first one is absolute addressing. That is, the effective address is specified in the upper-end of the instruction directly. Then when the program is loaded into a different address, the target address is also changed. And we have to modify the instruction. For the second one is PC relative addressing. So the effective address is calculated by adding the offset in the upper-end to the next instruction address. So the offset may be positive and negative. But when the program is loaded into a different address, then the target address and the next instruction address is moved as the next instruction is moved together. So the offset is kept the same. Then it is suitable for PIE. No relocation is needed. When talking about PIE, GOT may be used and combined with address randomization. What is GOT? GOT global offset. The global offset holds the symbol address which could be changed during the runtime. It allows the EF file to be run at a different address no matter what the code or data is loaded. Although GOT provides flexibility, but it results in slower and larger code. Firstly, it is introduced in direction that if we try to get the variable, we have to get the address first from the GOT and then use the address to access the variable. This needs one more memory access and one more instruction. Secondly, the address in the GOT may be changed during the runtime. So we need to perform the relocation for it. So kernel doesn't like it. The GOT in the final kernel F is the most empathy. Then next about kernel address space layout. This is the four-level paging kernel address space layout. As we can see, the kernel image is located in the top two gigabytes of the address space. And the module area and fixed map area are also located in this range. Then why the kernel image is placed in the top two gigabytes? The main reason is for the size and the performance. Firstly, x8664 instruction set supports send extension, which means that we can use the move instruction to load our 32-beta mediator value into a 64-beta register. And the 32-beta value would be sent extended into the 464-beta value. So the kernel has used this feature. Its starter address is at 0xfff8000. Its 31st beta is set to 1. Then the lower 32-beta value could be sent extended to match the original value. For example, when we try to move the symbol value or texture into the RDX3 register, if we try to use the move instruction with a 32-beta immediate operator, then it only takes up seven bytes. However, if we try to use the move instruction with 64-beta immediate operator, it would need 10 bytes. So the former one is smaller. Then for the 64-beta instruction set, it supports PC relative addressing, which is usually smaller than the absolute addressing. And it didn't need any location during the linking. So the building process would be faster. What these features can be enabled by the compiler option is by setting the memory model as kernel, then it makes the branch instruction as the PC relative. And it would use the same extended model for move instruction and load effective instruction. So for x8664 kernel, kernel image's virtual address is fixed. It is not safe and not flexible. Then an attack could do quarterly use attacks easily. So in order to increase the difficulty of quarterly use attacks, kernel address space layout randomization was introduced. It moves the kernel image as a unit in the top two gigabytes. And it allows for physical and virtual address randomization. However, there are a few shortcomings to it. First is low entropy, only two gigabytes. So the relocate range is only two gigabytes. So there are only a few locations the kernel can fit in. Then this also means an attacker could guess the address without too much trouble. For the second one is one single address that could expose the base address because the kernel image is moved as a unit. So how to address these shortcomings? For the first one, we could build the kernel image as a PIE to allow it to be put at any virtual address. For the second one, the community has provided FDKSR, which enables function-level randomization. Also, we have found out that other architectures, such as ARM64 and S390, has used the PIE for KSR. They use the kernel as a PIE for a relocatable kernel. So we think we could build the kernel as PIE to enhance the KSR. For the second part, then how to build the kernel image as a position-independent executable? So first is the compelling process. We wanted to use PC relative addressing for PIE kernel. Why? Because we wanted to try to move the kernel image down below the top two gigabytes. So when the kernel is moved down, the same extended absolute addressing wouldn't work correctly. So we have to instruct the compiler to generate a PC relative addressing. Then it is easy for C source file. We just use the compiler option, FPE, since the compiler could generate a PC relative addressing directly. However, for example, file, we have to change all the existing absolute addressing. But this would be difficult to maintain if someone forgets it. So we have extended the object tool to validate a PIE building, which we have to do some checks in the object tool for each file. The second problem is the memory model. We, the normal kernel uses a kernel memory model, but it is not compatible with the FPE option. So we have to change the memory model as small, which is still in two gigabytes range. However, this would lead to some problems. For example, the stack protector, the kernel usually uses GSEgment register for stack kernel access. This is implied by the kernel memory model. When we change it to use a small memory model, the compiler would treat the kernel imagery as a user space program. So the compiler would generate FS Segment register for stack kernel access. Luckily, we have a new compiler option, which can allow us to choose the Segment register. And x8632 beta kernel has used this new compiler option for its stack kernel access. We can use it to for 64-bit kernel. Then when PIE compiler is enabled, the compiler would generate a GOT reference for undefined symbols. As we talked before, the kernel don't want any GOT reference, so we have to use hidden possibility to tell the compiler that the final address or symbol is in the final F, then it can safely use the PC relative addressing like before. However, there are some exceptions. Firstly, the compiler would always generate a GOT reference for weak symbols. And some compiler features would also generate a GOT reference in discuss the hidden possibility. So for PIE kernel, the GOT is not empty. Then we will talk about some challenges that we have encountered during the development. First is the F2 Snowboard Patching. When the checkpoint is enabled, the kernel would patch the call instruction as a normal instruction for performers. But when PIE is enabled, the compiler would generate a 6-bit GOT in direct call instead of the original 5-bit direct call. Then the patch wouldn't work correctly. Actually, we can patch the 6-bit GOT in direct call as a 6-bit Snowboard instruction. But as we talked before, we didn't want any GOT reference, so we chose to patch the first byte as a 5-bit Nopal instruction followed by one byte Nopal instruction. Then the first 5-bit Nopal can be patched as before. That means when the checkpoint is enabled, the first 5-bit Nopal is patched as a direct call directly because the target function is still in the PC relative adjusting range. Actually, we thought this is a compiler bug because we have used the hidden possibility, but it doesn't take effect. We have posted a bug for this C and of the fix has been merged. Then the headcoder. Here we talk about the C source code. The headcoder runs in a low identity mapping address during the early booting. So the runtime address is different with the compiler time address. And there is no high-adjust mapping during the other time. So absolute address couldn't work correctly, but the compiler may generate absolute addressing for accessing globals. This would lead to booting failure. So the kernel has used a hack called fix-up point function. You use this function to access the globals. The function works like this. First, get the offset to the text section, and then plus the base physical address. So the final address is the physical address for the symbol. But whether to use this hack depends on the code generation or compiler. And the client differs from GCC. And it also is confused to the development, because they don't know when to use this hack, when should they use this hack to access the globals. Then for PIE building, all generated references are PC relative addressing. So we don't need any fix-up point hack here. But we still need absolute addressing for some symbols during the early-page table building. So we have to use the 64-bit more instruction in line example. Also, this makes us think if we could use, if we could build the headcode as a PIE for normal kernel, not only the PIE kernel, because if we build the headcode as a PIE, then no fix-up point is needed. You can get rid of this hack. After talking about the compiling process, let's discuss the linking process. Before that, let's first introduce water relocation. Relocation is a linking optimization. The linker would convert the memory operator as a mediant operator to reduce one memory access. This is useful for the linker. Then for PIE kernel, or then for user-space program, PIE compiling and PIE linking are used together. Should we use PIE linking for PIE kernel, too? However, the kernel is different from user-space, because if we try to use no PIE linking, then it is simple and compatible with the current kernel relocation. But it has a problem, because it would use the wrong relocation relaxation, because we keep the compile time adjusted in the top two gigabytes. Then the compile time adjusts could still be extended, according to the spec. When PIE linking is not enabled under the adjust as located in the lower 32 beta adjust space, then the linker would choose the first one. So for the first one, the symbol adjust would be extended, but if we move the kernel down, if we move the kernel below the top two gigabytes, then the extended adjust would be wrong. Then if we try to use PIE linking, because PIE linking is standard for tortures these days, and we can make the kernel same as the user-space program. And also, ARM64 kernel has used PIE linking for a long time. And we can use the tortures for PIE to the same as the user-space program. But it would generate a dynamic relocation table, but the focus doesn't like it, because they think the memory size is increased. Now we choose the first one. In our PIE patch set, we use the no PIE linking at the present. So for the third part, how to relocate and move the kernel downwards. Before discussing the kernel relocation, I will introduce the relocation table, which is used during the kernel relocation. So the relocation table requires information about where and how the relocation should be applied. Each relocation table section is associated with a section where the relocation is applied. Then each relocation item has three members. The first member points to the place where the relocation should be applied. And the second member points to the symbol in the symbol table. And it also includes the relocation type, which tells the link how to calculate the final operandum in the instruction. What is the relocation type? The purpose of the relocation type is for adjusted calculation. This table in the right contains what is the relocation type for x8664. Now let's understand how the adjust is calculated by the linker. For example, let's consider the first instruction in the final kernel F that is load effective adjust instruction. As we can see, the relocation type is x8664 PC32. So the formula is s plus a minus p. s is the symbol adjust. jata is the end-inator text. And a is the operator. jata is the frame size plus 4. y is 4. I will explain later. And p is the place where the relocation is applied. jata is the instruction adjust. jata is at ff8 10000. And while the operand is at 0.3, so the p is the place where the operand should be where the operand is. Then as we talked about before, the 4PC relative adjusting, the operand is the offset to the next instruction adjust. So the final offset is calculated by the formula is s plus a minus p. Then we can get the final offset as 0x01, 2, 0, 3, f5, 1. Then how to get the effective adjust? As we talked before, the effective adjust is calculated by adding the offset to the next instruction. So the offset is s plus a minus p. And the next instruction is p plus 4. jata is 0, 0, 0, 7. So the final value is s minus frame size, which is the same as the instruction I listed here. So the relocation table has all the information. So the link needs to calculate the final adjusts. Then for kernel relocation, jata uses this table same as the linker. Firstly, this relocation table is usually jumped into the final f. So we have to use this compiler operation to generate the relocation table for the final kernel f. Then for the second step, we have to use a tool to convert each relocation item into a 32-bit value to reduce the size of the relocation table, because we only need to keep the offset member. And although it is 64-bit value, but it can send extended. So we only need 32-bit value. For x86, 64 kernel, in the final f, there are only three relocation types. The first one is 64-bit relocation for data section relocation. And the second one is 32-bit inverse relocation for Percipium valuable. And the last one is 32-bit location for normal valuable, which can be sent extended. We have to generate a new relocation table during the decomposition. So kernel can apply relocation according to this new relocation table. So as we talked before, actually only absolute addressing needs relocation for PC relative addressing. The offset is the same if the program is loaded into different address. So actually, we shouldn't perform relocation for PC relative addressing. However, for Percipium valuable, for SNP data is special, because the Percipium section is mapped at address 0. And we use the GIS segment to access the Percipium valuable. So the kernel slaughter data is segment based. No relocation is needed. Because the symbol address is offset to Percipium address. It's already offset to Percipium address, Percipium area. Then if we try to move the Percipium area, we just need to modify the value in the GIS segment to adjust. Then for the PC relative addressing, the kernel tried to subtract the relocation offset from the immediate part. That is look like this. First, during the runtime after the relocation, the new IP is the IP plus the relocation offset. So we can subtract the relocation offset from the immediate part. Then we can still get the right address. For PIE building, there is a problem, because we try to move the kernel below the top two gigabytes. Then the relocation offset could be larger than two gigabytes. But the immediate part is only 32 beta. So we choose to subtract the relocation offset from the GIS base data as GIS register. Then once the kernel subtracts the relocation offset from the GIS register, then there is no relocation needed for PC relative addressing anymore. So the formula is look like this, GIS minus the relocation offset. Then we also think FV could use this method for normal kernel too, because actually for normal kernel, in the final F, we could find out that the PC relative addressing instruction are more than the absolute addressing instruction. So if we use this method, and no relocation is needed for PC relative addressing, then the relocation table is decreased. For PIE kernel, there is also a problem with the code patching. What is the code patching? That is, the kernel would modify the code during the runtime. For example, alternative patching, it looks like this. That is, there we list all the instruction and the new instruction and the flag. During the booting, the kernel would detect the specified CPU future. If the flag is detected, then it would try to use the new instruction instead of the old instruction for performance. However, there is a problem for PIE, because for normal kernel, then we try to access the global, the compiler would generate absolute addressing. And during the relocation, the kernel can perform relocation for the absolute addressing. Then the final address for the global is correct. However, for PIE kernel, whether reference is PC relative addressing, then if we move the new instruction from the replacement section into the texture section, then the offset is wrong. So we have to modify the offset for the offset. That is, we have to process the offset between the replacement section and the texture section. Then we can make the PC relative addressing still worker. For PIE kernel, the kernel relocation process is the same. The only difference is for step two. We only keep 64 beta relocation, because the data section relocation is always generated. And also, we talked about before, the GOT in the PIE kernel is not empathy. So we have to generate a GOT address relocation, because we use the no PIE linking, the link wouldn't generate the relocation for GOT. We have to do this by ourselves. And then in the final relocation process, after the kernel is built as PIE, we can put it at any virtual address. So we can extend the relocation range for KSLR. That is, the final relocation offset is PIE relocation offset plus KSLR relocation offset. What is the PIE relocation offset? That is, we first choose a two gigabyte hole for the kernel. Then we perform the KSLR relocation in this two gigabyte address space. Then we have extended the KSLR relocation range. After building the kernel as PIE, then we can move the kernel downwards easily. First, we combine the kernel image and the module area fixed map area as a unit. Allow it to be put at any virtual address. Why? Because we should keep the module area and the kernel image in two gigabyte range. Then we can still use PC relative addressing for kernel and for module. Then why we put the fixed map area in the kernel in this kernel image area too? Because the module area and the fixed map area use the same PGD tables. So we don't want to make it complicated. So we just put the fixed map area in this range too. Then this also means the fixed map area for PIE kernel can also be randomized. After moving the kernel below the top two gigabyte, then there are some changes we should make. First is the physical address and the virtual address transition for kernel image address. Because we have moved the kernel image below the top two gigabyte, the transition calculation should be changed. And also, there are some systems that are still establishing about the kernel address. First, for example, the top PGD entry may doesn't exist. We have made a problem in the V6 core page building for our page. Because it assumes the kernel is still in the top two gigabyte and the V6 core page is still in the top two gigabyte. Then after we move the kernel downwards, it triggers a warning. Then the second problem is the BPF core GIT code generation. If it would generate a code instruction by this function, and for PIE kernel, it would also trigger a warning in this function. Then we talk about the size and performance of PIE kernel. We have tested three cases. Jetta is no PIE normal kernel without a PIE patch set. We have without our PIE patch set. And the second one is PIE configure is disabled. And the PIE configure is enabled. As we can see, the result and we have tested two different configuration. One is the default configuration. And the second one is new bound to configuration. As we can see, for PIE kernel, the file size is increased because we need more instructions for PIE kernel. As we can see, the text section increased. So why is the text section increased? There are two cases we can introduce. First is if we try to access an ARIA element for PIE kernel, it first should get the best address of ARIA and then use the second instruction to get the ARIA element. However, for no PIE kernel, it only needs one instruction directly because the best address of ARIA can be sent extended. The second one is switch case optimization. For PIE kernel, the compiler would generate compare and jump instructions. And for no PIE kernel, it would generate a jump table. There are only one instruction as needed. So the text section is increased due to the two cases. We also tested the PIE on two benchmarks. The first one is Hickman Bench and the second one is Kernel Bench. And as the result shows, there is no notable changes for PIE kernel. And the final part, the current status of PIE patch set, actually the original patch set is posted by Thomas four years ago, four years ago. But it only extended the relocation range from two gigabyte to three gigabyte for KSR. And it would lead to some interference on some configurations. Then this year, we have posted our new IFC patch set, which is based on his patch set. But we made some design changes. For example, the new POSCPU valuable relocation. And our patch set allows it to be put any address space. We have extended the relocation range to 500 and 12 gigabyte as an example in our patch set. And we also fixed the boot fillings on Thomas patch set. And we have tested the PIE on various configurations. Then, except for enhancing the KSR, what we could use the PIE kernel? Or what are the PIE kernel could be used? First, we have used the PIE kernel for our guest PIE kernel, which is designed for a software-based hypervisor. We called it PVM. The paper about PVM has been accepted this year. We use the PIE kernel to make the guest and host kernel address space not overlapped to reduce the TLB flash. So our future plans, we want to improve our building process. First, we need more compiler support. For example, the previous problem we talked about, we want the compiler can solve these problems. And the second one, we want to use the PIE linking. We want to use the PIE linking if it is accepted by the focus. Then the final goal is to upstream. We want to upstream our PIE patch set. However, the focus concerns more users cases. However, actually, we didn't find more user cases. An idea is that if we could use the PIE kernel for new user model Linux design, for example, we could use the POPOS plus and PIE to follow new user model Linux, then we could use the same binary for user space to boot the user model Linux. However, it's just an idea. We didn't start it. So that's all. That's all. Thank you. Sorry, my presentation is time out. Have any questions? Sorry, sorry, sorry. If you have any questions, we can talk about it later. That's all.