Thank you. It's a pleasure to be here. Yeah, the working title was a bit different I'm just a little bit about me. I'm a back-end engineer. I work on the core back-end team at Yelp And I got quite a bit of content. I'll try to be quick enough so that we have time for questions One quick word about my employer Yelp. We have a website and an app. We help people find great local businesses We're actually like pretty popular especially in North America. We have over 30 million monthly unique users on our mobile app We have over 130 million reviews. So we do work at scale, which people seem to care about So what I'm going to be talking about What are type annotations and why should you use them? This will be like very short like one quick slide this talk basically is Hopefully starts where like the more introductory talks end. So I hope you already know a little bit about type annotations We're going to talk to how to incrementally migrate an existing code base to using type annotations What are some of the issues you might be encountering? and The last part which is actually going to be a relatively big part is how can type annotations help across services? So Across network boundaries basically one thing I'd like to say is that I'm by no means an expert on type annotations I just I'm just going to talk about what I learned as a user. So With a grain of salt, let's Let's start. Oh, and I'm also going to use a bunch of source code. So if that's not your thing, I'm sorry I think it fits with With my talk That said if you want a bit more Like introductory talks or talks that explain a bit more the why and and the how In in the more basic sense, there were like two actually really great talks at Picon US earlier this year One by Karl Meyer from Instagram the other money the other one by Greg Price Who is now at Zulib was a draw pox previously. So I encourage you to check them out really good talks But let's give you like a very like quick Hopefully reason to use type annotations. So if you have this code Super simple function takes a string returns a string. We call it with an integer We'll get an error not at runtime, but we'll talk about like when in a second Now you might say this is a contrived example. And by the way, this code works perfectly well with an integer It does in like when you run it One reason yes for type annotations is to help you find bugs and issues with your code The other one is to actually document your code So what this basically does it tells you that your documentation is wrong or incomplete Which I think already is a big big advantage because the documentation is now part of your code And if it doesn't match reality you get an error. We all know otherwise. What happens with documentation gets out of date and the reason why Many companies with larger Python code base use annotations is more like functions like these Where you get items and you don't know what items is actually like this is basically an example. I I took from Karl Meyer and We we know it probably like has a value attribute and that has an ID, but we don't really know And it gets even worse if it's a dictionary instead of like some other type of object So as you have a large code base that multiple people work on you probably joined the company like years after like this code base was started Type annotations really help with understanding What's what's going on? So let's talk about how you might migrate this existing code base large code base To type annotations the goal here is that we want to End up that all code is type annotated We want to do this incrementally. So this is not going to be a code size We're like just one person can sit down and spend a day or two or three and just you know Burn through it and everything is annotated And we want to make sure that for the code that we have already annotated That we do run checks on it and make sure like we gain value out of those annotations To check annotated code, we're going to use my pie. I assume most of you have heard about it It's the de facto standard type checker It's like a sort of linter that you'll just run on your code just like I don't know flake 8 or whatever And it will tell you When there are issues with your code It's still like not at a 1.0 release. There are some small issues still we're going to talk about a few of them Yeah, but it's it's really good and it's improving rapidly. I'm also going to be using the Python 3 6 Type annotations because they're just so much better You can use type annotations with Python 2 or earlier versions of Python 3. It's just because it looks better There is actually another type checker in this case for Python 3 only from Facebook or Instagram like That area it's supposed to be much faster, especially for large code bases We're not currently using it internally at Yelp. So I don't have any Personal experience with it, but you should definitely check it out Yeah So let's get started How do we actually enforce annotations like my pie has a lot of options? And we need to configure it correctly so that it does what we want to do again like our Goal here is to on a file by file level make sure code is annotated. So that Whenever somebody Changes something we can we know if the code is annotated or not. So the first three part configuration options basically say that non annotated code is an error The next two One say that any code that I'm not like providing you as an argument on the command like like any file Yes, you need to follow it to understand what's going on But don't tell me about any errors or problems in there and Then this is an important option. We want to make sure that you handle none correctly This is like a big source of bugs and issues in many languages that don't handle none explicitly In and even in Python oftentimes. Yes You kind of make sure like it's it's a list or or an iterable or whatever But actually like are you making sure that it doesn't crash or throw an exception when it's none And for the cases where it can be none, you need to be explicit about Dealing with it this I think is going to be I'm not sure if it is already or it's going to be the default in my pie But just in case it's not you should probably set it and this is kind of like I would say like a recommended By me so my personal preference for how to do it There's like actually a strict option in my pie that is stricter than this But you do need to find like a trade-off between How much time you invest in writing the perfect annotations and You know getting code to production So how are we going to enforce? These annotations or that people make annotations We're going to use a tool called pre-commit It's it's an open source tool Open source by Yelp that provides All kinds of hooks that you can run when developers commit code There's like a lot of hooks for a lot of different languages and file formats The Python support is really good because we're like a big big Python shop and Yeah, that's what's what we're going to make sure that that code is annotated So let's take a look at how to configure it. Basically. We need to add a separate hook that is my pie One thing we're going to do is we're going to pass it a special config file the one that I showed you a couple of slides earlier and and Yeah, this will make sure that That any code that is being touched is fully annotated to install Pre-commit basically you run a command just like this typically you can do this when like as part of your test suite or as part of your like Make or set up or whatever step. So just to make sure that developers have this installed And then when you run it or when you basically commit something this is how it could look like so you see there like there's a bunch of Tests or hooks that run on on the files I touched the one where it says skip This is they don't apply to that file type and you can see the my pie hook that we just configured is failing because apparently I I modified a file called media dot py and Yeah, it to get photos future thing I used like it has an incompatible type Also, I forgot to add a return type annotation and also an annotation for one of the arguments In in that function one thing I'd like to say is that all of this is like based on The team like everybody agreeing that this is a good thing so you can definitely skip individual pre-commit hooks or all of them So this is like when I say and force it's just like making sure people don't forget Not forcing people to do things. I'm going to address this a little bit at the end But obviously everybody should be on board that that type annotations are a good thing But we do want to run my pie also as part of our test suite To gain some of the advantages of having type annotations. So we're going to use a less strict Config option for that. So it's totally okay to have non annotated code Also, this is going to be run on all of our code not just files that were changed or touched and Yeah, we're like what what is going to happen once you start with type annotations Is that you're not going to notice like a lot of Issues being caught by my pie when you run it as part of your test suite That is because as soon as it encounters a function that is not fully annotated It kind of like uses this any type which can mean it could be anything and it won't basically stop type-checking and and the value that function returns will also be any so It'll take some time to to get started to really see results but remember that those results are going to come and that Already like from the first annotation you add that's already documentation. So that already adds value to your project That said it can be like a little bit tedious to annotate everything by hand so there's tools now that can help you get started Probably the best one as far as I know or as far as I have tried is monkey type also by Instagram Which really helps it can not only connect gather type information as you run your code and Write that in a file. It can then also use that information to automatically Annotate your code using the Python 3 annotation syntax That's a great way to get started. Just remember that it won't it's not a magic bullet like a developer needs to go through Those annotations check them Sometimes they are too strict so you can loosen them up Sometimes they can just be plain wrong. So if you run them as part of your test suite You might see like a bunch of like mock annotations in there, which obviously you don't want to have in there Yeah, and the second tool that I think comes out of Dropbox is pie annotate, which we've also tried internally It actually also works reasonably well the main drawback is that since as far as I know Dropbox is still like very much a Python 2 Shop they do The annotations in this comment form that you know works in Python 2 as well But if you are on Python 3 or Python 3 6, it's My personal preference is to use that syntax. It looks much closer to Like types and other languages. So that's maybe the main drawback here So but one of the important like the the most important thing for type annotations is to type your data like I Said in the motivating slide Actually making sure you know what data Is passed around is super important I mentioned the dictionary example When you when you're there and you join and you get ramped up and you have like well in our case We work a lot with businesses and you have like a business ticked and then well what's in there I don't know and it gets even worse if you then have functions that like modify the data along the way So all of a sudden it looks a little bit different and Do I already have the mutated form here or still the original form? I don't know So that yields like a lot to a lot of like productivity loss bugs frustration and name tuples are a great way to kind of Deal with that. I personally like them a lot. Yes, and Python 3 7 we have no data classes Pick whatever you like best the good thing about name tuples is that they prevent Mutation so you can kind of like be more less sure that once they are created. They're not going to be modified And as you can see here, like I'm using the new Python 3 syntax for name tuples I think it's great readability you can nest them So here photos is actually like an iterable of photo photo is like another name tuple that I'm not showing here Yeah, and I think that's that's really great like it's really clear what your data structures look like and We probably want to use those now There's also a way to type your dictionaries if you don't want to or cannot use name tuples As you can see this is actually like very similar syntax. It's basically just a different base class And unlike maps or hash maps or whatever like dictionaries are called in other languages In Python you can have different types based on the key like I'm doing here so It will distinguish Based on the on the field name what the type of the value is so let me show you an example in this case, I Don't know if you can even like spot that but this code these three or two lines really only it has a bug The I'm taking the business dictionary And I'm calling to get function and I'm saying give me this address to field and if it's like none Then use the empty string because I don't want to like I'm trying to be a good developer preventing like issues with with none When I when the code down the line expect strings And this is really like an insidious Bug because this will work at runtime 100% of the time it will never crash It will never raise an exception But it will also never return anything else than the empty string because I forgot the D In that like as a second D in address 2 is missing. So unless you have like really good Test coverage and you like, you know because you know like your code is not buggy So what happens is your test fails? Oh like and then in your test mark or whatever You also like use address 2 with just 1d and all of a sudden your test passes all of these things have happened already So this is like pretty bad But if you use like here like I'm I'm using this annotation of this business stick that we see up above And what will happen is that my pie will tell you with the code as written here That business stick has no key address to So even if you need to for some reason continue using dicks Using the time to write these These typed definitions of your dictionary And then using them just for annotation like that the business stick class you can actually use it in code So you can instantiate an instance of it and things like that But even if you just use it as an annotation It it works really well and and can help you prevent issues like these Then yeah, then you're on your own so It won't I don't think it can know like so what I didn't mention my pie like is a static type checker so All of this like no it has no effect at runtime So if at kind of like quote-unquote compile time or just by looking at your source code It cannot figure out what what the name of the field is then because it's kind of like dynamic or whatever The type checker won't help Does that answer your question? I don't know if it does do that that's like very specific I would need to test it Yeah, but that said I hope maybe you are convinced that you know It's a it's a good idea to use name tuples So the question is how do you do that? One thing we could do is we could say like we have our name tuple definition we actually like use that as the source of truth of how all the data should look like and Since like we might get dictionaries from some like external data source JSON or whatever So let's write a small helper function that converts that dictionary to To a name tuple That's what it looks like I'll I'll gotta speed along a little bit We're going to use this helper function to see like some of dishes that we might run into when we Type our code. So as you might notice this function is not annotated. So how would we annotate it? One first try might be we just Use the name tuple type. So we say like NT class here is well It's not a name tuple instance. So it's a type of a name tuple instance And dick values well, it's pretty clear and we return a name tuple this Doesn't work because what we return is not a name tuple instance We care about like what specific name tuple it is not just a general name tuple about the second arrow We're going to see about that in a second So what do we do here we learn about generics And we try to use those we create like a type for here that is bound to name tuple It kind of like says whatever Class or type this is the base type must be named tuple And then we use that here and we say like whatever like class you pass into this function as argument We're going to return an instance of that as output so that Should fix the arrow we saw earlier. Unfortunately, this doesn't work either because of The fact that we cannot use name tuple as a base class That is also the reason for the error on the on the slide On the for the second era on the on the on our first try Name tuple is actually not a class it's a It's a class constructor that returns a class that there's actually like no common Name tuple base class. So what we need to do here is we need to use protocols as Which is like a relatively recent but really cool my pie feature which Implements or is basically the solution to duck typing so we can say like hey This is how this argument or this type behaves instead of Saying like hey, it's an int or whatever and here we can say like hey We need something that has like an underscore fields Attributed has like an underscore make method and then we return like an instance of that So here we say hey, it's a protocol fields make Then we read we can say that is kind of like our base type anything you pass in here We're going to I'm sorry Needs to needs to conform to that and all of a sudden it works I'm going to quickly go through we might not have time for questions at the end Some of the other issues so if you do use name tuples It turns out like count and index are like pretty common attribute names But there is a base class actually to name tuple which is the tuple so that is actually the common base class for all name tuples and They have an account and index Method so if you use the name tuple like this, it will actually work. You basically just need to silence these my pie errors Just something to to know another thing I'd like to mention if you have a larger code base Maybe you have a descriptor so like an object that manages an attribute or a property of another object in this case We have this set once property descriptor, which makes sure you can like set the property once But then only read it never write to it again It's like like a generic class. We can see here like two types the type of the object that we're managing the property of and The type of the value we use that here and get and set I'm just going to go through this the slides are online And here we can see we create a concrete type bizarre context by the way you use Strings if you need to use a type that is not fully defined yet and we say like the biz user ID in this case is an integer and My pie will understand that this user ID like you can only assign ints to it and when you read it It's an int I'm mentioning this because descriptors are not yet documented. So if you want to do that Please do so open a pull request Another thing you should be aware of My pie doesn't support recursive types. So anytime you have a sort of like tree or graph like structure It won't work You'll get an error There's a ticket to like a solution to this is basically to Ignore these errors and then use like the cast function that my pipe provides To to annotate by hand But typically whenever you have like these kinds of data structures, you're kind of like lose lose type information So let's try to get quickly through to my last part of the talk How to do type annotations with distributed code like a problem there is really like testability How can we make sure that like yes when we have our code calls several other services? That all of this works that we like the data we're receiving and that we're using that we're using the right format At Yelp we have an API specification. We use open API From that we create a client library that we then use to make API calls Just quickly this is kind of like how an endpoint looks like So it's a get request. That's the URL and we return a business object as response that is defined like this Looks very similar to the like name tuple definition or the type tick definition. Just maybe a little bit more verbose and This is how we make with this like client lib that we have at Yelp This is how we make the call. It's actually like for those who know it's the wrapper around the bravado library, which is open source So you can see like for us It seems like we're dealing just with Python objects and then it does all the network stuff in the background Here we create the client then we make the call this actually like makes the call and So let's write a small help of God Yes, a small helper function to to get the review rating from a business, which is the aggregated rating of all of the reviews And let's write a test for that because that's mainly the problem, right? Like typically you have to do end-to-end tests or whatever to test network code But maybe we can use marks and get away with it So if we do that We just use a mark here and kind of like fake have a fake business and then we set that as return value It's like a little bit unwieldy, but it it works like here in this case for our like client our mock client And this test passes now what I didn't didn't mention. I'm going to go back a little bit here review rating is actually of type string That's for historical reasons. There's actually an example that we do have at Yelp There were issues with like floats and and serializing and deserializing them in earlier Python 2 versions Not a problem anymore, but that's still the endpoint definition. So actually This code has at least an annotation bug because it says it returns float when in fact it returns string And then if down the line the code like gets to the wrong type That's probably going to be a runtime issue. So our test passes, but It's actually like the code has a bug So what can we do here? How about we take the API spec and we write some code to generate annotations from it And let's start with the model stuff. So like these these data objects that I showed you This is kind of like a class that mimics what I shown you in the open API spec and we see review rating as a string We want people to have that generated like we don't want people to type that So let's use that. This is the same function just repeating it for So that you know what I'm talking about when I show you the new test code and this new test code Now it doesn't use a mark, but uses this model class As a return value in our code and if we do that We actually get an error from my pipe because it knows that the business model that we have generated Review rating is of type float. So it is of type string. So I shouldn't assign a float But can we even go a step further so there's like very Dynamic client object that I've shown you here that basically like reads the spec and Generates these attributes and these methods Can we annotate that as well? That would be really cool because just these model classes Even if we ought to generate them people need to like manually you like add annotations To their functions. Yes, you can use them in tests like I've shown you but we also want like our non-test code to you know To have annotations and it's actually possible. So this is like a shortened version of What we have at Yelp. So basically You create like and all of this I should really reiterate that needs to be generated. So in the end You don't have to like manually do any of this So yeah for this business service we we have this like business service annotation It has like well in this case one attribute to business. This is the client or business now It has an annotation. It's a business resource and what is a business resource? Well, the business resource has this business info method So there we now know it gets one argument business ID and what the type is so if we call this thing the wrong way my pie will give you an error and Sorry, and it will return like a future that is defined below That is a concrete type like we pass it the model that is returning and so if we like we Automatically get all of these type annotations and my pie will complain if you do this network call incorrectly or if you use the data it returns incorrectly and All of this like modern ID is like pie charm even give you IntelliSense and and auto completion and all of that stuff so takeaways Annotate your code to improve documentation and catch box earlier fine grained Data structures type data structures really help understand the data flow of your application Reduce the number of tests you have to write especially make the test you do write more valuable And you can use generated annotations to type check communication across network boundaries so check us out on Social media especially the engineering blog we are hiring Please come talk to us at the booth and you can find the slides over there. Thanks a lot


So hello everybody, I'm Alejandro Pinheiro, I'm going to give a presentation about ESPV, about the work in progress bringing this station to MESA, but first I would like to make some announcements. The first one is that the XTC is announced and it will be on Coruña on September this year and you can follow the old updates on Twitter. It's rated with that, but I would prefer to mention that beforehand, that the MESA drivers, the inter MESA drivers for Linus is now conformance, I don't know if everybody knows what that means, that means that basically Coruña that is managing OpenGL just has at the street, so you pass all the tests, you get certified as conformance and Intel got this to be conformance science day zero or day one as you prefer to see that. That includes passing the test for this extension that is part of the thing that I am going to explain on this presentation. So, what are the topics that I am going to cover, I will make as introduction to what is everything, what I am talking about, what is this work related with. I will also make a summary of the development history, what we found when we started, what issues we had and why, how we solved those issues. That led to the technical decisions, to the work we did. I also mentioned how we test the work that we do and finally a summary of the current status and future. So, introduction, ok. Who is doing all this work? The initial work was done by Nikolai Handel, I am not sorry if I pronounced that wrongly. Right now, this is myself, this is Dwargo Lima and Neil Robes from Eagalia and this work is supported by Intel. So, I... Can I interrupt for one second, we only have audio on one side, if I have a mono. Ok. Is this better now? Hello, hello, one, two, three, testing, testing. Ok, sorry. Ok, introduction, ok. I am going to make some brief introduction to add the main term that I am going to use. And the third one is probably everybody just, you are already used to GLSL. It is the open GL setting language. It has been used as the open GL language of science 2004. It is a kind of SEAL-like language. One of the things, I mean, one of the features is that the setting, the source code is included with your program. So, it is not exactly private. I mean, even if you try to do that, in the end the driver should be capable to get that code. Some people didn't like that. I mean, here we are, everybody is open source minded. So, probably all of us is not a problem, but for other people from the industry it was a problem. And then we have SPIR-V. It was initially introduced as just SPIR for open GL. In this case it is a binary format. So, they solve these private problems. Initially it was based on LLVM. And then SPIR-V was announced in 2015 and became part of OpenCL21 and Vulkan. That is probably the reason they added this V. And in that case it stopped to be based on LLVM. And now it is just one spec. So, what we have now is that, well now no, at that moment is that we have OpenGL and Vulkan and each one has different setting language. One was using gceler and the other one was using SPIR-V. But at the same time people wanted to use both. People were porting from one to the other. So, it was needed some kind of interoperability between them. So, the first one was this extension. But this was for Vulkan. This extension defines how a Vulkan program could use GLSL. But in this case it's not a kind of extension for Vulkan itself. It's a kind of front-end extension. It means that for example if you have a compiler that we are already doing GLSL to SPIR-V conversions, this one defines the scope for this compiler. What we expect from this transition. So, after this one they provided the equivalent but for OpenGL. That is the main topic of this presentation that is SPIR-V. This one defines, in this case defines two things. One is the ability for OpenGL to load SPIR-V modules. And the other is also this front-end definitions about which feature from GLSL we want in this case. So, in this case it's at the same time a driver and front-end extension. And there is also another small extension called SPIR-V extensions. It is because the previous extension, the main, I don't know how to say that. It's based on SPIR-V 1.0. So, what happens if we want to extend that functionality. So, these extensions allow to expose which extra features we have. And the spec implies that in some cases we will get some OpenGL-specific SPIR-V extensions. And, related with these extensions and related with this front-end, the chronos in addition to the extensions also creates a reference front-end. One is GLSL and that, for example, it uses the definition of these extensions to create SPIR-V binaries based on GLSL. And they also maintain SPIR-V tools that are tools to manage SPIR-V modules like assembly, assembly, validate, etc. So, we have, I mean, if you are SPIR-V with OpenGL you already know that GLSL has a long, has been used for a long time. So, one of the reasons they define these extensions is to reduce the scope. So, for example, the first extension that I mentioned remove a lot of features, most of them all GLSL features and add some kind of Vulcan-like features. And then we have the OpenGL extension that use that one as base but at the same time it changes a lot some things. Like, yes, I know, it's confusing. It was confusing the first time that I read that because the second extension removes some GLSL extensions, removes some Vulcan extensions and add specific ones and tweak existing ones. So, for example, subroutines were removed on both. But, for example, atomic counters were removed on the first one and then read it on the second one. So, and the specs say that, I mean, they have a list of these features were removed on the previous extension but we are now including again. But the big change here, in my opinion, was that in SPIRV, the names of the variables, the variables, uniforms, UBOS, etc. are considered like the book information or are optional and they are maintaining that here. So, for example, a frontend could get a GLSL shader and then create the binary without any names. So, that means that in OpenGL, when you load the SPIRV, it needs to work without any names because it's optional. In fact, I will mention that later, but in fact our approach is as it is optional, we are not implementing that for now. Probably we will implement that in the future but for now we are not taking into account the names that comes from the SPIRV binary because it needs to work without them. So, it's another way to test that we all call this correct. If you don't have names, then you need to use locations, bindings, indexes, etc. So, I'm going to make a kind of development history about how this goes. So, the pre-story, I mean, how the interest for implementing this extension started. It started in 2016. I mean, all these things that I have on quotes are emails that people send to Mesadep. So, I don't know that the name of that person, it used a kind of nickname. So, it sends an email asking about the interest to implement that extension. So, several drivers, several drivers developers started to say, well, I would prefer this way and the other way, etc. So, it was a month, almost a year later, Nicolai sent a request for comments threads to Mesadep. It includes a starting point with some code. It also has some questions. It was also focused on radiancy and some people started to jump about the approach. But it didn't went too far from there. It also mentions slightly the testing, but the thread was not really long. So, Galia, the company I'm working for, jumped on the implementation of this extension around September last year, mainly because Nicolai at that point didn't have plans to continue to work on this. So, but we used his code as a reference and was really, really useful as a starting point. He had a code for Mesadep. I will explain what is played later, but yes, it's a test suite, a different test suite for OpenGL. So, our first steps were integrating it with the Mesadriver with the Intel Mesadriver. There's a missing word there. I start to use the CTS test for this extension in order to evaluate what was missing. One of the things that was based on that code it was near, the thing I don't know if everybody is used to this terminology, but on compilers when you have a language, you usually create a kind of three intermediates in the representation. Sorry, I'm not a compiler. I am not a person with a background of compilers so sorry if I explain that properly, but basically you parse the language and you get a intermediate representation. And at the intermediate representation there are several approaches. So on OpenGL they have one and at some point Intel decided to create a different one and they started to use that and then for different reasons there are plenty of talks around if you are interested, but Intel started to use that and then other drivers started to use that too. So, not as an utility, other based on Neer. So, for example, on Intel Mesa driver we have a chain like you start with the GLSL shader you create, it is passed to the GLSL higher representation then this translated to Neer and it finally converted to the more low level interior. And when Intel created, they wrote the Vulcan driver they created a Spirby to Neer pass by passing one of the intermediate languages that were around. And why they didn't just remove that representation. The reason at that moment is that there isn't a linking based on Neer. So the linking is done on the previous intermediate representation that is Mesa higher. So Neer is this intermediate representation but the objects are already linked. And again a definition of a compiler's topic from a non-compilers, from a person with a compiler background. If you go to Wikipedia this is a kind of generic definition a linking is, I mean when you compile you usually compile different objects and then you need to merge them together and this is the link more or less you get the idea. On Mesa if you have tried a lot the GLSL link those two things, two main things gather info from all the objects because the GLSL shaders are really you can query a lot of state information from them and that information is also used for validate because you can have one shader and other shader but they are using the same UBO but the definition should be the same so it needs to be validated. So the first approach the first approach from Nikolai was trying to reuse the IR linker as much as possible. So he's taking account that we have the Neer but what he did was creating taking only the variables and converting them to GLSL IR variables and trying to reuse as much IR linker as possible and then after that it used the new shader as usual. That was a really good approach for the bodystrap that allowed it to have a lot a lot of the linking already done so it allowed it to test the code with several cases and getting some shaders working but well, the part is late, sorry so after all this introduction then we started to code we started to get into the dirt so the first approach was trying to get the CTS test passing the two main reasons is that the two main reasons I thought that would work with that the main reason to use the CTS test it was they covered most of the spec as no no as it was intended and at the same time they weren't too many tests they were like eight so it gave us a lot of it was a limited scope we can gather them the other thing, I mean probably other people on our team didn't expect that at least I didn't expect that that we needed to add more features on the SPIRV to need that intended but if you think that is that is normal the thing is that at that moment the SPIRV to need pass was focused on Vulkan so it was focused on it has only the features that Vulkan needed but there were some SPIRV features that were defined on the spec were let out because was only needed for OpenGL and that includes atomic counters transform feedback the selections and we also need to do some tweaking on how the SPIRV to need pass a 100 UBOs and SSBOs oop I don't know where I am do do do do do do... okey, so at that moment when we started to call trying to get the SPIRV we started to think about the link and the thing is that as I said this trick to convert Ok, eu colo que na oposita, o ároo é na posidilación, é a variabilidade near para a variabilidade IR, mas o ároo, como eu digo, nos permite agarrar o ároo para suportar, por exemplo, nós émos o ároo para os counters usando o ároo inicialmente, mas ao mesmo tempo é que o ároo é unha artífica, porque todo o ároo de formación é no ároo, no ároo sábado, e Niye o ároo é que o ároo é biná產, o ároo é que o ároo é fair, o ároo é bem-avidado, e o ároo é quí Wave ou o ároo é tapén dos áros, onde os árovs estão no ároo brasileiro, o ároo é f fractionidade self破da. no difenso de xaideis, o xaideis should be the same. Explicit binding is optional e, sin explicit binding, o link needs to set a binding for that UVO. Mas, now, as the names are optional, e o option means that needs to work without them, it's the opposite. Explicit binding is mandatory, so we need to base the linking on explicit bindings and forget the names. There are some layouts that are not supported, so for example in UVOs right now there are, we can only use one layout, there's a typo there, sorry. And that means that for example all UVOs are active because on the linker, on the GLSL linker there is a lot of code just to evaluate if the UVO is active or not. So that means that in the end for UVOs and others there isn't too much GLSL IR linkers to use. So the thing is that at that moment we were finding that we will need to write a lot of new code for the linker and we started to think it's really worth to ask that support for the GLSL linker, especially if we don't have a GLSL IR representation what we have at that moment is the new shader. And additionally we have other developers, in this case I mentioned Timothy Asher, but probably others, that were adding some helpers related with linking based on near. So we decided at the moment to switch and started to write a linker based on near. There I list all the regions that I mentioned there and for the people worried about, I mean it's true that there's a lot of work but the scope is really the finest because at this moment GLSL does include all the features that the current linker needs to support. The thing is that as I said GLSL has been around from 2004 so that means that Mesa linker right now needs to support features coming from more than 10 years ago. So at that moment we have a clear plan, the clear plan was greeting everything and everything based on the new linker and we focus on getting the CTS test passing. As I said, it's where covering most of the spec, specifically a lot of the corner cases related with mapping, some other things and there weren't too many. The tricky part here is that in the same way in that our implementation were a work in progress, those tests were also a work in progress. So we used a lot of, well there I say some but I can say a lot of time testing, reviewing, submitting feedback up even fixes for those tests. It reached version 21. So it was a long, long review and feedback. Especially because this review, I mean we were not the only ones doing the review. We were the ones from Inter for the Inter Mesa driver doing the review but there are some people from MED who are providing the feedback. So as I said in the beginning at the introduction we got those tests passing and that allowed us to get around October, November, I don't remember exactly when. And that was enough to get the CTS conformance test passing a lesson related with those tests. So at that moment we started to clean the code, we started to submit patches to Mesa dev, we'll mention more about that later. And the code was clean enough for the submission. So some people will start to wonder if that was enough, I mean, if we are passing the conformance test that means that everything is done. No. So we were passing the CTS test, it was not production ready. The CTS test was focused more on the, on following the spec in really on all the specifics of the specification but we're not really, they didn't test too much on this question. And probably they will extend that later but I don't know, probably they will keep the current focus and assume that if the specific for the station works then everything else is covered by the other test. I don't know what will be their plans. At that moment we have two options to how to approach things. One was mentioned, it improved Piglet and the other was working on a GLSL to speedv back end. So Piglet is another test suite. In this case was public, was open source since the beginning and it's really used a lot for the Mesa project. Okay, this is also a type, heavily used by Piglet developers, of course Piglet is used a lot by Piglet developers but what I wanted to say is used by Mesa developers. And one of the tilties that includes is Sader Runner. Sader Runner runs, it use a text file where you put the source code of the shader. The values for the uniform, for the UVO, etc. And then it also you can put what is the spetted outcome. So when you run this test with Sader Runner it's checked at the end. The main advantage of this is that it's really, really easy to write a new test. And Nikolai added support for GLSL for the Piglet test. It's mainly two things. It's a new script that parses this text file for the Sader Runner and then use GLSL to create the SPB binaries. And it also adds some... It also tries to fix the shaders. The thing is that some of the shaders as I said before the spec also includes what kind of GLSL you expect when you do the conversion. So for example, specific locations for uniforms are mandatory. But a lot of the test of Piglet doesn't include a location because are all and at the time there was not needed. So this script tries to fix the shader assigning locations and a lot of things. What they want to test. And the other thing that Nikolai added to Piglet was support on Sader Runner to load the SPB binaries. So you have an option, SPB, and then you can run... So I think I mentioned that on the... You can switch, actually. That means that you can run one test by default using GLSL and just passing one option to load the binary. And as I said, you can write test easily. For us it was a really, really useful tool because you could read one shader and run on the already working path and then just using a new option on the command line you run the SPB binary. Well, we love it. We use it a lot. It was most... Our alpha object was passing the CTS test in order to improve the... I think that these features were what we used more to get the code revolving. Well, we also added some features. The thing is that, as I said before, the GLSPB says that things should work with our names. And the thing is that the piglet, in order to fit the data, was using the open GL approach that was, for example, for UBOs that was using the name of the UBO and asked for the uniform and then with that index started to say that where is everything and put in there. But with the SPB we don't have names so we needed to change how to fit the data for UBOs and SSBOs. Well, no, sorry, for SSBOs was already in place, only for UBOs. And finally about all the work on piglets both for Nikolai and both for us, it is not clear if it is going to go upstream. The thing is that some people when it was debated in the past having some doubts about changing the self-runner and especially some people have doubts about adding a dependency on piglets for GLSL lang, for different reasons. The other approach also mentioned that it was first mentioned on May was create a GLSPB package. The idea will be that Mesa itself will include a way to convert GLSPB and in a badness will be provide an alternative to GLSL lang that right now is the main way to get GLSL shaders to SPB. And just in case some people think that is just a chrono tool for Amateus, I think that for example for the last version of the Doom game I think that they use GLSL lang to convert from GLSL to SPB. The other advantages were optimizations in the sense that the GLSL compiler has several optimizations so the idea will be, sorry? Right now this is not a proposal because as I have said it's not as easy as just the switch because also how you feed that is different. So at least the linking will be I mean all the test that involves linking we could do this as you say running it twice. But yes, I mean there is a lot of details about how this testing will work because it's not just testing running it twice. In fact one of the things that Nicolas was doing that I didn't mention that was also creating a kind of blacklist. I mean this and this and this test run that so if extended that approach the idea will be running twice only the white listed test. But yes, the testing is still the testing right now is mostly focused on getting the extension the developing of the extension going on and at some point we will need to define a better policy. In fact is what I said before it is not clear if all this change on piglet will be over stream because with this approach you don't need change on piglet to get the speed binaries. Because as I said on the next slide well it's not in the following one. In this slide the idea will be that internally I don't know probably a environment variable you will ask Misa how do you pronounce Mesa because we are Spanish and we pronounce Mesa but probably Mesa is ok. So that internally I don't know environment variable like use SPB that the normal change will be GLSL IR and then NEAR but internally create a GLSL IR SPB and then SPB to NEAR In that way there are two things that will be tested SPB to NEAR and also the support of this extension. The thing is that internally you will also need to do this phishing that a piglet was doing. This say doesn't have a specific location so internally we create a specific location for this test. That will allow to run the piglet test without change but the way to fill the data will be still different. I don't know probably in the end there will be some kind of compromise something like don't use GLSL to create the SPB about use this approach and then make a small change from other runners to fill the data to the test. In any case in relation why if people ask if we are using that it's not finished and for example one of the things that is missed because it's not finished in the same that in the previous slide Iron Romantic send the first version October and some people started to review that but it's not finished and this phiseries is not still fully reviewed and it's not still merged and it's also a work in progress. For example one thing that it was missing was UBOs so in my case that was working on that I only had I need to rely on piglet. Ok so I don't know how much time I have Ok ok Sorry if I was speaking too fast but it's what happens when you in the presentation. So ok we're working right now as I said we got all the CTS test passing and then we have a little everything partially covered. We have uniform, we have atomic counters, we have UBOs, we have distillation, we have transform feedback so we have well when I say plenty of programs I mean obviously tests because the current support is not enough for a real application and this small extension that is about the spore stations is fully complete but that was really really small. So what's missing from for all these filters it's missing to polish them, to test them it's also missing a rise of a rise that is the way that OpenGL calls multidimensional rise we have some support but we have still something to fix on the SPIRV to near pass we miss multi sample it must arrive that we need to add more validation the thing is that the spec says or spec that SPIRV binary is correct but spec that is correct each model but at the end we will need to make some kind of validation when you are using different models each model for each stage and obviously it is more testing and more testing means not only adding more testing but as we were just mentioning before agreeing on something I mean agreeing on what we do, on the policy or while listing etc it will be it will be complicated I mean it will be not trivial for the plans of streaming right now we have around 80 patches more or less and the plan is sending small patches now and then we send a initial one on November I don't know it was like 20 or something patches it's partly reviewed but it's still pending at the final review we send the 4 version in January and I think that's all I hope not forgot something so you have any question and no, no, no sorry we were really focused on just only the openGL but yes I know that there were some threats some misadepth related with OpenCL but I think that they are still using LLVM but yes, I know that it's curious that there are several people working on a kind of SPB support but yes, right now we are only focused for the OpenGL and in fact as far as you know the OpenCL is really out of the OpenGL support so I think that in the end probably there will be really different in both OpenGL and OpenCL I think that the support will be somewhat different it will be awesome to share some of this but and at the same time as I said before this extension defines the scope for OpenGL so I'm not sure that the OpenCL needs will be the same even after the SPB I'm not sure that the needs will be the same so you were going to be and so you are ok, thank you for the initial work but it would be somewhat rude to say that on a presentation this is the reason we use so much the self-run writing a test with self-run it is really easy one minute to 30 seconds if it's really small but with CTS it's like the other advantage is that with self-run it's just modify a test file and you run it with CTS you have to write the C++ code compile and is to add some support to CTS to this so in video you should so you should build it ok, it's good to know the idea is like well let's just run it twice in a different mode so like the shader cache is another thing that has no specific piglet test and the testing strategy is to run it once and then run it again with the shader cache enabled and you know like if you with piglet you can't short out the tests in any way to scale it to lots of machines and it's unreasonable for developers to wait for two and a half hours to touch anything in any case just one thing we are talking about this running twice we internally are adding tests and we are using internally but we are we internally we are not doing this executing two times other tests we are adding tests internally for us so what I mean is that probably it will be the two things probably when we get this test material enough and if they approve the GLSlang or SPIRV tools on Pili that I don't know if they are going to we will add so I don't know let's say about one hundred tests specifically for SPIRV so the idea of running twice the others will be extend the coverage so what I mean is that or at least is what I would like to do is having some tests only for this feature to test I mean it's impossible to test everything of it for example I have like ten ubo tests that probably could be go there I mean it's a ubo test it's a basic one, one with an array one with two ubo and I think that makes sense to include them and not just focus on the other extensions because as you say that would be problematic I mean because those tests for example I remember that when I was working on the SSBO I started using SSBO tests but those tests were not using special locations so I needed to use that ok for the first step we will do that but then I will write my own tests just for me so I think that in the end probably would be a compromise well at least I would like to be something like that ok so any other question ok so I think that's all thank you