This session, hello. Thanks everyone for coming, especially for your last session of DrupalCon this year. This session is the couple of Drupal with Sylex. If that's not what you're here for, then one of us is in the wrong room, and I don't think both of us are wrong. So my name is Larry Garfields. You may know me online as Crell. If you want to make fun of me during this session on Twitter, that's where you do so. I'm a senior architect with Palantir.net, where I work at a web development shop based in Chicago in the United States. We do mostly Drupal work, some symphony work as well. For Drupal 8, it was the Web Services Initiative Lead, Drupal representative to the PHP Framework Interoperability Group. It's kind of the United Nations of PHP with all the positive and negative connotations that has. Advice of the Drupal Association and general purpose lovable pedant. So that's me. My name is Hagen Nass. I'm a senior solutions architect at Uyala. Not as many titles as somebody else has. But let me tell you a little bit about Uyala. We are one of the leading online video platform providers. Company got founded in 2007. So we've been around for a while. We have by now over 300 employees worldwide. Global footprint of about 200 million unique users in 130 countries. So we do know video. And we're working with some of the biggest and most successful broadcasts and media companies in the world. Short list of those here. I'm sure that some of you have watched video on at least one of those sites. For this specific project, we wanted to build a kind of management system for an OTT solution. OTT, good question. Over the top, video solution. So I think Netflix type video service. Some of the features that we gave Palantir to develop were that we want to support a really structured, well structured data model for the metadata for the video. We of course wanted the UI to manage the actual metadata. We wanted to implement a workflow for the publishing of videos. It's very important because the quality of data that you typically get from the studios is not all that great. So there's at least two steps in that specific workflow. First a QA workflow, a QA step, and then the actual publishing to the service. We wanted to curate content as well. Some of it like lists and collections that could be displayed on the applications and websites, banners for the websites, and then some application CMS kind of aspect came in it as well, editing the home page, and then list of lists for other parts of the website. And not just of, yeah, in addition, sync the data with UYALA's existing DAM, Digital Asset Management System, because we need some data in there as well. And on top of that, APIs that work at scale, because we didn't just want to drive websites, but also applications on different devices. The service had to support thousands of movies and TV shows, tens of thousands of users, and as I mentioned, you have websites and applications. So we'd worked with, Palantir had worked with UYALA before, and they came back to us and described this project that they wanted to do for one of their customers. And we said, you want us to build Netflix in a box in Drupal? Cool, sounds fun. So we put together our team for this project. Our project manager was Amy Dosimo, our senior engineers were myself and Robin Berry, and then Beth Binkabits and Beck White were on the engineering team as well. One of the complexities for this project was that there was no front end. We're not actually building the user-facing part of the system in Drupal. We were not even building the user-facing part of the system period. We all wanted us to build a content management system that served a REST API, and that's it. And then web application, mobile application, whatever could consume that data. And we looked at this case, all right, so we've got data that we need to bring in and do content management, type stuff on it, and then serve it out as a REST API. There's a system that is really, really good at that. You know what that is? Drupal 8. Except we were saying this in early 2013, so no, there wasn't Drupal 8. All right, so what do we do? This is a bit more than our usual, oh good, let's just throw up a Drupal site kind of approach. So we looked at, all right, we could use Drupal 7 with a services module. Problem, services is not actually a REST system. It's an RPC type approach, and architecturally it's fighting against Drupal 7, because Drupal 7 architecturally wants everything to be a page, and so it's not gonna be that performant. It's gonna be kind of clunky, lots of moving cars that don't need to be there. There's the REST WS module. Who's used REST WS? A few people. In early 2013, it still was very new and kind of experimental. We didn't want to do that much with it. But it has since become kind of the model for Drupal 8's REST module. But at this point, when we built this project, it wasn't a serious option. Another option we considered was, let's just do it ourselves and go all symphony with it. At that point, we knew that Drupal 8 was adopting a lot of symphony components. Symphony is a good framework for doing custom bespoke applications. Why not use that? And we spend a lot of time wrestling with that question of, is Drupal even the right platform for it? And what we came down to on this one was, do we want to spend the extra time ramping up on symphony? Do we want this to be our first symphony project and learn all of the piece of symphony and have to build the UI from scratch? Drupal does give you an awful lot out of the box in terms of content management. That when you start using some other system, you realize just how much you're missing that you just take for granted in Drupal. And we kind of wanted to take that stuff for granted in Drupal. And then we looked at another option, Sylex, who's worked with Sylex before? Good number of you, okay. So you probably, those of you can probably see why we're looking at it. It's kind of symphony junior. It's a lightweight system built on the same the same components as symphony and as Drupal 8. So a lot of commonalities there. It's actually very fast. I did it some quick benchmarks for serving a rest response, just a little hello world level rest response and Sylex was three times faster than Drupal 7. And that's before we did any optimization of any kind. Downside there though, it's a micro framework. So it would do even less for us out of the box than symphony would. And so that's even more we would have to do ourselves. None of these looked like the right platform to build a system on. So all right, let's combine. Drupal 7, its strong advantages are as a content management system and displaying pages and displaying administrative screens. And it's really good at that, but Drupal 7 is kind of lame for rest services, truth. Sylex on the other hand, that's the Sylex logo. One of the nice things is they both have kooky eyes. So they work well together as projects. But Sylex is really, really good at HTTP handling. It's raw symphony, which is a very strong API for dealing with web services. The downside of course is that any kind of UI you want is going to be hand rolled artisanal from scratch. You're responsible for every tag in the markup. So let's just combine them because leverage the right tool for the job. There's a lot of jobs here, so let's use multiple tools. So how do we get these talking to each other? If you're talking about Drupal 8, instead, they're all using symphonies so you can stack them, but we need these to talk to each other somehow. So how do we get data from Drupal to Elasticsearch to serve the API? Well, Drupal's data lives in Drupal and it has its own very specific structure. But the structure that you need for editing is not the same as what we wanted to serve on the API. We had, you want to have editors edit content in these three chunks, but serve them as one chunk to end users or vice versa. So instead we decided let's put Elasticsearch in the middle. Let Drupal control Drupal content and let Sylex do APIs and the only place they have to talk to each other is in Elasticsearch. So who's worked with Elasticsearch before? Good number of people. Who saw the Elasticsearch session here at the conference earlier? Completely different set of people. So now everyone understands Elasticsearch now, right? Okay, this was actually the first time we used Elasticsearch, but from what we understood it's the same underlying engine as Apache Solar which Drupal uses all the time, it's a Lucene engine, but its API is a lot easier to work with than Solars. So if you're doing custom developments, we have found, we have done custom development on solar projects as well, and we found Elasticsearch just way easier to develop against as people fighting against it. The JSON API is a lot easier to work with the fact that you can mess around with a schema for it without restarting a server and re-indexing all data was very nice. If you're doing custom developments I recommend Elasticsearch as your search server. It's just a lot easier to work with. So we ended up with this pipeline then for the system where we'd have some kind of incoming XML from the customer that says the definition of some movie or TV show or whatever, the metadata that we need to be managing. And we import that into Drupal. And content editors in Drupal could then edit it, update it, make sure that nothing is misspelled, make sure the image is the right image for this, the right poster image for this video. You really don't wanna put an R rated movies poster image on a kid's movie by accident. That's just a bad idea, trust me. Drupal then would dump that it's data into Elasticsearch where Silux would read it and serve it out to whatever the API consumer is going to be, phone, website, whatever. All right, there's a lot of moving parts here, but we've really just broken it down into three sections. So our data model itself, I'm not gonna go into too much detail on, but this is Drupal node modeling. And you've all done this a dozen times. In our case, the major content to be at programs, go to the record for a movie or a TV show or whatever, the asset, which is the wrapper around the actual video file. We were not storing the actual video files, that's what Yala's back lot service was for, and we're really just doing the browsing system for it.  And then stuff like collections and lists like Hagen mentioned for other curatorial stuff. And again, I'm not gonna go into too much detail, it's just we had bunches of nodes. So where's the data coming from? We're not entering it manually, because that would be slow. Instead, the customer's sending us contents in XML form. And that XML looks something like this. Please don't be scared. You don't, I don't want you to understand all of this, but the thing I wanna call out is this one XML file includes several of those content types wrapped up together. So like most of this is part of the program. Something down here in the video, this is actually the assets. And we also have the exhibition window here is what helps to find the offer. And we also had device filtering. So certain devices can access certain content. So we had to work that in as well. And all of that was coming in as one big XML file. So how do we break that up? So importing process, what are the options in Drupal? Well, there's the migrate module and problem. Migrate's designed to run user triggered like from in Drupal, you push a button, or you wanna draw scripts or something like that, which is great for the use case it's designed for, but we wanted to accept pushed data from the customer. We wanted them to be able to push ingested data to us so that they could just keep throwing data at us and we'll consume it. So that wasn't an option. We thought about feeds. Problem is feeds really wants to map things to one object and we needed to map to multiple objects. It also has an awful lot of moving parts, some of which are better built than others. We have worked with feeds in the past for Drupal to Drupal communication, but we didn't want that overhead on this project. Services again, kind of clunky, not really restful and we did wanna add more and more layers to the project that we needed to. So went custom with this and it ended up working out really well and by custom I just mean a custom module. So we just parsed the XML with a library called QueryPath. Who's used QueryPath as a Drupal module for it? Wow, I'm impressed with a lot of people. Okay, so for those not familiar with it, I think JQuery for PHP.  And with that we had a little import engine we built using just standard PHP 5.3 tools. So if you wanna get into the developer part of it, who's seen my functional programming talk before? I've given it a couple of Drupal cons. The example I use out of that for the import engine is from this project. So for those of you who recognize that, that's what it is. And that gives us when XML file comes in, bunch of nodes come out, we call node save. And nice and simple, even testable, yay object oriented programming. So we've got that import engine, wired up to Drupal. And yeah, we said Drupal 7 is terrible at rest, but that's okay, all we need is posts. So we're not actually doing rest for this part. So we just had a couple of custom page callbacks, straight up Drupal behavior, Drupal API, to just shuffle data from the incoming request to that importer. And we did add SSL authentication for it because we didn't want anyone in the world to be able to shove data into the system, would it thunk it? So it's just simple HTTP auth module, I think that's the correct name.  Just for selected paths. Great. So we've got all of our data, we've got all the data we're gonna need. Almost. Problem is that that XML file I showed, is often missing stuff because the customer in this case had an awful lot of data that wasn't very good. So we've got to fill in this data somehow. How are we gonna get replacement data? Do we make someone just sit there and edit it all the time? And they have to come up with a new synopsis for these movies? That's a waste of a human being's time. So instead, we need to talk to a third party service and bring in some kind of third party data. And in this case, the customer contracted with Rotten Tomatoes, the website. If you know IMDB, they're kind of like that. But they have actually a really nice API to get at the content in their system for movie information, reviews on movies, other information like that. And they do offer, it's a free API for a certain level of usage and paid API over that. So the customer paid for a paid API, great, swap the credentials in and start talking to that. So to actually talk to it, we use the Guzzle library. Guzzle, for those not familiar, is an HTTP client for Drupal. So who's used Drupal HTTP request in Drupal 7? Okay, it no longer exists in Drupal 8 because it is the second most impossible to debug and maintain piece of code in the universe. And there's actually been measurements done of that and its systematic complexity is obscene. So we didn't even want to touch it. But Guzzle is the library that Drupal 8 is using. It's a third-party PHP library in Drupal 8 just pulling it in, great. And so we said, if good enough for Drupal 8, it's good enough for us. So brought that in. And then wrote a standalone Guzzle library, a Guzzle extension, to talk to the Rotten Tomatoes API. And this is a link, it's actually a standalone Guzzle extension that we've released on GitHub. It's just an open-source library that lets you talk to Rotten Tomatoes and just load movie objects out of it and review objects out of it, navigate between them. Just nice little wrapper around their API. And then a Drupal module that we've also released that leverages that and provides a mapping from their data to Drupal. Solve one problem in each of these little pieces along the way. What this means is we can then say, all right, we've updated this movie, linked it up by its IMDb ID because Rotten Tomatoes uses the IMDb unique identifier as their unique identifier because monopoly, I don't know. And so when we save a node in Drupal, it has, it's a movie, it has an IMDb ID on it. We go out to Rotten Tomatoes, fetch its data and just map that into a parallel node in Drupal. So now we have the Dark Knight, the movie from our customer, and the Dark Knight, the movie from Rotten Tomatoes sitting in our database. And then we can periodically refetch it if we need to to keep data fresh on Save or on Cron and various business logic around that. But the system was set up in such a way we can vary that whenever we want to. And then we also brought in reviews. So someone using the API wants to look at a movie and okay, what are the reviews on it? How good is it? That data is coming from Rotten Tomatoes. Okay, so now we've got all this data in Drupal. What are we gonna do with it? We edited it and Drupal's good at that, right? Almost. Drupal's node edit page, especially in Drupal 7 can get very large and unwieldy if you have as many, as complex a data set as we had here. So we had to do a bit of custom work on this one. So the main admin interface, this is straight up views. You know, if you're doing any kind of custom work in Drupal, you wanna build a custom admin views as your go-to tool. You rarely even need to theme it. So in this case, we're showing all the data that we imported. So there's the one of the offers. Here's this corresponding program. Yeah, so this is the movie Brave from Disney and the other information for it. Somewhere down here is the actual assets. And we've got QA state, published review state. These are just fields on the node as well. There's a lot of data in here. It's gonna be thousands of movies. So fortunately, views offers really nice filtering. There's really nothing custom going on here. This is nice and simple. Separate page to show the in Drupal content. So collections that people create, our custom lists and so forth. Banner images and stuff like that that are gonna get shown. The fun part is the actual node edit experience, which this is not what your normal node edit page looks like. What we wanted to do was offer not, here's a dump of the entire huge node, but here are the pieces of it and how they relate to each other. So in this case, we're showing the overview for the dark, wrong button, dark night program. And we've got some information on it here, but also here's the curatorial status of this data. And here's the offer it relates to and here's the assets it relates to. So an administrator can add a glance, see, all right, this whole movie in total is ready to be published. It's ready to be approved. Then we have an overview page. I made the screenshots recently, so there normally would be a preview of the video here, but my current dev copy doesn't have a connection anymore. So imagine there's a video screen here. But you can see when this was imported, you get the current QA state. So certain users with certain roles can approve it. That's just normal Drupal access control, like we're all used to. Put in a log message for why it's being rejected. We had some custom formalters behind the scene to say the log message is only required if this field is failed for some reason. Then the actual information itself shows below that, including very sized thumbnails of the actual poster images. Those are all just being created by Drupal image styles. And that's actually the images we then serve out to the public. And it goes down further. This is the overview page. And this is all built with panels. So we did all of this custom UI work with panels and a couple of formalters, essentially. It's a bit more than that, but when I say a couple of formalters, I'm sure a few people are chuckling at that, yeah. For editing then, we broke it up into several different screens. Rather than one gigantic edit page, you can edit different parts of the record, different parts of the node. So in this case, we're editing just the credit information. So that's the, who's the director, who's the writer, who's the actress on it, and so on. And so we've got a QA state for the record. We've got the switch between the different credit sources. So on this piece of content, I trust the incoming data from the customer's data source.  So I'm going to use the information from Rotten Tomatoes instead. And you can make that decision on a per node basis. And it's really just saving a Boolean or a toggle in the database for now and showing both the incoming data and the Rotten Tomatoes data. Then you can edit the generic content. There's a couple of other fields. Again, built with panels. This is a very long page. I'm not going to bore you with the whole thing, but you can see all this metadata and information that's in the incoming data that they want to be able to expose in the API and then build whatever filtering around in client applications. Synopsis again, same idea. Use the incoming data or use Rotten Tomatoes. And you can make that distinction differently from the credit data or the other metadata. And again, for now, we're just storing a flag as part of the node. It's just a field. I said this is built with panels. Who's tried to override node edit forms with panels? It's painful, isn't it? That's why I say there were a couple of form alters. I'm papering over an awful lot there that I don't want to go into too much detail on. It would bore most of the room. Most of it came down to doing black magic around the required fields. So they were required only on certain pages because Drupal 7 really, really, really doesn't like you doing this. So these are all panels using node edit as a context. This is actually right around the time while we were working on this project that the concept of form modes went into Drupal 8, which is basically this thing, this concept built into core out of the box as a fully supported feature. So if you want to do this kind of interface in Drupal 8, should be a lot easier now. We're putting this thing to Drupal 8, Hargan. It's unfortunate. This is what editors actually see. And again, we just threw the seven theme on everything. There was no front end. We had no themers on this project. Who was in the developer versus themer session earlier? Few people, yeah. So we bypassed that fight by not having any themers on the project. Okay, so we've got the data and we've got it curated. We've got it edited. We know what we're gonna do with it. Now what? Now we get to the decoupled part. This is headless Drupal in a sense because when a node is ready, we dump it out to elastic search. Problem, this actually takes a while because as we said, the data structure we want in Drupal is not the same as the data structure we want in the API. Do we want to reformat that data every time it's requested by a user? Absolutely not, that would be horribly slow. So instead, we want it to do, back up a moment. We also, the contributes module support for elastic search in Drupal in 2013 was rather poor. It has since gotten a lot better. There's modules that have been released since then that actually use the supported API libraries from elastic search, the company. And we looked at rules to do this preprocessing and again, too many moving parts, not enough places we can hook in and do, I just want to write some code. A lot of this project was, this is easier if I just write code. Fortunately, Drupal lets you just write code. And we also have this interesting problem where sometimes when you export one node, you have to export some other node, but only conditionally. So you export a program, you have to also publish its corresponding asset, but some pieces of the asset, we won't move on to the program and some pieces of the program we just delete. So we had to have somewhere to do this kind of logic, including merging in that Rotten Tomatoes content. So if you're exporting a program and it's toggle is set to use Rotten Tomatoes for the source for the synopsis, we need to pull in the data at that time, which means loading up the Rotten Tomatoes node, pulling the data off of that, putting that into the information we're exporting to elastic search instead of what's in the node we're about to export. Complicated, right? Great. Another little custom system. It's actually not that hard to do in Drupal, if you go object-oriented with it, because then you're not coupled to Drupal and, you know, object per node type. This ended up being a very simple, very robust way to handle it. So, alright, now when are we going to export content? What are the publication rules? Do we want to export content before it's been approved by an editor? That would be a terrible idea. So let's not do that. So we made the decision that if a node is published, that means it's going to be public. It's ready for public consumption. And if it's ready for public consumption, we're gonna push it out to elastic search. So, if it's published, we push it to elastic search. If it's unpublished, we don't push it to elastic search. Nice and simple. So when is a node ready to be public? When it's been approved? Well, it's a bit more than that. A program is publishable when it's been approved by an editor with some caveats. An offer is publishable when it's been approved and it's within a publication window because we may only have a license to sell this movie during the month of May or only in certain regions and so on. And so we only want to publish it within its publication window and maybe a few weeks before that so people know it's coming or maybe a few days after. It's more complicated than I'm making it out to be. For an asset, the actual information about the video itself, when it's been approved and it has an offer that's been approved because if there's no offer for it, no sense publishing that information. Or the offer is within its publication window because then we can talk about it but you can't actually buy it yet. It's complicated. Let's just go with that. And we don't wanna do complicated when people save nodes. When you save a node, its publication status may change and that's a terrible, terrible time to do all this complicated logic because then we slow down the UI, the editors are using the site and have to sit and wait for two seconds every time they hit save as we try and figure out if the nodes should be published and if so which other nodes that we publish with it and that would be a terrible, terrible thing for the user. So let's not do that. Instead, we used Drupal's queue system and Cron. So they spent all of that hard work of computing, do we need to publish this thing? Do we not need to publish it? What else do we publish? How do we reformat the data? All of that only ever happened in a queue. That's the only time it happened which means it's completely out of ban from the user. The user responsiveness is like that because all we're doing when you save a node is tossing an item into a queue to say by the way, check this node when you get a chance. So on Cron, we would say find nodes that we think are ready to be published based on their time windows and so forth, SQL queries. And find nodes that because it's not something else it's about to be published. I gotta get that one. Throw it all into a queue. Do the same thing with nodes that are about to go out of their window. Just find them, toss them into a queue, come back to them later. This is the exact same model that Core uses for things like the aggregator module. And I think Drupal 8 does it even more where if you have hard work you wanna do but it's the same thing a lot, use a Cron hook to say, all right, it's been an hour, it's been a half hour, whatever. Grab all of the things I need to update, need to check, throw them into a queue, and then just let the queue figure it out. And the queue will just churn through them at whatever rates it can. And update each feed, publish each node, whatever. And then, those will just toggle to publish and unpublish states if appropriate. That triggers a node save. Node save triggers another queue that says, all right, we've just saved the node which so go do the pushing to the elastic search you need to do. And then if it's published, we do all that fancy processing to reformat the data and merge content and so on and so forth and send that to elastic search and if not, we delete it. That way, again, the hard work is all happening here and here, never in node save. Node save becomes cheap. We have these sets of run very frequently so I think it's like a five minute time. If you want, you can even use a running queue module, we didn't hear, so queues are just all these processing. Another aspect of this is it's quite possible if you save a node and then edit something and save it again a minute later that a node will end up in the queue multiple times. That's not actually a problem in this case because what's the worst thing that happens? We delete a node from elastic search twice. Elastic search doesn't care or we push the same node to elastic search twice. Elastic search doesn't care. So overwrites and overwriting to elastic search was a non-issue, so we just ignored that. We don't care. If we process something multiple times, we don't care. So now we're content in elastic search. Now what? Now we get to Sylax. Now we get to the non-drupal bits. This is just serving an API, just serving a REST API. So there is no user interface. We didn't actually get to play with Twig on that project, I have to say that. Sylax is a very, very simple lightweight micro framework. It gives you a routing system, some glue, and the ability to register callbacks and that's about it. And that's all we needed. Everything else you build yourself and so we built it ourselves. It's using the same core pipeline as I said before as symphony and it's Drupal 8. So we already had some familiarity with that based on watching Drupal 8 happen.  Sylax, you add what you want. It's a very minimalist system out of the box. So we added Elastica, which is an open source PHP library for talking to Elasticsearch, which is the same one we used in Drupal in fact, Guzzle to talk back to Uyala's Backlot Dam on occasion when we needed to. And this HAL library. HAL stands for Hypertext Application Language. It's a JSON spec, it's an ITF draft currently. It's a very simple extension to JSON that provides hypermedia links. So you can actually build a RESTful API with it. It's actually really, really nice. And it's what Drupal 8 uses out of the box. And it's what like Zen Dapagility uses out of the box. It's really becoming quite popular. And there's an XML favor of it, but I don't know anyone that uses it. Fun little story actually. February of last year of 2013, I was at another conference in Miami. And I was going to the onsite with Uyala as soon as I got back. While I was there, Lynn Clark, who was from the Drupal 8 REST team emailed me to say, Larry, this thing we've been trying to do in Drupal 8 using JSON-LD is not gonna work. I suggest we switch to HAL. And at Sunshine PHP, the conference, there was a presenter talking about building their entire company on HAL and showing some of the benefits of HAL as a format and the tool train around it. And I came away from those two inputs saying, so, Hargan, this API building for you, let's use HAL for it. So we used HAL on this at the exact same time that we were switching Drupal 8 over to HAL as well.  So what are we doing in Sylex? Again, I'm not gonna get into code, but it's a very, very simple pipeline because Sylex is simple and all we needed to do was suck data out of Elasticsearch and serve it. And that's it. Pull data out, format it, and serve it. So we've got the routing system that happens in Symphony or in Sylex or in Drupal. Load data out of a little repository wrapper, rebuilt around Elasticsearch, get that data back, convert it to a HAL object and return it from the controller. And then in a view listener, which is the kind of post step in a Symphony application, we take that HAL object and convert it to an actual JSON string. We actually support XML2 on it because it took an extra three lines of code to do that. And did all of that hard work there. And also HTTP caching. Fun fact, we're actually moving to this exact same model in Drupal 8 of doing, I'll return an actual object here and then do the hard work in the view listener. That's what Drupal 8 is gonna be doing internally. Based in large part on our experience in this project, convincing us that this was a good way to do it. So this played into Drupal 8's architecture in fact. And then all the HTTP caching is easy to do in Symphony, they've got the tools there for it. We don't do any caching at all in the Sylex application. There's not one bit of caching in there. HTTP takes care of everything. One of the nice things about HAL, I'd say the best thing about HAL, is that there's this nice HAL browser available for it. This is a simple downloadable single page app that lets you browse any arbitrary HAL API. As long as it's well-structured, you can browse through the API entirely. So this is the index API, so a client application only ever hard codes this URL, domain name slash or domain name slash API or whatever it's going to be. And then there's various links that you can traverse just like a webpage to get to some other resource. This is how you navigate around a REST API. This is the proper way of building REST APIs. Some of these you need to do a device API on, like we said before, a lot of other complexity I'm just gonna wave over for the moment. But if we follow the link to a program, this is what a program looks like. And the full response that comes back is it's over here. This is the actual structure of the content itself. It's just a blob of JSON, cleaned up, formatted the way we want. Some bits of it look like Drupal, some don't. That's fine. This is something we negotiate with the team that's doing the front end. You can have a look at all of the headers come back for debugging, like in this case we're in dev mode, so there's no caching turned on. In production, we turn on caching. And from that, you can actually just follow the link to a person who was on this program, an actor who was in this movie. So in this case, we're following it off to Heath Ledger. And we have links back here that you can traverse back to movies that person has acted in. Which means in a client application, it becomes really, really easy to say, all right, I wanna find movies that this actor was in. So I look up that actor object and I see, oh, here's the relationships to movies that this actor has been in. I will go get those, it's nice and simple. Navigating an API as if it were a webpage by just traversing links around, that's how you do a proper REST API. That's actual hypermedia. And of course, we wanted this to be highly available. We wanted this to not crash. Wanted it to be fast, wanted it to be stable. Fortunately, the Sylex app is pretty much stateless. It has no state of its own other than the Elasticsearch server. So you spin up three or four of them, tell them where the Elasticsearch server is, and you're good to go. You've got three, four, five web heads that you can load balance very, very easily. And they are themselves very lightweight too. Elasticsearch as well is almost disturbingly easy to set up in a replication mode. You just spin up multiple Elasticsearch servers, give them the same ID, and if they just work themselves out. So in production, we've got at least two Sylex heads, at least two Elasticsearch servers, and I believe we do have two Drupal servers. The Drupal servers are completely behind a firewall. It is not actually possible to get to them from the public web. The only thing you can get to is the Sylex servers. Yeah, we actually had, for Elasticsearch, one of the admin systems that the client kept asking us, so how do we set up replication in Elasticsearch? And we said, you turn on a second Elasticsearch server and give it the same ID and it figures itself out. So how do you set up clustering with Elasticsearch? You spin up multiple servers and it figures itself out. So can you provide this documentation for spinning up clusters in Elasticsearch? You spin up multiple servers and it figures itself out. It was too easy for him to be able to grasp. Systems that are too easy to use for a person to comprehend is a good sign.  We're using HTTP caching. Varnish is an HTTP cache. Great. Everything we're doing is read-only, so everything we're doing is read-only, so it'll cache just fine. If it's a little bit stale because of updates from Drupal, we don't care. Five minutes, 10 minutes, time window is fine, which means almost everything's getting served out of Varnish. Varnish is blazingly fast. To the point that I think we just have the one Varnish server and that's not even load balanced because we don't need it to be. Or is it load balanced for redundancy? I'm not actually sure. No, performance was never actually an issue for us. So that's what we did. That's how we put this thing together. So actually, how are we gonna, a year on now since we built this thing, how's it working out? Really well, actually. The service, the service launched on web for desktop only shortly thereafter, they launched applications on Android and iOS as well. All running against the same API. Just recently, they launched Chromecast also. Again, without any changes to the actual APIs or the backend of the service. Yeah, unfortunately, we couldn't involve Palantir in building the website, maybe for the next project. And the service is growing. Highly reliable, Larry told you all that's wanting it. I don't know of any outage that actually happened, so far not a perfect record. Library of thousands of assets, they're about to launch TV box sets so they're adding high volume of TV shows. Thousands of subscribers, maybe not quite the numbers that they wanted yet, but steadily growing and again, no issues in terms of the service running. We're also still adding new features like I just mentioned TV box sets that require a minor change in the data model. But the architecture in general held up. Modular, it's easy to add and change things. And that's it. All right, so thank you. We have some time for questions. We can go deeper into the architecture or not as deep if you want. Please use the microphone here in the middle of the room so it gets recorded. And what do you wanna know about this project that we haven't already covered? Here's somebody. I was just wondering what is the latency from pressing save to the content actually being available in the front end? Worst, I think, it depends on what you set your cron time to be and what you set your HTTP cache to be. I think, as I was last on the project, they were both sets of five minutes. So 10 minutes is your worst case scenario. Set that down to a minute in a minute, then two minutes is your worst case scenario. So you have no problem processing all of the... No, no, okay, cool. And one of the advantages of Drupal's queue system is that it can be run synchronously. So if the cron got way too big, you spin up, just run drush queue run or queue process, whatever the command is, three or four times in separate windows, and it'll just burn through the whole thing and shove it all out. So when they get to processing that much data at once, spin up more queue workers and you're good to go. You mentioned that you made the imports from XML files and I'm wondering if you had some tests made for those and if you had, how did you make the tests for testing if the import is well done from the XML? The way you would test that is just have sample files on hand and run them through it and make sure it nodes save at the end. Drupal 7 really doesn't make unit testing easy. So I think we just did integration tests, level testing on that, but we did have a bunch of sample files lying around that we could just run import on, make sure we got the nodes out the other end we want. If a new XML file is provided from the customer, which happens several times with slightly different format, then we've got the tests there already, tweak those for what we expect for the new content, tweak the mappings, keep running it and you're good to go. So Drupal 7 means you do integration tests of the simple tests and I wish we could have done better, but Drupal 7. I had the same, that's why I asked. Yeah, thank you. In Drupal 7, that's pretty much all you can do. In Drupal 8, you can wrap most of that up in PHP unit tests and then just do a simple wiring test at the top with a simple test. I was wondering what the added value of elastic search was in your case. It's for search, kind of search requests because as you were caching everything anyway, you could maybe have used the primary database as well. So we were actually doing keyword searches as well and for that elastic search was the obvious choice and once we had that in there, it made sense to also use that to hold the denormalized data. We technically could have just used HTTP cache for everything. Again, problem there is anytime we pass through the cache, it's gonna be very expensive for that call and whoever that user is, well, it sucks to be them. We didn't want that. It also kept the systems decoupled. So the Drupal site can go down. We can take that offline to change the data model there and as long as what gets pushed out to elastic search doesn't change, we can just leave the asciatic servers untouched and technically we can even completely replace one or the other and we just have elastic search in the middle. Yeah, it's layers of separation of redundancy as well as we actually were doing searching. Thanks. Did you care about invalidating the varnish cache or? We talked about that? We actually... I'm not sure if they actually implemented it. They were asking for it and we told them how to, but... Yeah, in practice, the lifetime was short enough that the customer didn't care. You certainly could do, push an invalidation command to varnish if you wanted to. I think we had a user story for it. They just never wanted it because their timeline was short enough, but you certainly could do that. Yeah, so that was the other question. I had the agile approach you mentioned in the description of this. Yeah, this was actually the first pure agile project Palinter did and it worked out really, really well. Probably the most important thing for an agile project is having a product owner that is active and aware and able to ejaculate and then adjust on the fly. And we had a really, really good product owner on this project. What you're trying to say is I couldn't make up my mind and change things all the time. Yes, but you were okay with that. And so we had the Rotten Tomatoes integration that we had talked about a little bit, but didn't actually figure out until halfway through the project. And that was okay because our contract and our product owner were structured in such a way that we could do that. We also had handling around genres for different assets, which are far more complicated than you would ever expect them to be. But we had the same problem of we've got genres that come in from the source data, we've got genres that come in from Rotten Tomatoes and sometimes one is good and not the other and you wanna change it. So the incoming genre is called action adventure, but you wanna just use the word action so you have to have mapping logic. We figured that out halfway through the project, we just paused, all right, let's just stop and take two days and figure this part out rather than all that upfront design. So it actually worked out really, really well. This is a project that convinced me that Agile can actually work. Was there any consideration for using Node.js instead of Silux? We talked about it briefly, but we're a PHP shop, not a Node shop. And so we would have had to learn a completely new stack in a language that we spend less time in. And that wasn't worth it at that point. Silux was a shorter jump for us in terms of the technology stacked to learn. You certainly could do this kind of work with Node.js or something with Rails or Sinatra or something in Go or whatever you want on the other side of Elasticsearch. And that's kind of the point that that's independent of the Drupal part. So we could totally junk that entire Silux piece and rebuild it from scratch with cool technology to your and nothing else needs to change. It's just a different thing, pulling content out of Elasticsearch and serving it. We'd be fine with that. It was the time to support that kind of evolution. But yeah, the initial decision for Silux was we're a PHP shop, so let's use PHP. Guess that's it. So thank you everyone for coming. Hope you enjoyed DrupalCon. Do review this session online. You can follow Palantir at Palantir. Follow me at crow. Hagen doesn't really use Twitter, so don't bother following him. Yeah, let's enjoy the rest of your time in Amsterdam.


Welcome traders to another Tick-Mill Charter with me, Patrick Manley. Want to draw your attention to the sterling yen. We've been running up against this trend line resistance, excuse me, at the 152.30 area. Yesterday outside reversal candle taking out the weekly and now sitting right at the monthly pivot here at 151.20. I'm looking for any break of 151 as an opportunity on the short side, looking for a test of ascending trend line resistance down to 149. We'll also have monthly range resistance coming in 148.80. That's going to be a key test for the market rather than bullish reversal patterns there and extend higher again. All you can see is a significant breakdown. So watching for that break of 151 on the downside to set short positions, looking for a test of 149. That's always traders plan the trade, trade the plan, most importantly manage your risk. Until next time, thanks very much.


I'm Timothée Arvier and I work at Red Hat in the chorus team, mostly on federal chorus, but I do a lot of work also on other RPM voice tree-based system such as Federa 7 Blue, and today we're here to talk about, of course, Federa Kinoite, which is another RPM tree-based system based on Federa as well. Let's get started. This works. We have a fairly packed agenda and we'll see all that very shortly. First, quick basic things. What's Federa Kinoite? Well, Federa Kinoite is Federa, so that's the main important thing to remember. We're attached to the key DEC. We built from 100% Federa RPM packages, so there's basically no changes. It's like, you get Federa packages like they are in the regular distribution. What's the difference? We're calling Federa Kinoite an immutable desktop operating system, so it's not immutable in the sense that you cannot change it, but in that you can control how it gets changed and how things change in it. It's featuring the key DEC Blastmap desktop, and the first release will be, of course, for Federa 35, which is coming up in the couple of months. Well, I pre-wonder that you mentioned why we chose Federa for that because we're here for Federa, so it's not really news here, but there's a lot of great stuff around. We have to date, we're upstream first, we follow all the data to release, and we do also set for key D packaging in Federa, so this makes things great to use Federa as a base for this project. All right, so what makes Federa Kinoite actually different? Well, we based on RBM or Street to manage the system. That's like the basics. You get your system working as a consistent mage and not a collection of packages that may or may not work together. The second one, the good point is that we're using flatbacks to manage applications, so you get most of all of your applications using flatbacks. You can bait them on their own and uninstall them as you go. And finally, we just put them to manage containers of different tools and things you might need on top of the system. So, you might have noticed that we're a really close sibling of Federa Silver Blue, which is also an RBM Street-based desktop with the GNOME desktop environment. All right, so what does it look like? Well, essentially, it just looks like Federa because Federa Kinoite is just Federa. It's just not a brand with a little twist on it, but essentially the look is basically the same instead. We just have Kinoite instead of of being named Federa Linux. So, there's not much difference. We'll see that use discovery to install applications using flatback and you get the classic KD desktop. All right, so the status of this project right now is we're in full development program in progress. We plan for, we've been accepted as change for Federa 35, so we'll be there when Federa 35 gets released. And right now, it's in progress. You can get official development boots that are now available in Rohide. We can fill the links over there to install them. Or if you don't feel like using development boots for Rohide for your day-to-day use cases, you can use an official boot that I'm currently maintaining for Federa. They are based on Federa 34 packages. So, it's an official, but it's still 100% Federa RPMs. Essentially, the main difference is that they're not signed by Federa Infra. It's just signed by my personal GPG key, but that's mostly the only difference. Well, you still have to trust me and put them only from Federa RPMs, but essentially it's what I do. All right, so what's the goal of this? Like, why do we even bother creating a new edition, new variant Federa? We are just not doing that for the sake of doing it. Why are we doing that? So, the main idea here is we want to make sure that the desktop experience is great for users. So, we get this by using RPMs. So, RPMs gives us a really nice update experience for the base system because the whole system is shipped as a single consistent image. So, when you do an update, you either update the whole system or not at all, and you do that offline in a sense. So, you get the new version and you reboot to a new version. The update is atomic. You get rollbacks. You can go back. Finishing happens. And so, this makes like the day-to-day experience of updating your system really great. The second thing that is great with that is using flatbacks makes installing and updating application really easy. So, this decouples the updates of the applications from the system. So, when you get the date for file files or something else, you can get that directly. You don't have to wait for the system date or anything. You can update each application independently. You can even rollback them if you want. And you even have to update them if you want to keep a specific version. And it doesn't matter which version of the system you're running. Each application is independent. And it also opens up a really great opportunity to access a lot of applications. So, you'll see Flathead has a lot of applications available as flatbacks for Linux. And, of course, you'll still have the flatbacks which are provided by Fedora, which is full open source applications. Finally, the latest piece for the greatest of experience for users is the Toolbox, which is a common line tool which basically brings all classic Fedora packages directly to your desktop, just like you would have. So, even if an app is not available as a flatback, you can just like install it in the Toolbox and you get it. And it's just working almost as if it was already installed on the system. All right. So, that's for user. And then you get, all right, we'll use this first. And then you try and do things more. So, usually, you find a better thing or you want to test a new version of the applications or you want to try a new version of GD Plasma, because you think, hey, this feature is kind of cool. So, maybe I want to try it. So, the idea is, yeah, we want to be there for testers, early adopters of technologies, but I want to be like on the very front of the development thing. Before it's happening. And this has made it much easier, thanks to RPA industry, because you're not giving up stability for early version testing, because you can always go back to a non-void version if you know, for example, if you find a bug. So, yeah, it is that you try the version, try the new version. And once you're done, you just go back to the original version. And it's just easier. You don't have to reinstall everything. It's the same thing with flatbacks. Flatbacks, you can install several versions of flatback at once. So, you can have a testing version of the flatback installed as well, at the same time as the stable version, which makes it really easy to test a new version of an application without, like, certifying your stability, your day-to-day work on the stable flatback. Yeah, you can try nightly different flatbacks. You can try nightly versions of the system. And you will have the stable version as fullback. And all of that is installed at the same time of system. And the last piece, which I love is using PRs, building flatbacks from PRs. So, if you get like specific bugs or something, you can use, for example, this key of action which builds an application to the flatback for your application from PR and give that as an option to your user. So, they can go ahead and, like, try if this new version of the flatback, the application actually fixes the bug that they're having. And if it does, then you're confident that, like, merging the code will actually fix the bug. And this is, like, really nice user experience because for testers, they don't have to build the code. They don't have to understand everything. They just, like, install the new version of the app, test it directly. And when they report the feedback, you will have really good confidence that the bug is actually fixed. So, that's great. And finally, we want to make sure that it's also a great experience for developers. So, this is all enabled, again, by RBMistry in flatback because RBMistry enables you control change, well, code control change. So, that's, like, the thing about the immutable desktop. It's not immutable that you cannot change it. You control or you change it. So, essentially, the idea is that you can basically override any package on the system using RBMistry and it will tell you. So, it will tell you what has changed on the system. And you can go back to the previous version, to the stable version of what you had. So, that's, like, the nice thing about it. This is, like, by default, you will just go back to the stable version to the version that worked. But you can always try and change things to see if you want. Yeah, this version is just with, to the back is always to the reverse. So, you are installing some developing packages, things broke, okay, you just need to go back to the version to the stable. For flatbacks, the benefit comes with the way of flatbacks work in the sense that you've got everything included into flatbacks and build environments and frameworks, everything. The libraries, you can have anything you want and you don't really have any restriction or compatibility issues with what's already included on the system. So, essentially, you can use whatever you want, whatever you need for your applications without strong constraints. Oops. Yeah. So, all right, so that's what we want to do. But we're in there with Fedora, right? But for all that work, well, you also need applications to live on the system. And so, let's take a look at where are actually right now, where we can find key applications to install on Fedora. Well, the first thing is right now, the flatbacks in Fedora are still in progress. So, there's a good collection, good selection of key application already, but most of the key apps are available as RPMs that they are not yet available as flatbacks. We're working on that. If you want to get key applications, you can get them from FlatHub. We're a progressive packaging. A lot of them are in FlatHub. And hopefully, at some point before the Fedora certified release will have Fedora official Fedora flatback packages for key apps. They are also in good selection of neon key apps already available as flatbacks from Fedora. And of course, there are a lot of them on FlatHub too. So, the main difference between the apps that you would get from FlatHub and perhaps from Fedora is that maybe they're not necessarily the same version. But the main difference is that if you get them from Fedora, you will have the full guarantee that it's fully included on the open source software. Whereas from FlatHub, you won't really have that guarantee because you can have external proprietary software added in certain cases, or some restricted software, such as media codecs and things like that. So, yeah, if you look at the flatback from FlatHub, we've got a quickly expanding list of key applications. We're working closely with Substreaming every time to make sure that when we fix something, it gets Substreamed, so it gets there for everybody, and that key application works well by default in Flatback. So, it's a work in progress. We're doing that progressively, fixing applications as we go, as we try. There's a lot of easy and small tasks for beginners to get involved if you're interested. So, if you're interested in packaging or helping us with fixing some little small bugs in key applications, adding some features, it's a really great way to get started because packaging, once you've got the basics about FlatHub packaging, it's really easy to get started and re-package key applications, much easier than classic development outside of Flatback. Yeah, so, if you take a look at Hubs and FlatHub, you'll get a ton of them, or the classic ones, some proprietary ones, a lot of popular applications out there. And, if you want to focus on only the KDE application, which are there, you can search for all of the KDE, and you will get only the KDE apps that are already available on FlatHub. So, yeah, the next part of this presentation is about what we're going to do next. So, we're very welcome to ask what we want to go in the future, in the coming months. Well, once we've released it to our certified, what we will be doing on top of that. Well, the idea is that we want to be the best platform to try the next version. So, the next version of what? The next version of the Plasma desktop. So, of course, on Fedora, you will usually push the next, the current release of the Plasma desktop, but it's not always easy to try the next version. So, let's say we, the Plasma, KDE Plasma release the new beta version, unless you're running a low-high, it's not always easy. We can try and make available some other ways to try and run latest data beta Plasma releases on top of stable, the stable base of Fedora, to be able to try a new version of the desktop without compressing all the stability of the system. And at the same time, we also want to make sure that the nightly version of the KDE applications are easily available for you to consume at the same time as you can get the classic version of the flatback. So, yeah, being able to test both the desktop and the new applications at their release as beta before they get really fully released. So, the idea here is that we probably make some different data release streams. So, right now we'll have, when Fedora certified gets released, we'll have the stable version, of course, so the Fedora certified branch and the low-high stream, which is already there for Fedora Kinoite, and which we still a bit there always. But the idea is that maybe at some point we'll have this sort of stream, or another option where you get the Plasma, beta Plasma desktop and the stable Fedora content underneath. Yeah, and the last thing that we really want to try and focus on is trying to do the testing before we do releases. So, this, we're putting in Fedora right now is that usually when the images are composed, they're built and then the testing goes onto it and trying to make sure that the images work. And we might want to try and do that through the testing and then actually make sure before release that everything is fully tested. But this is like a much bigger piece of work and we'll see how this goes. All right, and finally, taking a look here is that we don't have any logos. So, we still don't have a logo for Kinoite. We're still looking for ideas. If you've got any ideas, if you've got any talent in this space, feel free to join us. There's an in progress issue about that and any help that here is appreciated. And yeah, you can join us for the ASU. So, we hang out in the, in the, in the kitty with the kitty SIG. So, everything is connected attached to the kitty SIG. And we have the issue you can report issues into the kitty SIGs. We meet fairly regularly on Mondays at 1800 UTC. And yeah, you can easily try. I have the official booths which are right now Rohaid until Fedora certified gets released or the official booths which are based on the stable version of Fedora right now. And all the links are there. Of course, I'll post the slides, link to the slides, just right for that. Okay, so let's have a little quick demo and some time for questions if you don't have any questions first. So right now I have one question. Thank you for that. Well, Kenoite had the same issues as Silverblue, like missing support for easy way of extending copper packages or missing ABRT reporting. Yes, probably. So we share a lot of the code between Silverblue and Kenoite. So a lot of common issues are, a lot of those issues are and I'm looking into that. The installing copper packages is, yeah, it's a little bit different, but you essentially have to pull out the repo file manually and add that to your system. But apart from that, you can directly install packages. So yeah, it's not great UX, but it's still fairly easy. For ABRT reporting, that's something I'm definitely looking into and try and figure out some points. But yeah, right now we're not great on this front. We have to work on that. All right, second question I have is, do daily drive Kenoite? And yes, I do. So I'll just show that to you. The idea is that right now I'm not daily driving the official release version because it's a little bit too soon. Still, we haven't even branched or satisfied. So it's like reasonably working, but it's not fully there yet. But I'm driving the official version. So I'll just show that to you right now. So let's just see if you can change up if you see my screen correctly, hopefully. I'll just have to switch. Yeah, seems like it's working. Yeah, great. Thank you. All right. So here's like the current version that I'm running, which is still an official boost, but it is still Kenoite in a sense, because it's still Fedora packages. It's Fedora 34 packages. It's an official boost from which are hosted by Fedora. Yeah. And you can see here, so Kenoite version is this boost I made this Monday, this week. And I have some packages of OLA here on top of it.  So yeah, you can see from this lid here, but I don't really have the KDE application over there on top, because essentially all the KDE applications I use, I use them on flatbacks.  So I've got Spotify, Steam and everything. And you also have like a lot of KDE applications there, which are the classic ones, which are all, unfortunately right now, so they all come from FlatHub, because that's where they are packaged, I'll say. Yeah. What should I show next? I'm on the other side. I'll just like give a quick try and see if there's no question. Yeah, there's another question. If someone wanted to create a federal Opium industry spin with a different desktop environment, like MixQ made something, would they build it on QR2? Well, neither. That's a good question. So essentially the idea is that you don't create, so there wasn't supposed to be another session just after this one about specific uses of to create desktop environments based on Opium Street, but we had to rescue them. So this would be for another time. But the base idea is that you don't, you base that on a common sort of packages and that's mostly done. Let's just like link to the config here, which shows you how it's done. So essentially the config is here. And yeah, and you essentially is a common base, which makes an Opium Street desktop based on Fedora. And then you add on top the desktop environment that you want. So essentially silver blue is adding on top of that. Key white is adding all the kitty plasma desktop of that. And if you want to make an XFC variant, which is something I've already done and officially is not an official variant, you can create, you can have this, you take the same base and you add the XFC packages on top of that and you get a full Opium Street based XFC desktop. And yeah, it looks very well. So you can, you used to have the official bits about XFC for that in my in the same links that I'm making here first. Yeah, so thanks Micah, but thanks for the knowledge in the sense, because I don't like to say that we remove gnome and had key, because since that's not really what we're doing, we're not removing gnome form to make it's all right. We're not removing them to make you know, because essentially just using the same base technologies and we share the same packages, it's just that instead of adding gnome to create silver bill, you had to create you know, so yeah, we're not taking away something from silver bill, we're just using the same common set of packages and creating something new. So yeah, it's not that it's it's kind of true, but like I don't like it feels bad. This is not like we're taking anything from from from them or silver bill here. It's just another option. Yeah, we need to base another yeah, I've created a base image for that. And but it's not there's not really an option. Another option is about like rebasing on top of federal chorus, which is all the bigger topic. But we're not there yet and we still have to to try and make that available. So yeah, there's no official XFC. So let's like go through the question that has it on which key comes from flatbacks and holy toe, is it in the history base? So essentially, what we have in the history base comments is like only is the core application that you cannot go and live with that. So we have had all the key deep blocks my desktop shell in a sense, and all widgets. And then you get Firefox, the terminal, Dolphin and discover and Arc, the Archive Manager, because that's like the basics. If you don't have that, it's painful in a sense on desktop. So that's like included in the image by default. Everything else you install their flatback. The main goal with the federal certified release is that I have just enough flatbacks built in federal so that we can pre install them, just like what it's done in Silverville where a lot of the GNOME application are pre installed in the image by default. And that's the goal here. So I mean, you just enough key application so that we can pre install them. So you have to get the calculator, the, I don't know, the music player thing like that available by default on on your desktop. So yeah, that's the sense in since the idea. No silver, blue, I have choices for more to stop them and key. So yeah, this, this is like the and will there be a nice observation silver. This is like where things gets tricky. So silver blue is GNOME. So essentially, the silver blue version of it's an up and sweet based desktop with GNOME. So it's only one thing. There's no XFC silver blue or anything like that, just like Kenoi. There's no GNOME version of Kenoi. It doesn't really make sense in a sense. It's a whole. So you cannot like speed it up. But you can create an XFC based off the industry desktop. It's still possible. It's just, it will just be something different. It's like, yeah, it's, it's really weird. So if you take a look at the config here, just to share the screen again. Yes, you can install classic RPMs on top of or should I, should I share my whole screen? So if you go, if you use RPMs 3, if you go there, so when you say RPMs 3 status, you've got a list of layer packages. What this means is that essentially what I've done is I've called RPMs 3 and I've said install and I've given hit some packages. So here I can, for example, set a stress. And if I go ahead and start a stress, we will just create a new deployment and install a stress on the system. So this won't be available right now, but you can reboot into the new version and you get package install. And there's even new option right now. If I just come to that, you can go ahead and say, okay, let's do that directly. You can go ahead and apply live and you will get a stress installed directly on the system. So it takes a while and but you know, it's going to maybe we can give it a try. If it's going to be the top of the 30 minutes, I'll have to start here. All right. Thank you everyone for coming to the session. Hopefully you get a bit of view of the work that we're doing right here is right now. And yeah, if you got any questions, please feel free to reach out to me either on metrics, the RC channels in the QDC, if you feel you want to take us and anything like that, if you want to contribute, you're welcome, of course. All right. Thanks, everyone.


Thank you for making it out today. We got about 800 kids. Let's take our time and shop for these kids the right way. We're doing Christmas crusade. Denver Police District 4. We are here to shop for the children who are not going to have a Christmas for themselves. I'm going to go see if I can find his jacket first. We get stacks of forms. Last year, the clothes went quick, so I thought I'd get those first. They have children's names, their age, what they're asking for. He's looking for a coat large in kids. This is a nice one. We come here and we shop for them. That looks like a nice coat. Fleece lined pockets. We do our best to do everything on their little Christmas list. A Bronco's shirt. No child should be without a Christmas present on Christmas Day. We have $35 for each child. That's a nice shirt. I'm going to try and do my best to get them as much as we can. 12-year-old girls are hard. You like what you like. I'm like, I was in seventh grade. Did she need some extensions? Oh, maybe. We are looking for an eight-year-old boy. An eight-year-old boy wants a lightsaber. So the mossy-o winter cat, he's got a good little theme there going. We as law enforcement officers love to give back to our community. I'm like, hey, Anthony. Bronco's jersey and football. We come in contact with underprivileged families. That one goes here. Children who are going through situations that most kids don't have to go through. Lots of girls clothing. When you see a child that wants a jacket or scarves or gloves. It breaks your heart. A lot of needy kids. It makes you feel very humble, grateful for the things that you do have. That's a nice hot rod. We hit the jackpot here. Now I got the $35 spent on everyone. They've got a whole section set up for us over in the garden center so that we could get our checkouts done and get them bagged up for the kids. That was fun. I would love to continue to be a part of this Christmas crusade. It's such an important part of what we do as law enforcement is what we do as a community. And we're good.


So just a couple of notes just on exactly what I'm doing here. So many of the people on this call will know many of the things that I'm about to talk about very well. For instance, Irina and Leslie and others are very much involved with those various organizations, Geoscience, Community Vocabularies, et cetera. What I hope is new or newly discussed, at least as some of you hear today, is just exactly where some of these communities are up to, what the latest steps are that they are doing, and how they are similar and how they are different, and then some thoughts around that. So I hope that even if you are familiar with most of this, the rest is something new and interesting there for you. Yeah. OK. And yes, in case it wasn't very obvious from the logos and the things, I'm changing where I work. And so it's sort of through that new work that I'm representing now, because some of the work that's going on here is through a project that I'm working on through Karawong AI. I'm still also working elsewhere, but this is sort of the work here. All right, so just a small agenda. And exactly as it said in the title, I'm going to talk about some steps with the GGIC and their vocab things, what they're currently doing, a proposal, and some future prospects. Then I'm going to talk about the ICSM and the same thing, what they previously did, currently, and future things, and then mention some other voc governance in this case, a vocab with governance implementers, so the CGI International Geoscience Organization and the actual ISO itself, International Standards Organization. OK, so Meakin has actually over viewed who GGIC is. There's a link there in case people are reading this after the fact. But yes, they're the part of the Australian Geological Survey Organisations Network, which has changed its name, but that's what it's called now. And it's the committee that governs information assets. Now, the current scope of GGIC, it's perhaps not super officially defined. Sorry, let me restate that. The current vocabulary scope of GGIC is probably not officially defined, but in practice, GGIC have a number of vocabularies about four that derive from or are related to standards. And those are the vocabularies that they, in some sense, govern, but certainly present. And now I'm going to see if I can click through to them from here. Now, can someone tell me if you can see a vocabulary screen or still my presentation? Meakin, what do you see on the screen? Oh, yeah, I can see the vocabularies. Oh, great. OK, so there's just this. Sorry, I'm sorry. Nick, I was just trying to work out where the chat was on Teams here. So I can't see a chat. But my apologies. So perhaps raise your hands once the presentation's over with questions. Sorry to interrupt. Thank you. Yeah, so this is just a technical thing we're looking at here. And it's got the old logos on. But these are the four vocabularies that GGIC knows about at the moment. And they've been around for a while. And they don't require very active governance. I mean, two of these were recent. Two of them are older. But they're there. They're used by the community to some extent. They're fairly static. But there's potential for them to be added to and handled. But what's happening now? And that's been the case for some time. But what's happening now is that there's a. So there's no specialized vocab governance, but there is information asset governance in general in the GGIC. So they have meetings and you can go along and talk about governance of assets and make proposals, et cetera. But nothing vocab specific. And that's been known for a while. Now, what's happening newly is that there's a proposal from some of the geological surveys with me assisting to consider the GGIC taking on another vocabulary and then testing out the acceptance of that, the management of that, the governance and delivery of that. And if that all works, what about other vocabularies? Now, the vocabulary in question, the sort of the prime mover here, it's a very standard minerals list vocabulary that's been developed by the Geological Survey of Queensland. And really all they've done is they've copied international standard mineral lists and put them in semantic web form and delivered them online. So that's all it is. Now, there are some people think that that's great and some people think it's not great and that you could improve that vocab list. And so one of the ways to resolve this is just to push that vocab up somewhere else and have it as a target for discussion. So the proposal for the July meeting of the GGIC is for the GGIC to accept that this vocab as a sort of a proposed vocab for the GGIC to deal with and then to appoint a kind of custodian and to make the vocab available for development. So for instance, one of the issues with the vocabulary is just a structural one about the way it's presented. There are groupings of minerals in there which are not themselves minerals. So a grouping of minerals is a grouping of minerals. It's not a mineral, but it's rendered as a concept in the vocabulary and a better construction of the vocabulary might see those groupings as collections rather than actually concepts themselves. So we know that there are these kinds of issues and we know that there are probably multiple different opinions on this. But the goal here is to move that vocabulary out of the Geological Survey of Queensland's kind of personal holdings into the GGIC where it might have a sort of a status that indicates that it's in development or who knows what, but to take on that vocabulary and then request contributions and additions because it's pretty obvious that for the Geological Survey agencies, minerals are a big deal and there are part vocabs and other minerals vocabs at the Geological Survey of South Australia, WA, within GA, et cetera, et cetera. So there's lots of minerals work going here, there and everywhere. Let's take one vocab of minerals and try and improve it and deal with it. So now there's nothing stopping you sending an email to GSQ and saying, hey, like your vocab, but can you do this? And they've always expressed that this was a possibility. So if you wanted to use that vocab and you want it to be fiddled with, you can ask them right now. But that's not very visible and it's not perhaps the easiest and best way to do it. A better way is to put it into a community governance kind of mode. So this is the proposal for July. We don't know all the details yet. Will GSQ remain as the kind of main custodian of that thing? Does GGIC have people that could act as GGIC custodians? We don't really know the answers to all of those questions yet, but we'll try with this vocab and see how we go. Okay, so that's the proposal. Let's see what's coming next. So future prospects. So it's likely that if we get success here, then more vocabaries could be delivered in this way. And when you've got more than one vocab kind of being dealt with actively rather than just sort of sitting there and especially when the vocabaries are at different stages of maturity and development, we're going to need to have a better handling of roles and responsibilities. And so for this, the proposal, which will only really be a formal proposal after this minerals vocabulary has been assessed, is just to reuse the roles and statuses and so on from ISO 19135. So that's the standard that talks about elements in registries, IT registries and the status and management of them. And you can see there on the right of the slide, there's a workflow. That workflow is a 19135 based workflow for the approval of persistent identifiers by the linked data working group. So what's it got to do with vocabaries? Well, items in vocabaries could follow a very similar kind of workflow. And I'm just going to zoom in on that for a second and show you what this looks like in a bit more detail. So replace the word PID or the acronym PID for persistent identifier with vocabulary concept and we're in the same sort of territory. So a submitter submits a concept to a vocabulary or submits a whole vocabulary to an organization. That would be equivalent and it's given the status of submitted. Now, there has to be an assessment done is the submitter actually eligible to submit things to this register? I mean, you might have automatically given them the status but you might discover that actually that they're not allowed for some reason. They're not a member of GGIC, for instance, in which case we would have to say, look, sorry that that request is invalid. We can't really handle those requests unless S can become eligible. So maybe it's, I don't know, the Geological Survey of Norfolk Island which is not currently a GGIC member. And they say, hang on, we can be and we should be and they become valid. Okay, fine. They say valid then we can accept the request. Now, there are certain groups that have certain roles and responsibilities according to this standard. And so there's a control body. That's one of the groups within these role sets. They're the ones who do reviews of things. So the register manager, the overall organization, GGIC in this case would assign somebody to review this vocabulary. Now, it's up to communities to implement what they mean by review. But where you've got a domain organization, you would imagine that they'd be doing two kinds of reviews. So sort of a technical review is this vocab structurally valid and then kind of a domain review is that scientifically or in some domain sense worthy and valid. And then they can say no, and it's invalid or they can say yes and it's accepted. And if it's accepted, it might go through sort of mechanisms. Now, again, this is talking about pistice nut and fires but it would be the same for vocabaries. Once a vocabulary has been accepted, you might then press a button to publish it. And then once it's published, now it might become stable. Okay, so that's the kind of very general workflow and this is the proposal for at least GGIC and elsewhere which you'll see in a second. Okay, so I'll come back to those kind of things in a second. Next thing is that a likely follow on proposal for the GGIC is to implement that kind of workflow and roles and responsibilities and governance structures which by the way is what geological survey of Queensland users internally. They already use this kind of structure. And so we're talking about replicating it maybe a variant of it but replicating some form of it at that GGIC level. Now, geological survey of Queensland and elsewhere already have automated tooling to do some of this. And here I'm gonna click on a link for the GGIC vocabaries as they currently are. It's over at Geoscience Australia. So what you see here, it's a technical screen. It's a set of workflows built into a code repository. And when you submit a vocabulary to this code repository, so I'll just go to the top level and show you. You'll see there's a number of vocabaries in here. What the four that we've seen before. When you submit a vocabulary to this system, this technical system, it will trigger a couple of actions. And one of the actions that it's gonna trigger is this vocab structurally valid. And it will actually do that assessment at a technical level. And if it's not, it'll give you a red cross and you can't proceed. If it is, you'll get a green tick. And so you can see that a few months ago, some of these vocabs were fiddled with and that's what we have here. But essentially, a mechanism like this checks the technical validity of the vocab. Yes, it's able to be published in this technical form. Obviously it has nothing to say about the scientific or other content of it, but it certainly ticks off the technical. So some of these steps in this workflow then can be handled with the automated tooling. The publication step two. So if you think that the vocab is technically valid because the workflow tells you and your scientists or others have said, yep, the content is good. Then there's a kind of a green button. You just press that button and it publishes. So really we're getting the reviews to focus on the content, not so much the structures. A couple of other things to say about the future prospects for GGIC. One is that the intended or expected flow is from geological surveys or members of GGIC up to GGIC. So if you're the Geological Survey of South Australia and you've got your U-Bute vocabulary about rocks or whatever it's about, maybe you publish that thing, maybe you don't, maybe you have never published it, but you suggest that this would be useful for the community. And so you propose that to GGIC. We don't know how regular that would be, but there's already GGIC meetings. And so one could, and we are in July, proposing a vocabulary and maybe each meeting every few months, three months, however often it is, more vocabaries get proposed. Certainly my hope that, and it's the hope of the Geological Survey of Queensland and others that anything that's useful for the community should be pushed up to that community level. Why would it be held at a individual survey level if it's of generic utility? And now the next logical step from that is, where to from GGIC? Well, if there's a vocab in GGIC that's of potentially international utility and a minerals vocab could be it, why doesn't that get pushed up to CGI? Now CGI is a global commission for geoscience information. So it's really like GGIC. What GGIC is for Australia, CGI is for the world. And so within the domain of geoscience and so on, if the GGIC has a vocab that's generally useful, why not propose it up and I'll come to CGI in just a minute. I should note here that this minerals vocab is reasonably, I don't know about controversial, but lots of people want minerals vocabs and people on this call have been involved in minerals, minerals lists, minerals vocabs. There's no clear, fantastic international minerals vocabs in the technical form that these communities want them to be. And so these proposals could, it could be that a minerals vocab is proposed to GGIC and maybe proposed upwards to CGI and other international players come along and say, we've got a better offer for you. Well, we welcome those offers. We would love to see them. But for the moment, this is the process of pushing the minerals vocab but then also other vocabulary sort of up the chain. Okay, ICSM. So ICSM has introduced, they are a committee on surveying and mapping and they have working groups that deal with issues like addressing cadastra, et cetera for Australia and sometimes New Zealand. And in the past, there have been several requests made to ICSM to manage certain vocabaries. For instance, within the geocoded national address file, which is an address database of all Australian addresses, there are a series of codelists that are, that could be and have been turned into vocabaries. And the requests have been made to ICSM over the years. Hey, could ICSM actively manage those rather than just sort of having them appear in this data set? And the answer from ICSM is it sounds like a good idea but it hasn't quite happened yet for a number of reasons. One of the reasons for this is a lack of technical capability to actually deliver these things. And then another one has been a lack of kind of governance and so on to, even if you had the technical capacity to actually do what GGIC is planning on doing, which is actually manage these things effectively. So that's the previous state is that the bin proposals but nothing's been delivered yet. The current state is that there is a demonstration organizational tooling, which is the vocab server there and I'll just click on the link. Many of you have seen this, I'm sure. It's a test server in no way is it official. And it's got a series of vocabaries here which are ICSM or ICSM relevant vocabaries.  So like the GGIC work and elsewhere, the technical capability to store and deliver vocabaries, that's kind of done. And then the question comes down to governance and appropriateness and so on. So there is certainly a renewed desire for vocabulary management, coming from a couple of the committee. So for instance, this example of place name categories, people on this call and others are involved in updating that place name categories vocabulary and the desire is that of course, as soon as it's updated that it's actually pushed through and published and made available. So when new place name types are added, that they're available for use. So there's a renewed desire for vocabs like this address types vocabulary, not just from the people who manage the Geocoded National Address Fire, but in fact the work that's in some sense sponsoring what I'm doing right now is actually with the Queensland government working on an updated address model. And that address model is dependent on those address vocabaries. And so there's a renewed desire to see those published. Okay. So what does the July plus state look like for ICSM? And again, this is potential. I'm not claiming that this will definitely happen. This is proposed. So there are a series of vocabaries now that have been submitted to that test machine with a request to ICSM. Do ICSM, these are vocabs that we, some members of your community would love to see published. Would you consider publishing them and let us know how you go? And again, just like GGIC, if those procedures, if procedures are implemented to review, maybe reject, but hopefully approve and publish those vocabaries, then those should sort of pave a way for more vocabulary publication. So just looking at some of the vocabs that have been submitted here, again, it's very similar to what we saw before. We see the GNAF codelists as have been proposed previously. And I know this is very technical and so on, but essentially you've got 16 vocabulary files here. And they are very, very standard codelists that are extracted entirely from the GNAF database. So let's just look at one of these. What's this one? This is address alias type. So in what ways can an address be an alias of another one? Well, it's a flat number, no first suffix correlation. I'm not fully across these details, but this is one way in which an address might be an alias of another. Let's just choose one other one. Flat prefix suffix deduplication. So you could have two addresses which are only differentiated by the types of suffixes being used in flat types and one is actually an alias of another and this code tells you that. So nevertheless, these are fairly standard. They're very well known and the submission here has put these 18 vocab, or 16, 17 vocabaries to ICSM and said, please review these. They passed their technical steps. So they are technically valid. Another question is whether the ICSM wants to actually govern them or not. So these have been generated, these vocabaries, even though they have existed before in vocabulary form, the current form of them and another vocabulary, which is bureau statistics, mesh block categories, these have been generated through a current project with Geoscience Australia to build a semantic baseline set of data sets for the digital Atlas of Australia. So that's the technical work that's actually produced these and it's expected that that work will produce another five, 10 vocabaries still in the next few weeks and all of those vocabaries could be delivered online in a couple of different places, but clearly in our minds, the best place for that would be via ICSM or some body like that. And we hope that ICSM will publish them and therefore the technical platform can rely on that publication as the point of truth for those vocabs. A couple of other notes on the July plus states, the technical vocabulary support is growing in the sense that the test vocabulary publication service that you've seen, the foundational spatial data framework project at GA is building a new version of those systems and it's integrated with other infrastructure within GA systems with operational support. So the vocabulary service should be rock solid, it's not gonna go away anytime soon and therefore as to what it can publish is a governance question, not a technical question and it's got very large capacity. You could publish thousands of vocabaries there if you wanted, but we haven't got a review and approvals process yet. So that's what we need July plus and we hope that this current submission will sort of trigger that process. Okay, so just a couple of notes on other vocabulary governance implementations and implementors that are currently underway. So one is the uphill from the GGIC and I've already mentioned it, which is Commissioned for Geoscience Information, CGI internationally. In theory, GGIC could propose a vocab to them to manage, but there's no current proposal there, but there is a current proposal from the ICS, which is another acronym, but it's the International Commission on Stratigraphy. So they're the people who maintain the geologic timescale. Now we already have a geologic timescale, I'm gonna use sort of air quotes, vocabulary at CGI, but it's a complex vocabulary. It's a very powerful but complex technical artifact and there's a proposal coming along for a simplified form of that vocabulary. It's related, it uses much of the same content, but it is technically a different artifact and it would need additional governance. And so there's a proposal then to CGI to do with that what the GGIC would do with minerals. So here's a vocab, please can you publish it? You already have some vocabs, here's a new one, how will this be managed and governed, et cetera. And in the same way that the Geological Survey of Queensland proposing to GGIC as a member submission, the International Commission on Stratigraphy is a member and therefore it would be a member submission up to CGI. Same procedures, same proposal for governance review process, the same technical tooling, et cetera. And so it's wheels within wheels. Now the last thing to mention here is the International Standards Organization. So yes, I chair a co-chair, a little working group within that that is aiming to push the technical committee 211. So these are the people who maintain essentially geospatial and geological standards to push them to using more semantic web-friendly vocabularies. There are a couple of vocabulary projects going on within that space. So there's a thing called Geolexica which is a standard vocabulary of terms and definitions. But the one we're concerned with is to publish standards codelists in the kind of SCOS vocabularies that many of you are familiar with. And essentially we have many of these codelists, there's sort of in principle approval to publish them in this way. And now what we're doing is we're working through some of the technical elements of publication. So how will they be updated? Where will they be? Who will manage them? What information is needed? And something that I'm just gonna show a couple of links in all of a minute here which will just highlight a property that's been added to these vocabularies that will be very useful for all of the situations that I've just talked through. So CGI, GGIC, et cetera. So let's just have a look at one of these terms. So this is just term valid in the register item status codes vocabulary. And it's got preferred labels and it's got a definition and all the kind of normal vocabulary things but it also has the status. And you can see that valid says it has the status original. And what that means in this context is that original as per the standard that delivered this codelist as opposed to being sort of an approved member submission. So this allows us to have a codelist of terms with stable or valid terms in them all original and then new terms to be added. And if approved by the community to also be valid but to be valid in a different way valid through member submission and approval. And in a very self-referential way the codelist that allows us to talk about this being original is in fact this codelist. And this codelist again in a referential way has two additional terms. It has the additional term of addition and the additional term of original. The original codelist here didn't have those terms and so things were either essentially submitted or valid and now we're nuancing valid and saying that there is valid additional and there's valid original. So if we look at original we'll see it's valid additional. If you see what I mean, it's very confusing there but the point is that we needed a codelist that gave us the descriptive power to talk about terms that were originally valid sort of grandfathered in and then new terms. And with this status marker here and governance procedures we can now have live codelist, codelist that retain all the information about what was originally delivered in the original standard form but that can be extended by member submissions. And that will be true of all of the codelists in GGIC and CGI, et cetera. Thank you. That's all specifically on this. I hope I'm to time.