So let's kick things off. Come on in, have a seat, welcome. Today, I'm talking about the evolution of deployments. We're going to be talking about how to choose the right workflow for your team. Before I jump into that, here's just a little bit about myself. I'm John Richards. I head up developer relations at a company called Paladin Cloud. We do cloud security, AWS, Azure, GCP. Before that, I worked at Pantheon and in the Drupal hosting and platform business. So I spent three years there. Before that, I had spent about a decade developing web applications. And then the last five years have really been about contributing, being a part of open source community, as I really got to grow and love the value that comes from that. And a little fact, a random fact about me. I was born in Pennsylvania, very the central south part of Pennsylvania. And somehow I never made it to Pittsburgh or Philadelphia. So here I am many years later, first time back in Pittsburgh. So really glad to be back here in my home state and actually visit one of the major cities here. It's been amazing. I love the city. So any of you, anybody here from Pittsburgh? Do we got any local folks in here? All right, we got a few. So thanks for having us here. You guys have a great city and we appreciate inviting all of us here. All right. So let's get to the topic at hand, the evolution of deployments. Picture here of one of my favorite video games called Ancestors. It's about the evolution and learning. So I put that here and maybe it'll tie in a little bit later. But we're going to talk about why, talk about the past. So like, why do we care about how this has evolved? Why does the history of this matter? Then we'll talk into what that actual history looks like. I'll talk about some of the complexities that have come about over time as we build out modern tooling and as we've adopted new workflows and techniques. Then we'll talk a little bit about what to consider as you're prioritizing building your workflow. And then we've got some time at the end. I've got some different scenarios and we can explore what it's like to build out workflows for maybe different environments, what you're prioritizing, what you're working on. And then ending with some general tips to keep in mind as you think about the workflow for where you're at. Alright, so why talk about the history of deployments? Well, if, you know, you're here in this talk, why does this history matter? Well, I saw this quote recently, if you want to predict the future, start by explaining the past, then work up to describing the present. And this came from a former coworker of mine, Steve Perch. He posted this on LinkedIn. I knew I was prepping for this talk and I was like, I'm still in this! That's so great! But I do believe this is very true as we think about what we want to do with our workflows, as we think about how we want to predict what we're going to do in the future, what we're planning for that. It is so important to start by understanding the past, understanding the reasonings and the choices we made so we can make informed decisions now and we're not just blindly copying. And part of why that matters is even more for deployments because deployment workflows are complex. As we go through this, we'll be understanding some of the complexities that come into building a workflow. They also take a lot of time to rebuild. I've been through a few of these at different organizations and it's always taken a huge amount of effort from the team to come up with to build out these processes. And once you've done all that effort up front, you really don't want to have to change this down the line. And so it's very rare to go back. You might only do this once every two, three, five years. It depends on where you're at. You may only change this occasionally. So it's important to understand why you're doing it or you can get locked into really bad workflows that don't mess things up. So it's worth taking the time to do it right. So with that, let's talk about the evolution of web deployments. Now, it may not come to any surprise to any of you, but early on people were shocked when they said, hey, people are really using this worldwide web. We're trying to put a website out there, and people are coming to this. We need to do something about the amount of traffic we're getting. And early on we started using specialized machines. We moved away from... I was just talking to somebody who was working at a university and they still had some old machines sitting under a professor's desk that they were using to run websites.  So there was people doing that, but very quickly we moved on to some specialized machines. These servers, no surprise. And hey, how do we change the stuff on there? We need a way to work with this. And that brought us to our good old friend, FTP, the File Transfer Protocol. And this was cool stuff back in the day. I remember in college the first time I played around this, and it was magical to me. It's when I decided to be a web developer. I was learning programming, and all of a sudden I was able to take my CSS, my HTML, and it's a PHP and send that up onto a server. And when I did change something and there was a visual output. I had been working with C++ before that and it was just everything was kind of in the terminal. And all of a sudden I was like, this is incredible. People can see it. I want to do web development. Now, there were a lot of problems inherent in this workflow. We really quickly ran into these problems. Like, oh no, I was working on a project and Frank came in and he overwrote my file. Or my co-worker Sarah, she just pushed up a change. Oh, I put the wrong break. So ignore that. Putting the breaks on the project. Made a change and all of a sudden now the site doesn't work and was it Sarah's problem? Maybe it was mine. We're not sure. And maybe I need to work on a project that's Jose's project but now I'm suddenly been assigned it and I don't know what's there. So we suddenly were like, hey, we need to do something about all this code of writing. How do we track this? And along came version control. I've got this image here I found from 2015 because I love it. It's got a plethora of open source version control options on here. And back in the day, I started out on subversion. If you know WordPress community is still hanging on really hard to subversion. So I'll poll all of you. Is anybody here using version control that is not Git? All right. We have nobody in the room who was doing something and that seems to be the case. We've really as a community back in 2015, whatever that was, eight years ago, it was like, hey, maybe somebody will run it. Well, we'll pull away here. But really it's become Git as the standard for pretty much everyone. And it's amazing. So that's why we're using it. Now, the next thing was happening is all of a sudden, these sites, they were becoming really important parts of our workflow, important parts of our business. And we say, hey, I've got a release coming up and it's Friday. I don't want to have to deal with if something breaks. I'd push out something. It would break. I'd have to work on it. I hated Friday coming around because maybe if we deployed something, I would be stuck working over the weekend to solve that. Some of you may have dealt with that same thing as well. And so this idea of having multiple environments came along. Maybe we can have a development environment. Maybe we can test this before it goes live. And I remember talking to somebody who talked about this workflow and being like, that's a lot of extra work. I don't know if this is going to catch on. And then it would only be a year or so before I was like, how did I live without this? How did I think it was OK to cowboy code right on top of the server and break things? So we moved into this world of, hey, we'll have multiple environments as we work on this. We can be confident on what's going to go out. Then we started having challenges around the idea of, you know, I'm tired of waiting for my code to be deployed. I worked where I was working. There started to be concerns about, OK, who controls the server? What's on there? I think it may be a little bit about compliance. And the IT team was like, well, we're not sure we want developers to have direct access to our production environment. Now that we've got multiple environments, it's OK if you have access to these other ones, but not to production. Well, then I wrote code and I needed that code to go up. And it was starting to bottleneck there. Or we might work on a project. I remember I would get a new project and I would go in and I would pull up the FTP program and I would download all of the code to try and get what the latest version was. Because just because something was in the repo, I didn't know if that's what was matching up on top of the server. And I had to be sure about this. And at the one place I was working where they had this bottleneck, we found a workaround which was at the time we were using a WordPress.  We would take all of our Git code. We would compress it. I should stay on the mic. We would compress that down and we would upload it through this file uploader. Now, I don't even want to talk about security implications of running this and executing it on there. But there was the problem that if I missed the semicolon, if there was a problem in my code, it would break the site. And because the site was the only method for me to get code onto it, the moment it broke, it meant I now had to call up the system admin and say, hey, somebody with actual access has to get on there. So there was ways to try and work around this, but it was really bad. And so thankfully, part of why there's kind of this universal adoption of get in here is people started to get into the GitOps or Git-based deployments. Well, if we are tracking all of our code in one spot, and that's kind of the source of truth, this is the place where everything's at, then maybe it makes sense that that drives what's on our website as well. Why do we need to have these as separate things? And so this was a beautiful step for us. And we see this growing even now. So you've got some kind of deployment process that's going to happen here. You see like GitHub's adding in GitHub actions. The providers are becoming full on kind of DevOps platforms. We've got GitLab building out their own things. You can have in some automations. Maybe you're using Jenkins to take care of this, but you need some step here as you get to this Git-based deployment to be able to push your code up. And then you can deploy that across these different production environments. And we'll talk a little bit more about the hooks and actions on why they matter here. And our next steps, as we start to think about, hey, I've got all this code now. These sites, these applications are getting so big. And now I kind of, so if you're a developer, there's two core skills you need to have. One is being really good at Google, and the other is copying and pasting. And skill one's about to go away, and instead of being good at Googling, it's going to be how well you can write AI prompts, but we're still going to be copying and pasting what we got from the app. That skill is still fundamental to us. And as we think about this idea of copying and pasting, there's something about it. What if the code I copy and pasted has a vulnerability or bug in it? Well, maybe it would be great if I could then pull in a fix for that vulnerability or bug into my code, because my actual code is kind of dependent on that original code. And it would be great if I could do this at scale. So I'm going to copy and pasted code in that automatically a copy and pasted in code from all the places they wanted it. Okay, I'm getting a little abstract with copy and paste here, but what I'm talking about is we needed a way to manage all of these dependencies. We were starting to have this struggle of I have dependencies, and I want to be able to pull them in and manage them. And we also started to want to write code that would write itself, so we're getting into CSS preprocessors, and maybe you're using certain JavaScript frameworks, things like that. This brought us into the world of needing a build process. We started, you know, maybe you're using gulp and grunt on this side to handle your assas file, something like that. And Composer became such a key part of managing dependencies that it is now baked right into Drupal, because it's accepted that we're copying and pasting from everybody. These are our dependencies out there, and being able to manage them is an incredibly important part of this process to make sure our site doesn't break. And then we ran into this challenge around quality. You've got a developer out there, they're working on code, and we know now, hey, we've got multiple environments. We can catch if something's going to break in that pipeline whenever we're testing it. It doesn't have to make it to production, but what if something does break there? It starts to slow down our process, because we've got code that's sitting there, or maybe the code goes through, it doesn't technically break, but it's written terribly. It's not following our standards, our best practices. And so we started to say, well, what if we could fix this? I once had to work alongside a developer, he was really smart, but he hated testing his own code, and the moment he worked on the code, he'd just push it right up and say, I'm done, and the moment you looked at it, you're like, this doesn't even run. You never even tested this on your own. And so the manual way you can do this is you can say, hey, listen, we're going to pull your access to the main branch of the repo, and until it's tested, we're not going to let you have access to that. But there's a better way. We're starting to add in this build process, we're starting to add in some automation. What if we started to automate things inside of that pipeline? After we've got this GitHub repo now, and we've got some hooks, we've got some actions, or GitLab, Bitbucket, I don't want any of the Git repos we'll say you're using out there, and you're like, I want to add in something more. I want some testing to be happening in here. So there's different levels of tests that you can be running. Maybe you've got unit tests, you're checking to make sure things work, and this allows us to suddenly be more confident in developers pushing out to a main branch if we're starting to think about better workflows, because we know there's some minimum level of requirement for code that's going to make it out and make it into our kind of deployment cycle. And the big thing about moving to this step is it gives us, as we start testing here, finally kind of all the individual pieces we need to step into a real robust DevOps cycle. And there's going to be more evolution we'll talk about, but this is the spot where we can finally start to see this idea come together. We plan, we have our section here, then we have, our code gets written, now we have a build process at fires, we've got Composer coming in, maybe we're running some node packages and doing some pre-processing, now we've got tests that come into this step and say, hey, does this work? And then if it passes this test step, we can move into this release, we can have a release here that we can finally deploy out, run our ops to make sure that goes across our different environments, we can watch this, we can learn from it, and we can start this cycle over. And so, this kind of level is where you're wanting to get, if you're wanting to use this DevOps cycle, you want to use that to speed up what you're doing with development, but this didn't stop the kind of growing, there were still ways we could get better, and so we started to run in this challenge of, hey, our monolithic site is starting to get overloaded and different organizations hit these at different times, depending on what your path is, it's not like there's a one trajectory of maturity across your DevOps process or your workflow process, but it's almost like eventually as you go, you will start to hit these. So it's like, hey, our site's not able to perform at this level, we're getting too much traffic, it went down, I worked at a place, we had a whole bunch of traffic come into our website, it took everything down across campus, and we suddenly were like, hey, we got to do something different from this, and this is where this idea of containers or maybe some other scaling things start to show up, and containers are massive. While I was trying to find an image for this, I found an image that was the evolution of containers, so I don't have time to give this talk inside of my talk, but no, that has its own whole evolution cycle happening there, but using this is a huge, it's a massive market out there, and fundamentally underneath, this is letting us scale more than we ever have before. And then we're talking a lot about the DevOps, the back inside, but we're also seeing as we mature this process, we want to make sure that even if all of our code is working, sometimes the site looks terrible when a change is made. I've made changes and suddenly the navigation isn't where it's supposed to be, even though technically PHP is running, all of the things seem to be working well, and now we're running on this issue, plus there's tons of viewport widths, you know, a lot of this happens manually, but all of a sudden it's, hey, we need to have some kind of visual regression testing built into our process as well. How can we make sure we're not breaking things for the user, even if the code says things are okay? Developers have been using diffs for a while, that's why one of the great reasons with Git, we can see what the differences are between commits, but this is starting to let us say, hey, what differences are you seeing? As they kind of go through this. Now, you know, be careful, this can be fraught with false positives, you're going to have to make sure you can get this tweaked right for your use case, but when you can really land this, it's really beneficial to say, hey, I now know whenever I deploy something, I'm getting this response back that I didn't break anything across the site. And then this challenge here around speed where you may be at a place that needs to deploy things out where I was at the university, and we suddenly said, hey, we want to have every little department have its own sub-site, how are we going to do this?  And so we get into, well, we might need a workflow for running a Drupal multi-site, or maybe you're using custom upstreams to deploy this out, but we now might need a workflow that says, hey, how do we take one set of code and start to apply that across a whole bunch of things? And then another challenge starts to come up, well, how can we speed up our complex app? And this is, you know, we hit containers, but we need something faster, depends on what you're doing, not everybody is out this tier, but we are, people are pushing what they can do in the Drupal space, what they can do with their machines, and so this kind of most recent step, if you will of saying, hey, what's next for us as we look to scale up? And this is, in my mind, the rise of decoupling, there's a lot of discussions at the conference here about this, the rise of microservice architecture where you might have Drupal as your back end, and then you're serving up some static content, you're hitting other APIs out there, and this can get you really incredible gains on the speed side as you need performance as you're looking to really scale up. Now, this can go too far at a recent conference, I was at DevOps Days Baltimore, and we had some open table kind of buffed style discussions and somebody was talking about is the future that we just run everything on 100% lambda functions, and we don't have any actual code, like is that the true microservice? And of course the resounding answer was like, no, too much of that is terrible. There was a headline from Amazon Prime's video where recently they said, hey, we're leaving microservices and going back to a monolithic architecture. Now, at the end of the day, they were really just scaled down microservice, they were still part of a much larger ecosystem, but there is a case of adopting too much, and so within this recent development, we're still as a community trying to figure out when's the right time to make this call, sometimes it's obviously incredibly beneficial, but other times, maybe it's too much for what we need. And that leads me into kind of our next section. So we've walked through a little bit of the timeline that's happened here, but all of this stuff hasn't been free, if you will. There's been a cost to evolving our processes. We've gained a lot, I tried to tag at the top, a lot of these gains were really focused around speed, and speed here meaning the speed of development, and the speed of a team releasing a product, less so than the speed of the actual application, that there was a lot of improvements around the quality, like how do we make sure things are working, how do we get reliability, those were really important, and scalability is a little bit more on the speed side here, but how do we make sure our things do well. But all of those have come at a cost of complexity, and that's not the only thing kind of pushing complexity on to developers. There's this idea of like shifting left, where responsibilities have started to move, hey, maybe we don't need a whole team to do this, maybe our developers, we could shift this on and handle this. The idea of just DevOps in general, or being a full stack developer, instead of just being a front-end developer or a back-end developer, there's often people who want a full stack developer, someone who can do everything, and that again raises the complexity that you're dealing with here. And on top of that, there's also other pieces coming in here, there's compliance needs of your specific organization that you might have to deal with, HIPAA, FERPA, FedRAMP, PCI, there's so many different compliance. Security side of things, the more you tackle on this, the more you need to be thinking about what security concerns you're going to have, how are you making sure that what you're doing is secure? I didn't cover any of that while I talked about that evolution, we're assuming you're keeping up with that along the way.  I pulled up this image here of the CNCF landscape of the open source ecosystem sitting out there inside of CNCF. There's 1184 different cards represented here.  And again, this is from the CNCF landscape talking a lot about Kubernetes, containers, and that side of things. There's a whole layer of other technology that isn't even inside of that that gets layered on top of this. And that's... So what I want to bring that to is this idea of DevOps literacy. There's a cost of learning all this information. It takes time. It takes knowledge and effort. How do you skill up a team to know what to do? And it's requiring now that you've come up into Drupal. Drupal 9 now, we've got people talking about, hey, we're using Drupal 7. Well, the reason some people haven't moved from 7 to 9 yet is there's a lot of changes. There's a lot of knowledge that needs to happen to skill up, but also you're restructuring that site. That's a lot of work to have to do. And so the knowledge entry point for Drupal has come up as we've talked, you need to be more DevOps literate if you're going to be using Composer as part of your build process again. And so we have to realize that we only have a certain limited amount of time and depending on what your role is, you will have more or less times to commit to learning this. So you need to be judicial as you're saying where do I want to put my time? Now if you're in a role, maybe you're a DevOps engineer, a platform engineer, site reliability engineer, something like that, it may be that your primary focus is nearly 80% on your DevOps pipeline and you can really dig in and build something amazing, but it may be that you've only got 10% of your time to do that. Or you've been given two weeks, solve this and get this out of the door. So you may say, hey, how do I start looking at reducing complexity here as I tackle this? So there's some different ways to do this. It's probably no surprise with all that complexity coming in that people are trying to say, hey, how in this ecosystem can I fit in here? And one level is to say, hey, this whole process is just too much for me. I don't want to have to deal with this. And people saying this may very well be clients of many of you in here. If you work at an agency, this is what you're looking for. They're saying, hey, I don't want to have to deal with this. Somebody else do it. So go find a agency or a consultant to help. You can have an agency do everything. You can bring in a consultant to say, help us figure out the best way to build our pipeline. If you don't want to hand that off. And this is my little pitch here.  Go check it out. I was going to put some logos, but there were so many wonderful ones. They've got spots on the floor show out there. So if that's your case, go check it out there. Now, the other step is saying, hey, well, we're handling this. What's another way to scale that down? Another way is to have somebody else run the stack. So I've got to call out for some of our sponsors here. And then there's the big cloud providers out there. You can look at saying, hey, somebody here is going to run either the infrastructure as a service to help take some of that off or the next layer up is a company like a Pantheon or an Aquia that's a platform as a service that says, hey, we're going to simplify more of this for you. And again, this is a tradeoff as you decide what level of workflow and complexity I want to use. And so as we think about these deciding that as you're trying to figure out, okay, what are the tradeoffs that are acceptable for my organization? You do get into, well, how do we start to prioritize our workflow decisions for our organization? And so I want to bring us back to this DevOps cycle. And really what we want to think about is what are the parts we need in this cycle for our organization? And what are the important parts on here? Now, that real kind of our first step, the minimum thing you need is some code and to be able to deploy that, but you're not, that's not a robust way to get about this. So we want to start with this idea of planning. This is the people side of things. Often it's much harder to wrangle than the tech side, but it is certainly equally as important. Many people are, you know, you have a bunch of people working on a project, you're going to need to keep track of that. Who are the stakeholders? Who all will be contributing to the process? Do you need to have design or content? Quality assurance that need to be a part of approving things? And maybe what compliance you need to be understanding is you're in this section. And then from the plan we move into the code side of things. When Git arrived on the scene, it was the most complex challenge for web developers. I remember me and my friends getting around and being like, this is really difficult. How do we set this up? And it's not gotten much easier, but we do have more challenging tech around nowadays too. And there's better resources for learning it, so that's definitely helped. But still, you know, if I get into things like subtrees, I really quickly get to my level of like, Git literacy. And suddenly it reminds me, so I took a couple semesters of Korean and now I'll hear a Korean word and I'm like, oh Sundubu Chigae. I've heard that word. I don't remember what it is. I'm going to go use my developer skill of googling, find out what it is and now I've got it. So that can be an acceptable level of Git literacy which is, I've heard of subtrees. I don't remember what they do, but if they come up, I can go look that up. So you try to figure out how much you want to go because there is a really deep level here. I got here like Git flow. There's different ways of deploying across your using Git.  And I just want to end here that you do have to learn this but there's some great resources. I love cats. So go check out this resource from 2017 which talks all about how to learn Git using cats. All right. And then the next up is the build process. And thankfully as I mentioned most of the folks doing Git out there have realized that DevOps is really important and have started to build in ways to do this. At Lassie and you've got pipelines, you've got GitHub actions. I know over at GitLab they've got a whole kind of platform for deploying this stuff out. But this is a really important decision as you think about where build is going to happen. You've got your CSS pre-processors. You've got the compiling. Maybe you're pulling in some of that testing phase in here. Where is that going to fire? And with D9's Embrace of Composer, this is now a mandatory step in any kind of D9 deployment process of where is Composer going to happen? So where do you want to build this? You can build it on your machine as the run Composer locally and push that up. And if you're doing that or you have a pre-hook, you're going to be saving that artifact in your Git repo. So you're making this decision of what's going to live inside my Git repository. Am I going to have a whole artifact in here? Or do I want to keep this bare minimum? But then I'm going to need a way to run this afterwards. I talked to a company and I was asking them, okay, well how do you handle Composer on your end? And they said they use Bob deployment. And I was like, I'm not aware of that. And they're like, oh, well what that means is when we're ready to actually deploy this to the server, Bob goes into the GitHub repo, runs a Composer compile and then he pushes that up to the server. So that's one way to do it. Not what I would recommend. Automate something there. But figuring out where you want this to live. I got on here on the provider.  You need to make sure if you're doing this, then your build process is happening over there. You're kind of handing that over. And you may not be able to then run your SAS pre... your pre-confiled SAS. You're going to have to figure out something because you've handed over this step and you're no longer controlling that. All right. Then we get into testing. And again with this what and where. So your build process, where it happens, is important because testing is a key piece of this. You're going to want to, after you kind of do that build, you're going to want to then run all this different tests that you're doing. And we can see here there's a lot of different ones. The unit tests at the very... then you got your functional ones. You might do benchmarking. You might check for code standards. There's visual regression testing. And I call out visual regression testing on here because if you're looking to add this into your pipeline, unlike a lot of the other pieces where you can run this as part of the underlying code, visual regression testing really needs that final artifact. And so if you're running visual regression testing, you're going to need to make sure that that build process is happening before you get to that because you can check and make sure developers are writing good code easily in the pre... I can't think of the word... in the files that you're writing for your Drupal site. But you don't need to compose it. You don't need the final format for those. But to be able to view that site, you do need to be able to do that. And so as you think about doing this visual regression testing, that kind of sometimes has to happen after release. So you've got this release piece that's going to happen, this artifact gets created, and now you can think about, well, maybe some of our testing will happen after this. And then we move on to deploy. So once that artifact is ready for release, we can look at moving them into production. All systems are green. This is ready to be developed. You can have however many different environments you want. All your checks have gone through. But you would need to be thinking, as you do this, if you've got multiple environments, how do you make sure they stay in sync? Because we're at a place, we had testing, we had production. But we didn't have a great syncing process for them. And we started rolling out updates to PHP, and they worked well in test. When we hit production, it broke some sites, so we rolled them back. And then for six months, we had a different version of PHP sitting on production than we did in test. And we would write code, and it would work in test, and then it would break in production. And so if you don't have a good way to keep those in sync, you lose the reliability of that. So this part of the workflow of deployment really gets into, okay, if you're going to have these different environments, how do you make sure they're all staying in sync? You might also be dealing with something like important data. Maybe you can't have that data sitting in dev or test because of HIPAA related. It's PII, and so you might need to have some Drush SQL Sanitize running to make sure as you pull that data down, people who have access to those other environments can't read the important production data. And then there's this operate phase. This is where you start using your code on the live site. This is really where you've deployed this out now across whichever environments that you have. And now you're moving on to the important spot of monitoring. Maybe you're throwing on Google Analytics. You're going to want some uptime checking as you check to say, hey, is my site staying up? You're going to want logs so you know what's happening. If something breaks or there's a problem, you can track that down. You can figure out what's going on. And then hopefully there's just a way for customer feedback or whoever your end user is that you're looking to work with. How do you get that information back in? Because the final step of this process is coming back to plan and saying, let's iterate. This thing is a cycle for a reason. How do we mature? Because you can think of DevOps maturity, like one measure for it, not the only one, but a key measure is how fast can you go through that helix, that cycle there. How fast can you make that loop? The faster you can do that, the more you're adopting and able to iterate and grow faster. All right, so we're coming to this idea. We want to customize our workflows. So for this last piece here, I thought we'd look at a few examples and kind of see how we could take some of this and look at that. So one option is maybe we've got this challenge. You've got your main job, whatever that is, but you've also got a friend. They've got a site on the side and they say, hey, can you help me start managing this? And so do we need a full on, a coupled website for this with a robust pipeline? I don't think so. And so for this as we think about what's on here, well maybe the planning, we don't need a tool for this. The occasional email comes in and I'm going to update it based on that. I'm the one who's coding any changes to this site. Maybe I do build this all locally and I take a peek at it in my local development environment. Maybe I'm using Lando or DDEV, something like that and I make sure it's working. I push this up to a Git repository and then I upload that to maybe a cheap server.  And if it's just a side thing, sometimes that's all you need.  But there's a lot of people who this isn't enough. If this site needs to stay up, if it needs to be robust we need to think about something else. So what if we're maybe a medium-sized agency that's building custom apps using Drupal? What's a workflow look like for that? So here for this on the planning side so for this organization in my mind I imagine they're using infrastructure as a service to give them a lot of maximum flexibility for their customer needs and this group is investing really heavily in the DevOps process maybe they've got two or three people who are doing that because they see application-specific workflows are a value add for customers. And so they might use GitHub for their coding but they would need something here where they're planning, how are they getting this information back from their clients they could layer in a build process on top of GitHub so maybe they're going to use GitHub actions or hooks and use something like a CircleCI to build out so whenever they want to test and make sure that, hey, you're following our linting standards, you're following our coding practices I want to do some compiling, we spend something over there using GitHub hooks they're maybe considering transitioning over to Jenkins if they need more automation as they go along. And then for visual regression they're going to use some backdrop to be able to make sure that they can check their websites to make sure that they aren't regressing visually. They've got most of their standard sites are sitting out their own Drupal 9 but they maybe have a few mobile apps and these are driven by a decoupled backend for content management and then they use a couple other APIs to be able to pull in data so with that they might also have some advanced tracking for customer advertisement campaigns as we kind of move into this monitoring section that they're using and then they implement whatever the customer's tool of choice is and because they have, they're looking to structure their agency to have regular ongoing contracts or at least having regular contact points so that they can have this loop happening over time so you know agencies out there you want that contract you want to be able to do this loop instead of, hey, we built it once and then we're done so that's one way to do this, this is an organization that has because they're building some custom apps, things like that they've really prioritized the value of a custom workflow for each one of these but that's caused the overhead and they're paying for folks to do that and then kind of a third option this was the one that I myself was in at one point and so this was, hey our planning phase here happens as JIRA tickets come in oh and I went past this but thinking of like a large university but they are building as much complex things they're standardizing a bunch of departments and kind of figure out how their websites might work so planning could happen in JIRA as tickets come in and so if you're using JIRA for that planning phase maybe it makes sense that on the code side instead of GitHub you're using Bitbucket you can tie those tickets together and hey, we moved away from our on-premise and that one system admin bottleneck and now in our case we ended up going with a platform as a service we used Pantheon and that lets us adopt custom upstreams for deploying this out to a bunch of different sites using D9, some integrated composer and then mentioned earlier with visual regression part of why Pantheon's like hey we've got this cool tool autopilot to do visual regression testing for you is because if you're relying on them for the build you're going to need that somewhere in there so that's a value, that's why they're doing that whereas if we were looking to say our tradeoff is we can't run you know a bunch of linting on our end, our build process because we've handed that off and so if that was a blocker we might not be able to use this route so that's where if you're getting value add from a provider understand what tradeoff you're getting so you know like is this acceptable for me in this case smaller sites, unit tests aren't as important, we're okay trading that off but you might need a more custom workflow if that's not the case and we use Google Tag Manager, we're going to throw it across all the sites as we move into monitoring and we'll use New Relic for our logging and uptime checking and so that's a whole different workflow than what the agency model was using or somebody at a small place so as we kind of come here to wrap up we come back to this idea of we talked about why this mattered we walked through the evolution we saw how complex things got and then we did discuss a little bit about prioritizing what are the important things to be keeping in mind and then we walked through three examples of custom workflows and I'm going to end here with some general tips for us as you're looking at working on a workflow for yourself so the first thing I want to remind you is it's not about it's not just about you I built out a really cool workflow for a place that I worked at that's why they kept coming up earlier and I thought it was magnificent it was a piece of art except I was the only person who knew how to use subtrees and when I left I had to write up this giant document and they were copying and pasting in commands and it just didn't last and so it's not just about you and building out what you want remember other people are going to need to know this they're going to have to handle this so make sure you're building for the team and not just yourself this is where we're at now but we're at a specific point in time and DevOps is not static we know it's going to keep evolving keep changing and so with that in mind tip three here is to build more for one year out so you don't want to build for where you are you'll grow but don't try to build for five years out it can make you think that you're trying to get the perfect solution but you don't know for sure where you'll be at five years what technologies are going to be around and so it's better to build for about a year out focus on where you think you're going to be so you have some expansion but without wasting time trying to hypothesize five years from now how much AI are you going to need to deploy your site you don't know don't try to set that up right now focus on the essentials what are the non-negotiables for your team so as I mentioned you know if certain parts of the build process are absolutely critical to you then you want to make sure that whatever workflow you do is built around that and if you've got certain compliance policies as I mentioned maybe like HIPAA that's going to require certain environments to only allow certain people to have access you're going to want to make sure that you start from that instead of building out a workflow finding out we can't actually use this and needing to shoehorn those essentials in later and I want to warn don't blindly copy that's why I wanted to give an example of three very different workflows that teams might use because it can be easy to say hey I went and I heard so-and-so was using this workflow I'm going to use it but they have a very different use case and you've wasted your time by doing that which leads me to my next point which is make sure you copy from other people because copying is great but don't do it blindly understand a little bit why so if you hear somebody has got a great workflow take them out grab some coffee hear about why they made those choices see what parts work for you what parts don't and then hey I told you I'd finally tie this image back so one of my favorite things about the ancestors video game is that you're trying to you're an ape out there in the wild serengeti and you're trying to survive and they have a key component on there of how do you learn and the way you learn is you have to take a small ape one of the babies you put it around your shoulders and you go out and it watches you as you gather food or climb a tree and it learns from that why does that matter well that's how Drupal grows is through mentorship through sharing what you know with others you can be brilliant and amazing and do all the cool stuff yourself but if that's not shared out it's going to end with you so make sure you're sharing with others check out the amazing mentorship booth at the sponsor area that's most of my talk here thank you I'm with paladin cloud we've got an open source project it's aws azure gcp but I don't know how many people here use that but if you don't just give us a star makes the investors happy it's open source and free that's it thank you all so much if you have other questions I'll be down by the pantheon booth where I used to work so follow up there also I think we got time for questions so I can take them here as well for the next couple minutes


That's the scariest thing to the ego and the most joyous thing is that in the end accepting that who you are Who your essence is is the gift? You don't have to look for gifts who you are is the gift, but it has no past reference point at all so What does that mean in everyday living? Well? It means that that You know like I I've seen Nikita in the communities when people come up They they may think they have a problem and she'll just kind of sit there and just kind of look into their eyes like What are you talking about?  They'll say you remember that or that and just know I don't what are you talking about? a friend of ours was kept hearing over and over In his mind for like three days in a row give Nikita to David give Nikita to David He finally came up and he said what does that even mean give Nikita to David? What does that even mean and I said all it means is let go of every thought that you ever had of Nikita Just come completely blank and open without the sense that you know Who she is or who she ever was because it's just it's only perplexing When you try to relate using some kind of past reference point That's where you get into this idea of trying to resolve something, you know interpersonal relationships are really sticky Did you ever notice that? You know, I mean at some point it dawns on us like these interpersonal relationships are really Sticky like what's going on here, but it's I don't know but it's sticky It really feels sticky and the only way that we can get unstuck From the stickiness is to really honestly admit That we can't really judge anything from our past reference points You know because that was just a bunch of judgments and Unload behold the characters really seem to act it out. Well, you know, you hold on to a judgment And oh, yeah, I can act that out. I could give you the Academy Award You want some drama? Oh, it's very convincing. It's like you're you're at the theater or Shakespeare theater And you're in the front row and it's like oh bravo. That was vicious You know like I couldn't I lost myself in that but we don't want to lose ourselves in that We want to know thyselfs like the Greeks said know thyselfs real you So it's kind of fun because that's what our sessions are anymore You know, we don't really have themes or topics or whatever It's just a group of people coming together and then you know, there's all this joy and if somebody actually Thinks they have a problem It gets pretty funny pretty quick Because the problem is like trying to draw attention. The problem is like trying to say look at me Look at me. This is serious. This is really serious. And then you got we've got a whole group of people born And Don't look at me and this is serious I've got a real problem a real problem here and then there's this softness of Essence that's And you know how it is whenever you've even joined with someone when you seem to believe you have a problem And they see that you don't they just all come here They give you a hug. They don't join you And go oh, that's terrible Because that's that doesn't work, you know false empathy.  You know, you don't hear that anymore.  You know, so this is what we're coming to now. It's just this is joyful. It's light It was a couple days ago where we had something going on in our community and and one was Seeming the perceive a problem in somebody else and the key to just grab the one The problem was supposed to be just was touching his head going. No, you're just a boy. He just said boy Reading like he's innocent. He's just innocent. He's not that thing at all. Whatever you think you're seeing in him is not there It's and it's never been there That's what the the essence was and then the lightness came swirling in immediately because it diffused Seemed to diffuse something but there really wasn't anything to diffuse It was just again to stand in true empathy to stand in that innocence and that's it's a fun life That that really brings it into the joy and the fun when your whole purpose is to to see the truth See the love see the Christ. We did a church service recently. Wasn't it? Christ be before me Christ be behind me Christ be below me Christ be above me Christ be to my right side Christ in my left side, you know, it was love all around me It's really the essence of what is real and true. So I'm not going to fight against that I'm just going to Experience that and it all just came from someone telling a story of driving in a car and having fear and then that's that was the answer that came You see that everything's a reflection of your mind like when we were coming We were we were flying across on frontier and we were just flying into Cincinnati and and one of the people had had opened the newspaper on the back of the newspaper was was a weather map and It had all these like concentric circles of what they're calling like the polar Vortex, and I kept looking over it. It's like it's like it's right over Knoxville, and it's just radiating out like giant waves of color love purples and Blues and some greens and yellow was way out in California, but it was all these things. It was right over She's like, yeah, yeah, this is This is our reflection of our our gathering in Knoxville Isn't it cute that it's on a map, you know, it's like it was all And it's like Knoxville is like the boy It just radiated out and we could just see that it's like, oh, yeah, that's our first stop of our tour And it's it's moving south. Oh, so are we It's just all or flexion now That's just reflections of your mind when you start to interpret everything that you perceive even weather maps as Reflections of the love in your mind, then that's what this no problem thing is. It couldn't be anything else You know, and when you think about it really Cold and hot are not the problem You know, if it had been that simple, I think we would have tried some other kind of solution But it's been a perceptual problem Holding on to a false idea of identity and then this was a beautiful reflection of that very colorful, too


They care about the nuts and... They pull some out of here. Yeah, like pull on the rope. That's why there's a ton. Yeah, that's why there's a ton down there. As much as you need. Be careful, it's slippery. Stop! We have to pick up on the ground, okay? Yeah, that's it. Change! Change! We're just gonna come up here and go down. Go down. That's in the dirt. What are you doing? They put some...


stress again. I just went in here. I got to do this quick for the second time. I just was in here and got half of it and memory card ran out. So I gotta make this quick. We're on a state route. Just found this house by accident. So, some ceiling for you. And I came up here a little bit. But, there's the stairs. And she says nothing in there. There's an attic hole up in there. But, ceiling. We got a baby girl in the car. So, I really don't want to fall through here and scare her. But, if we didn't have her with us, I would go ahead and go up there for you all. But, I mean, you can see floors and ceilings. This place is bad. I'm sure the floor stairs here would have held me. But, I'm just not sure about them steps. Because, as you can see, up through there. So, we're going to go ahead and go. We're heading to another pointed attraction thing. Not far from where we're at. So, but there we go. Already got the intro shot. Alright, there's the cobalt. And Pooh Bear. Making her, her appearance. Hi Pooh Bear. What you doing? Alright. I thought you already got stills. I got a couple. Oh, you ain't going in there are you? No. It's pretty bad in there. Okay. Okay. I mean, I can take some stills for you if you want. But, no, it's okay. Alright. Alright. Alright. Say bye. Pooh Bear out. Pooh Bear out.


In the offense so from so he had he knows what he's supposed to do from an assignment standpoint and now again it's continuing to fine-tune the fundamentals and techniques and then whatever you know whatever he can see maybe catch a tip or maybe understand what the defense doing is trying to do a little bit better those things will help him anticipate things a little bit better as he continues to grow within the offense. As we know repetition is the mother-of-all learning right so he's benefiting greatly from getting a lot of these reps and getting to play with that that first group of offense alignment and kind of starting to develop some of that continuity with them and understand how they see things and how they block things and being able to communicate with the centers and the protection part of it so it's been it's been really really good and then going back to the way the offseason went last year with all the COVID stuff where he missed all those reps as a rookie so obviously it's been really good to get him out there and hopefully we can continue to make up ground from that standpoint. Obviously he's a competitor so he wants to do more but you know we have a plan and we're executing the plan with him it's I think it's been a part of him having some success the past couple years through the season and toward the end of the season so he'll continue to progress that way but we've kind of started calling coach him because he's back there with a script now and he's calling out he's calling out all the plays and he's helping me substitute the guys in and so that keeps him involved in practice and then it's my little sneaky way to help him continue to grow in the offense because now he's calling out full formations and you know so maybe he's my thought is maybe he's continuing to gain knowledge and understanding by hearing how the quarterback calls things and hearing how coach Downing's calling things so it keeps him involved it keeps him watching and seeing what's happening and he can coach those guys when he comes off the field as well or when they come off the field I should say. Sure you know it's about the team first and Derrick is absolutely a team guy and he puts the team before he put before himself and but obviously there's a little bit of individual investment with everybody out here you know so we have not talked about goals other than team goals and and then being the best teammate and the best player that you possibly could be at all times and that's what he's focused on I'll be curious to see you know all of us like yourself you study the game and you see how people play football and obviously with two guys like that along with the other receivers that we have you know how are they gonna play him I don't know we've seen the damage that AJ's done when he gets single coverage and how it's helped the team and obviously from afar we've seen the career that Julio's had so it'll be interesting to see how they they help people line up and play and Todd and the offensive staff will adjust accordingly


And I think it's about time. Welcome everyone. I'm Cliff Lynch, the director of CNI, and you've joined us for one of the project briefings for our virtual Spring 2020 membership meeting. We will take questions at the end. There is a question asking tool labeled Q and A that's down at the bottom of your screen. Please feel free to put questions in at any time during the presentation as they occur to you and we'll just let them collect up and answer them at the end. Diane Goldenberg Hart from CNI will come on and moderate the Q and A at the end of the presentation. Our talk today is about the IIIF at the Getty. This is an architecture for managing and using image information and we will hear from Stefano Cossu, who is a software architect at the Getty. So thank you again for doing this Stefano, welcome and over to you. Hello, good morning, good afternoon, good evening, wherever you are. I'm Stefano, I'm a software architect at the Getty. I'd like to talk about adopting the IIIF at the Getty from a strategic and tactical standpoint. So IIIF is, well thanks to my colleague Rob Sanderson's keynote address, I hope I don't have to delve into what IIIF is too much too long, but I would like to point to a few things that we at the Getty really care for about what IIIF is and is not. So IIIF is a set of standards that are meant to deliver and share images and other audio-visual media over the web and there is a community of IT professionals in the mostly in the cultural heritage and scientific community who work to develop these standards as well as related software tools and also keep a discourse and communication alive to develop new ideas and collaborations between institutions. What IIIF is not specifically in spite of the most visual part of it, which is the viewers, IIIF is not a simple software product. So meaning that when I say that we adopt IIIF, it doesn't mean that we just bought or downloaded the software from from some open source repository and the property on our architecture is means we adopted a standard and we are building tools around the standard. Also as a note, you will notice that all the sidebar images in this presentation are taken from our amazing museum collections and they are all available as IIIF URLs that you can visit. So beyond the visual aspect of the viewer, IIIF offered several major advantages for us that were very compelling for to drive its widespread adoption in the institution. IIIF offers a logistical advantage because it gives us a methodology almost disciplined to organize our media archives and our media collections. We don't have to figure out every time a project comes up, you know, where which kind of derivatives we have to create, how we have to create and where did it go. It's all figured out from the outset and the architecture takes care of the rest. Also IIIF is designed for sharing which is of course part of the core mission of the Getty. We share a lot of information and knowledge and IIIF is an indispensable tool for sharing our visual materials. IIIF can be adopted by institutions of any size. You don't need to be the Getty to implement IIIF if you have a limited scope. However, when you adopt IIIF, its advantages become more and more apparent and grow with your collections growth in volume and complexity. There is definitely a learning curve and an upfront cost, but this upfront cost pays its dividends as your collections grow, which is the case for us. IIIF is not technically linked open data, but we have been, we have, you know, the Getty has been investing in the linked open data for a long time. We are, we publish the Getty vocabularies. We are behind one of the driving forces behind linked art and we have a very strong presence at Loudlam. So IIIF has been very compelling for us because it's very close conceptually to linked open data. I couldn't say technically is 100% linked open data because it doesn't share the semantic language IIIF manifests are meant to be read by humans mostly. They don't contain irreferensible permanent URLs for most of them, but IIIF can be a visual support for linked open data. In this way, you have a very clear separation of concerns. You can use IIIF which is optimized for presentation and the LOD for semantic discovery work alongside with each other. Short word about my department before we go on, Getty Digital is, you could see it as a traditional central IT department within an academic institution. We consolidate IT knowledge and centralized services that only a few years ago were maintained by individual Getty programs. And also we support the institutional role as a scholarly knowledge hub and a scholarly authority. And the way IIIF helps in fulfilling these goals is by providing, some of them are already provided by what's in place and some of them are in our goals. It provides an access point, a standard access point for searching, viewing, and comparing Getty images and other audiovisual media collections. It offers a single destination for the media production pipelines, as I mentioned, for actually producing the images. And also it is made up of modules. So it's a module of framework that allows us to build tools very efficiently and reuse them for many purposes, be it for new scholar use, for learning, for fun. We can build applications on top of the architecture without having to change the underlying architecture. So we invested heavily in IIIF. Actually, I was hired about a year and a half ago for the specific purpose of, with a specific goal of implementing IIIF to Getty, which already has been used before I came on board. It was initially, IIIF was initially promoted by Rob Sanderson when he joined the Getty several years ago. And it was implemented for specific purposes. When Rich Fagan, our current VP of Getty Digital joined the Getty, IIIF garnered an institution-wide importance. So it became part of the core mission of the Getty Digital. This is very important because we don't have to answer questions about what IIIF is and what it does and why we are adopting everybody knows about it. It's a very good thing. And as a sign of this commitment, we joined the IIIF consortium as a full members in 2016. The IIIF consortium was founded in 2015. And it's a group of institutions who are particularly committed to sustaining the IIIF community and therefore financially and logistically. Of course, there were challenges in adopting IIIF. We didn't really meet any resistance, any significant resistance. On the contrary, we had too many expectations, too much excitement about it. And we actually had to go through some information and education campaign about what IIIF does and what it doesn't. Also, we had to kind of, you know, one of the still present challenges is to manage project requests and prioritizing those requests because they are all very large projects and we have a limited capacity, human capacity, that the only bottleneck that we have. So managing what gets implemented first without overdoing it is been part of the challenges along with growing complexity and volume, which I will get to in a second. Also, there were some concerns about rights management. Some of our rights specialists were concerned about the openness in some ways of IIIF. Not about the openness of the images themselves, but the fact that IIIF allows you to implement any viewer on top of it. There were some concerns, for example, that the rights information would not be displayed where the rights managers would like to or there were concerns that it wouldn't be displayed at all. So we had to kind of come to terms with what I call bad old habits, such as embedding XIF metadata in the images themselves, which I find horrible and very unsustainable, but it was a way to kind of ensure and double and triple ensure that the rights information is always with the images. Also, we had to kind of tighten our quality control because by automating the production pipelines, we needed to rely on the authority for the image quality, which is our imaging department. We don't want to fix an image if it doesn't respond to the quality parameters. We need to just pass the images through because we can take the responsibility and the authority to change an image when we publish it. So we do very minimal changes. Basically just the format of images. We just format them to be served by IIIF, but we have to rely on imaging to give them give our blessing for the images. So we have this scaling pattern which goes along three main directive. One is volume of our archives and collections. One is traffic of our users and one is features, which means applications that can be built on top of our IIIF architecture. So when I started with my job, I took this challenge with a scientific approach. I thought I could not just choose whatever looked good at the first glance or even after some testing. I had to do a field survey that would be as comprehensive as possible and that would foresee this huge growth, which was not, it's unlimited basically. So the first thing was sorting out hack boxes and black boxes.  They just have to work. You don't have to worry about them. Hack boxes being systems that you on the country want full control over, such as ETL systems, which are very ingrained in the Getty data model and IT architecture. Then started reviewing everything that I could find that was a IIIF implementation, all the possible combinations of image quality, image compression algorithms, image formats and so on. So discarding what didn't seem, what didn't respond to certain initial parameters or criteria. And when I came up with a short list of these elements, I ran a benchmark of all these combinations, a few different image servers with a couple different image types and formats with different image compression algorithms and came up with a clear performance winner. When I had a clear winner performance-wise, there were some considerations that were not tied to performance, such as the sustainability of the software or the standard that I was testing and the fit within our department skillset and so on. Then when we came up with a clear winner, we started implementing it. The benchmarks that I use for testing all these systems are available publicly. You can ask me for the link afterwards. I'll be glad to share the benchmark, the results, and the tools, and as well as the test data set, which is a set of opening content images from the Getty. The implementation didn't take that huge effort. It was basically me and another health full-time employee for, I would say, less than a year, maybe 10 months for the first phase. The most challenging and time-consuming tests were building the ETL pipelines because, as I mentioned, they were completely customized and written from scratch. Mapping the metadata, which involved a lot of back and forth with stakeholders and the content managers from the different programs. And designing the autoscaling architecture that would sustain all the fluctuations in traffic. The challenge here was not as much the complexity of architecture but R being new to an autoscaling architecture. So there were some assumptions in the software architecture that were made initially that didn't fit with the autoscaling paradigm. So we had to do a lot of adjustments to it. But now it's basically it's live, it's out there. And we can start reaping the benefits of it. And we have some clear, you know, some advantages that are very apparent. This is an iterative application that will be launched soon. It's based on Edba Rouchet's The Streets of Los Angeles project. Edba Rouchet took hundreds of thousands of images of sunset pool of our building by building and we have acquired them and rearranged them digitally in this application.  So you can imagine that this will open a fire hose of images on the users, browsers, multiplied by the number of users that will peak when the application gets launched. So it's a big scale in IO performance challenge because we want this panning to be as smooth as possible. We ran some initial tests and actually the architecture responded very well. And we didn't have, so far we haven't had to make any adjustments to the architecture or do any tricks with caching or static things that are not triple AF. It's just been working. This is primary, of course, but so far it's been very encouraging. This is a load test that we did with some simulated traffic with Locustis, the traffic simulator program. And you can see that as the number of users grow at the bottom part of the screen, we get an equal growth in a proportional growth in requests per second and the response times are pretty consistent. This is what we wanted to see. We are testing here. This is a graph with 300 concurrent users downloading everything that's possible with large, small tiles and so on. So it's made us very comfortable about scaling up because when we reach a certain amount of traffic, a new server comes up and helps with sustaining the traffic. And when there's low traffic, the server goes down and we maintain a baseline of servers that are able to sustain the day-to-day, the regular load. I don't know almost anything about this animal crossing except that it's a video game and that there is a screenshot here with some blotchy images of our museum collection. The reason why I bring this up is that our interpretive team developed this tie, developed an integration with this video game using our museum images with AAAF in less than a week. So I was barely aware of this effort until it was launched. I didn't have to change anything the way that the API works. It just happened. And you can see from the bottom graph that we hit about six million requests in five days, in the first five days from the launch, we hit, I think, a peak of 380,000 images in an hour. The architecture didn't suffer from it, apparently. And going forward, we can make more. We can use more of AAAFs, of this architecture to do more things. There is nothing preventing us, for example, of course, it's not been decided yet, but nothing will prevent us from using AAAF manifest to build the collection web pages because AAAF manifest has all the metadata that's human readable and all the links to the order collections of images that can make up the web page. We can also build more ephemeral interactive apps. And when we don't need them anymore, we just turn down and retire the app only, not any data or any API. And we can also open up our API because it's a public endpoint and it's well documented because of AAAF. And anybody can use our API to build other presentation tools. So what are the next steps for us? There are implementing authorization and authentication, so we can offer richer access to rights protected images while giving a certain amount of access to the general public for some resources. We want to, of course, load more and more datasets. There is a continuing digitalization effort, especially on the GRI, the data research institute, which acquire a lot of archives. We want to start using manifests for more complex data structure, including user-generated data structures such as user collections. And we want to build more monitoring infrastructure so we can sleep better at night without having to get those nasty wake-up calls. Everything should be automated there. Also, we want to build an annotation server that allows scholars to annotate or whoever to annotate images and those annotations will be stored on our servers and that can be shared among other users. That's a very powerful scholarly tool, of course. So this is it, and I can't yet see. I'll stop sharing my screen so I can see the chat box if there are any questions, which I will be happy to take. Thank you. Thank you, Stefano. That was really interesting. What a great architecture and some of those load stats that you were quoting were really impressive. And I see that your talk has already inspired one question, so let's get started with that. Robin asks, first she comments, that was a great session, Stefano, and she asks, do you have persistent IDs for the IIIF images that are open for people to use? The reason I ask is if someone refers to the image instead of having it embedded, the ID might change otherwise. Yes, so the reason why most of our manifest images haven't been advertised is because we are making the final decision on some of the manifest IDs. So when those manifest and those images will be advertised as public, they will be permanent. And we also will build a redirect service in case we have to retire or change some identifiers. So we have a reliable path to get to IDs that have been saved by someone, might have been saved by someone in Bookmark, and they have to either get to the current resource or to 410 meaning the resource is done because we retired it or we removed it. But there has to be a very clear indication of what happened to that resource. So yes, the identifiers are permanent. Great. Thank you. Thanks for that response. And Robin, thanks for that question. I'd just like to remind you that the floor is open for questions. If you're interested in asking a question, just type it right in there in the Q&A box. And Stefano will be happy to answer that live. So we have another question now coming in from Charles Blair who asks, I'm curious why you would use, why you would not use EXIF metadata to embed metadata? I can say I might not, but I'm curious about your reasons. Thanks. The reason is it's hard to support change in metadata. So imagine you have everything that is about an artwork embedded in EXIF metadata in an image, in an image, which is our case. We actually have artwork metadata in the image. So if we change the title of the artwork or any other metadata that's already embedded in the image, we have to go back, fetch the image and change it. And that's not sustainable. One solution that I suggested in order to keep the title is to have the basic technical metadata embedded and possibly the rights metadata because you can't get away with that with our legal team. And a link, a permanent link to a web resource to a web page that has all the artwork information. So unless you retire that link, you don't have to touch the image if you change the title of the artwork. Thanks, Stefano. And Charles has a follow-up question. He's also curious to know more about the API you mentioned. The API, well, the AAAF API is basically defined by the AAAF specs. What we are doing is we put, we just put a gateway in front of it, sort of a shim software that takes care of other things that are not AAAF, such as caching the source images or implementing auth when it will be implemented. But the request, the API is, you can look up, you know, the AAAF.io website has all the specs. Basically, there are four different APIs. There is an image API that allows you to retrieve an image or the derivative of it based on your parameters. And the presentation API, which allows you to see the context of the image. So you can actually create a context around the multiple images, create a structure around the images, and embed the descriptive metadata. Okay, thank you. I'm not sure if that's in the kind of question, you know, if that answer, Charles? Charles says thanks. So Charles, let us know if there was something else you were asking about and we will field that as well. Moving on to Carmelita Pickett's question, she asks, how do you track images in copyright or images in the public domain? That comes from our system of records metadata. It's the, so since we gather images from different programs, we have different sources for the museum. It's TMS. So there are some rights metadata in TMS and some rights metadata in Rosetta, which is the GRIs, the Research Institute's archive, where I get the images and the metadata. And there are some specific fields. So what we are doing is determining the copyright status and the rights and the access criteria based on those metadata, as well as on the user's parameters, whether the user comes from an internal IP address within the Getty, for example, that's assumed to be, you know, someone who has full access. If it's not coming from the Getty, they should be logged in as someone who has particular rights. So based on that intersection of all these parameters, we make a decision of what kind of quality image we can provide. Thank you. And thanks for that question, Carmelita. Patrick Yacht has a question for you, Stefano. Wondering what implementation of triple IF server you chose? We used IIP image, which turned out to be the most resilient, the most efficient, and the most compact solution. That was the best solution for us. I'm not claiming it's going to be a general best solution for everybody, but that's what we were aiming at. I have to say we had some challenges in fitting that particular software we in the AWS architecture, because that software makes assumptions such as being able to access a file system, and that file system has to have a local path. We can access S3, for example, with triple IF. So that's why we had to create a shim, and we had to create a shim that actually downloads the images from S3 and puts them into a fast cache, which we have to do anyway, because we can't pull the images from S3 all the time. It's too slow. Also, we had to adjust our auto-scaling architecture, so we have to have a shared volume, because these IIP image instances can come up and down, and they all have to point to one shared file server. So there is something in the AWS ECS architecture that doesn't really play well with that, but we worked around it and worked out in the end. So if you need more details, you can contact me. I'll be happy to go over the details. That's great. Thanks. Thank you, Patrick, for the question. Thank you, Stefano, for the answer, and thank you all for these great questions. I see we have another one from Ray who asks, is the Getty thinking about future possibilities with the next version of the IIIF server and other imaging innovation possibilities here, IE video, etc.? Also, is the Getty thinking about integrating IIIF with any other software for further innovation for its audience? For future possibilities, I assume you intend the version 3. So we are actually implementing IIIF version 2. IIIF version 3 will allow us to deal with audio-visual, so moving image and audio. It's a hot topic at the Getty. I realistically don't see it coming very soon, because we have other priorities, i.e., getting stuff out there, and it will be a major effort. However, it's definitely a big advantage, and it's on our high priority list. We want to we want to implement it, you know, also being part of the construction. We want to, you know, be kind of on the forefront in adopting it and giving an example. And just to confirm there, Ray, it was version 3 that he was asking about. So okay, thank you for that. I see folks thanking you for this wonderful presentation, and I would also like to thank you on behalf of CNI for being here today, Stefano, and sharing with us Getty's implementation of IIIF. It's really fascinating and so amazing to see this standard applied to this incredible collection. So looking forward to exploring that further. I see other folks are thanking you, and hi from Chicago. So thanks so much. Thank you to our attendees, and I just want to say that we we do have the capacity to turn on your microphone. If I will turn off the recording and end the public version of this presentation. If you'd like to stay back, raise your hand and approach the microphone. I'm sorry, approach the podium as it were and have a chat with Stefano. If you're thinking of implementing IIIF for your collection or have some questions or want to make a comment, we'll hang around for a little while and be watching for those hands. So with that, I will close the public version of this and the recording, and thank you so much Stefano. Thank you Diane. Thank Cliff. Thanks so much, Stefano. That was a great talk. Very, very enlightening. Thank you.


Okay, right, I'm from Switzerland, by the way, in case you're wondering. Not from Germany, that would be an insult. Sorry, right, Exdebug, who here is using Exdebug, or Centibug, or PHP debug? Okay, awesome, some people, not a lot. Who is actually a PHP developer here? Okay, you should all be using Exdebug, but I'll get to that later. So it does a couple of different things. It provides new functions that you can use for debugging, etc. It changes the behavior of some core functions. It can do profiling, but most importantly, what I'm going to focus today, mainly, is that it implements a remote debugging protocol, which is super useful. First, though, do not run it on live servers.  So it's purely for development purposes. It's purely for finding bugs, etc. Do not run it on live. Also, if you get used to using Exdebug to solve issues, you might get really annoyed watching people who do not use Exdebug, trying to solve an issue by, you know, it works like wild amp, printf, throwexceptions, change files around, etc. That is the reason why I'm talking about this, because I want to see more people use it. I always get annoyed when people try to solve big problems in a very complicated stack using frameworks or stuff like Magento, and they're trying to understand what's going on over several layers of abstraction using printf debugging, or printr debugging in PHP. Right, so just quickly, all this stuff it does besides debugging the new function, new functions, it adds to PHP when you install it. You get co-coverage analysis, which is handy for people who like tests. It has a couple of functions that give you more information about whether you could output where you are right now, which file are you, what line, etc. Also one that's particularly useful I find is Exdebug get headers, if you're working a lot with sending HTTP headers back from all over the place in your code. This gives you a list of all the headers that have been set to be sent back with the HTTP response. This is now complete. I'm a bit minimalistic with this. I didn't really have a lot of time to prepare, so I didn't go to everything it does. It may do other things you might find useful. I do not use these things. It also changes how PHP behaves in some ways. One thing it does, it includes the stack trace in error messages whenever PHP shows an error. I think I have some screenshots of that later. WarDump can do colors on the console if you want it to. A limitation it introduces is it introduces a function nesting limit of 100 functions deep, so if you have really weird software, there's a chance it might not work anymore, it might crash. It will output that and tell you what's going on. Again, other things might change, this is not complete. So this is a normal error message, as you may know, if you have display errors on in PHP configuration. This is all the same error message it looks like when you have exabug installed, which gives you a bit more information, can be useful. So for profiling, it generates files that you can analyze with the usual tools that you can use for other cache grind files, cache grind, there are some good utilities. So basically you tell exabug to profile a particular session, you get a file, you analyze that. Yeah, you see where the performance issues are, but that's the topic for another talk. This is like performance optimization. I'm more focused on finding bugs. So if you want to do remote debugging, usually when I have like a new system running or something, if I just want to get it running really quickly, because that I think is the main reason why a lot of people do not do remote debugging because it seems a bit complicated to set up, but it's actually quite easy if you do it this way. You need to install the exabug extension to PHP. You can either install through your distribution usually in Ubuntu or DBM-based installations. It's a package. You can use a pickle if you want to do that. And for my way of using it, you basically need these two lines in configuration. You will get a new configuration file for exabug somewhere in slashetc, slashphp5, slashfpm, if you use php5fpm, slashconf.d, slashtwenty-extabug.com or something. You put in these two lines apart from the usual line about loading the PHP extension. And I'll show you later what that is, the connect back and the remote enable. And then you have to set up your IDE if you use one. Exabug also ships to the common line client, if that's your thing. I find it super useful. I use it together with PHPStorm. Who uses PHPStorm? Some Eclipse users, maybe? Okay, so Eclipse also supports the protocol that PHP-ex-debug uses for the debugging sessions. So when you set up your IDE, you need to first configure your IDE to accept incoming connections for the debugger. And you need to set up path mapping because the protocol uses, when it talks about files, it uses the absolute path on the server. So the absolute local file path on the server. So you have a local copy of the source code. And you need to tell it how to match an absolute path that it gets from the server to your local source code. I'll show that to you later. And then you can start remote debugging. You set breakpoints, or one breakpoint at least. You enable your listener in your IDE to wait for incoming sessions. And then you start your debugging session in the browser. Back to the configuration, I have ConnectBack mode enabled. So again, there's different ways to set up Xdebug. To me, the most convenient one is the ConnectBack, which basically means you tell the server, the web server basically, it's running your application. I want to start an Xdebug session. ConnectBack to me, and it connects back to your IDE. You can do it other ways. You can set it up in a way that your IDE kind of connects to the server, et cetera. This is the easiest one, very quick to set up. Again, potentially security, a security problem if you have shared servers or something, if the server is public. If you configure it like that, everybody can start the debug session. And if you start the debugging session, you can get a lot of information. You might not want people to have. Yeah. Live demo. Let's do this as soon as I get out of this. You're awesome. Right. So for the setup of the IDE, oh, that's a bit small. I'm using PHP storm here, as I mentioned. Just going to show you very quickly in the settings. You can't read it, but I'm using the filter here and put in Xdebug. I got some configuration here that says, can accept external connections that is set. So I already have it configured. Some other options that may or may not be useful to you. And then I have this little icon on the top left. This little thing here, this is telling me whether I'm listening or not currently, so you can enable and disable it. You don't have to do it in the configuration all the time. If I want to listen to sessions, to incoming debugging sessions, I enable it. So now my IDE is listening. And I set a breakpoint somewhere in my code. Let's say I have an issue with my code. I don't even have records. I have an issue with my code right here. I set the breakpoint. I go into my browser. And this setup, by the way, it's a vagrant setup. Who's using vagrant? Okay, some people. So I'm running, this server is running in a local virtual machine. So it's sort of remote, but it's still local on the computer. You start a debug session by attaching a URL parameter called, again, you can't read it, but it's called xdebug session start, and then you can give the session a name. I usually use one, that's it, but I can give it any name. When you do that, and you set the, as soon as the request goes out, and you hit the breakpoint, that should probably change the resolution. Does that work with your setup, Michael? It's not, it's probably, maybe? Scale. Oh, no, the scale's weird. Is that dirty? Yeah. Let's see if that works. Yeah, okay. So now I have a running session. The process running PHP on the server is suspended, let me sit down. And my debugger realizes that I hit the breakpoint, and the debug window here pops up automatically in PHP storm. I don't know how it works with xlips. Sure, it's similar if somebody's experience tells us. And I have the whole, not only do I have the local, contacts of local scope that I can explore with all the objects that I have that go like deep, several layers deep. This is a bit more information than you can get out by randomly wall dumping stuff into your browser, I think. There might be smarter ways to do this as well, but this works. So I have to complete local scope. I have the global variables from which I might be able to extract, say, stuff like database passwords, et cetera, depending on how your application is set up. This is a typo tree CMS application, so it's relatively complex. So I have the local function and its scope, and I'm at my breakpoint, I can step over functions. Yeah, for people who are not familiar, in a debugger you can step, like just align further in the current file you are in. If it calls a function, you can step into it. So let's see if I find one, where are we currently? Yeah, this isn't gonna work. Let's go step over each. So this, for example, now I'm at a point where I'm about to call a method on another class. So I can say, okay, I wanna see what's happening there, so I step into that, and here I can either also step into the next method or step over, so go to the next line in this slide. This way I can navigate and execute step by step, and yeah, if I need to go back, if I need to get the whole, say the scope of any earlier, basically I can move here to the whole stack, to the whole call stack, all the way to where the thing is actually starting. And again, explore local variables, all the objects that are in the current scope of that particular position in the stack. And then if I'm done with my problem, or I wanna let it run again, I'm gonna stop stepping, I can let it run again, and the browser session loads. If it hasn't timeouted until then. All right. You should also check out, there's other debuggers, there's sent debuggers, which is commercial, so I'm not sure if that's actually still a thing, people still use it. But there's also PHP debuggers, which is part of PHP 5.6 now. Who has used it? Okay, nobody. Kai has, but he's, okay, I see. So I'm not sure what's gonna happen in the future with PHP debuggers, at the moment it doesn't implement a remote debugging protocol, but nothing that I think can talk to IDEs that's like widely implemented. So Xdebug has the advantage that it works with a lot of different IDEs. Yeah, a lot of different implementations, client implementations of the protocol. Okay, one thing that happened to me last week when I was using Xdebug, I was trying to debug a problem on the other side of the planet on a server in Switzerland. I set up Xdebug because I didn't want to copy the whole environment to my local computer, I was just supposed to find one bug. I set up Xdebug, I asked it to connect back to my machine, it didn't work. After a couple of hours of searching, I found that it has a hard-coded connection timeout for the connect back and that timeout is 200 milliseconds. So if you run it over a slow connection and it takes more than 200 milliseconds to establish a TCP connection, Xdebug will just die before it actually starts the session. So I actually, I tweeted the developer of Xdebug. That's all I use Twitter for anyway, to render people. And yeah, he seems to be open to make it configurable. But until then, do not use it over slow connections because it might not work for your purposes. Also, it's one of the very few projects that have a flatter button if people use flatter. That's like the first time I actually could flatter something so they got my whole like $5 or something. If you use it and you should really use it, please use it, please go home and set it up if you don't already have a setup. If you use it to find bugs, if you find it useful, probably a good idea to flatter or donate to the project. All right, are there any questions? Yeah. Does the five-part mapping work with the five-part system? So if I'm packaging my modules into five-fives. I would assume it does. Yeah, as long as the files are the same on the server and... Okay. But will it still be able to pick it up from within my IDE and be able to track? I wonder, can the... Do the current IDs actually browse to the Power Files? The Power Files is basically a zip file. I'm not really sure how... The problem is, can your IDE go like, oh, we need this particular file in this Power File and we'll display it to you now? I don't know, I don't work with it. It's structured exactly as... Yeah. It's basically a zip file. It's a zip file of different files. So you'd have to have a client that can do that. As we said, I don't know. But the library gets to a web server and most of our code is packaged into five-fives. Yeah, I would not use... I mean, as I said, you use it on development servers or like local development instances and I would not use far on that, I'd say. I don't see the point in that, ideally. And maybe the path mapping because I didn't show it the server configuration or server-specific configuration. It's very simple in XDBack, in PHPStorm. You set up a server connection. Name doesn't really matter much because it's a connect back and then here you have the mapping. Here you have the local files and here's the absolute path on the server. And that's it. I would assume something with file files. You can probably debug it but I would think not all clients might support it. Does at least everybody get what this can be useful for? More or less. It's not like I was just talking to nobody. Okay, cool, awesome. Because it is really, really, really useful. And whenever I force someone to start using it, they keep using it. It's just the initial, you need to set it up. You need to know how to use it. And then after that, it saves you literally hours of time. Especially if you often have bugs that take five hours to find. If you work with code that you haven't written yourself, some complex frameworks. How does it work with a MVC framework? Have you tried using this for a MVC framework like KakePHP? Yeah, I mean I do some symphony stuff. Typotree is basically like part of it IMVC. And I think where it gets a bit tricky when there's like too much proper software engineering in place and you have dependency injection and things like that, things like that.  But I find it still quite useful. Yeah? I'm quite new at this, but recently, I was using a XD bug and you've seen to me and it was a database, it was some database connection and it keeps getting out of the tri-catch block. So after one day, I figured out one of the settings was that I didn't set it to not now. Is this like a common, is that? XD bug, you didn't set it to XD bug? No, the database. So XD bug didn't help me out, it just keeps going out of the catch block and it couldn't catch one of the arrows. And I don't know, maybe it was related, I mean, my question I guess would be in that sense that could XD bug do something bugging database. So I mean, my question might be quite simple. Well, you might want to use it. Say if you're using a framework, you have like several layers of abstraction until you hit the actual code that sends queries to the database, so you might want to use this about to hook in there and see what gets sent to the database. Maybe that helps. No, I'm fine, but... Yeah, we can take us offline. Yeah. What he says is true, you can borrow it at the point before it hits the database and inspect the variables. And inspect the variables before it actually gets thrown into the database connection. I think it sometimes turns a lot of... Oh yeah, they have it. It's just that one thing that got me... Did they click me, not now? That's enough. Okay. Right. All right, that's it. Thank you, Claudia. Awesome.