Hello, dear developers! Let's talk today about the background services available in the modern browsers for us to build webfront applications with improved user experience. In my session, we'll go through the list of available background services and I'll also explain how to use this superpower for organizing the best-in-class webfrontend applications. My name is Maxim Salnikov. I work as a developer engagement lead at Microsoft Norway. And I'm a big, big fan of the modern webfrontend development. To share my knowledge, to share my passion and expertise with this area, I organize multiple meetups and conferences and I'm often speaking and presenting at events like conferences, meetups, workshops, hackathons, all for the technical community. You can find me on Twitter, WebmaxRU. I tweet mainly about web platform, about progressive web apps, service workers and everything around. I'll be happy to stay connected with you. Back to our topic. What is the current state of web platform in terms of what can we do for our frontend part of web project? Well, we can provide some functionality which was never available on the web before. For example, we can keep our webfrontend application and data it consumes always fresh. Or we can organize notifications for some future point in time, regardless of the connection status. So we somehow can inform our dear users about some scheduled event, regardless of their connection status. Or we can automatically replay requests which were failed for whatever reason. For example, when our dear user suddenly went offline. Or we can provide long fetches which are connection resilient. I mean, of course, we cannot have any magic secret internet channel, but at least our fetch will not be canceled if something went wrong with the connection. It can be paused, it can be continued after, so very good user experience. You can tell me that some of these points we can implement in our JavaScript code in our frontend application without any need to learn background services, without any need to use service worker. Some of them with one limitation. This functionality cannot work after user closed the tab with our application. So the code we have in our main JavaScript bundle will disappear right after our application tab was closed. But not with the background services. We can have all these features available in the background. This is why it has a name like this, right? And I will explain how this works. But first, what is a background service? This is a part of the browser, of the browser's engine, which allows us to write and run a code which will work in the background. And of course this code is quite specific in multiple terms compared to our main JavaScript code. First, it runs in a parallel thread, parallel to our main thread. It's always good because this way we have less competition for processor resources, for memory, so it allows us to build smoother and faster UIs. It has completely different life cycle. As I already mentioned, life cycle of the main application thread is quite trivial. We start our JavaScript code right after the user opens the tab and detects our URL and some first bytes of our JavaScript bundle were fetched and we close everything, we unload from memory everything right after user closes the tab or maybe the browser itself. It's not applicable to the code we have as a part of background services. It started by a completely different occasion, which is event. So basically everything we have in our background services is a reaction to this or that event. So event came, our code ran and then this code disappears. So background services is not something that we want to run for too long. So it's a fast and short reaction to a particular event. There are multiple reasons for this and I believe the main one is we don't want to overuse the resources of the browser for running code, which is part of background services. Just imagine if every application will register something that will run for long in background. Well, nothing left from processor time and memory. Another very important part of this difference is the fact that this code is always ready to, let's call this wake up, regardless of the state of the main application itself. So the code can wake up and run even when the browser itself or to be more precise, its visible part is closed. And looking at these parameters, I have an analogy with Phantom or Ghost because our application is dead, it doesn't exist in memory and the tab application is closed. But pieces of this application, which we have as a part of background services is waking up here and there, they live their own very short life and then magically disappears, like Ghost, like Phantom. What kind of events can wake up our code, which is a part of ServiceWorker, of course? First, everything connected to the network state. For example, the situation when we go from offline state to online state. There is a corresponding event, which we can receive in our ServiceWorker and write corresponding code. Of course, in that case, we are mainly talking about the code which will try to replay failed request, which we tried to do while we're offline. Also, if we do a long fetch, if we download some bulky resource, it will also send us statuses to ServiceWorker. Is it succeeded or failed and what is their progress? Next is family of events based on timings. Specific timings or some intervals will have a closer look on this in the next chapter of this slide deck. And of course, we can wake up our ServiceWorker from the backend. Here we talk mainly about web push notifications. I already mentioned that all the code we have as a part of background services intended to be located at ServiceWorker, inside ServiceWorker. Basically, this is the only place of our application where we can get life cycle we need, which is completely separate from the main thread, and this is the only place where the code is always on duty, regardless of anything. And multiple sources can send events to our ServiceWorker. First, application itself. Then browser. Then operating system, proxy via browser. Then internet. Again, here we go back to the case with push notifications. But how can we react? What kind of payload, what kind of code can we write in the ServiceWorker to provide better user experience? First, we can organize bidirectional communication channel between ServiceWorker and application. For example, to update number of unread messages on our UI if we build something like mail client or news client. But what if the type of application is not open? What if it's only ServiceWorker who is always on duty? Well, we can start the browser and open this application from our ServiceWorker and maybe then to organize this bidirectional channel. Or maybe for some scenarios we don't need to do this, let's say, obtrusive action like starting the browser. Maybe we just want to show the notification. It's possible. Yeah, in that scenario we only need ServiceWorker. And maybe the most interesting scenario when we can run some custom code without any kind of user notification. I mean, we do not start the application itself. We do not show any notification. Sounds a bit scary, but I'll explain you why it's still cool. Back to background services. How to find the list of what's available in the particular browser? First, it's not a surprise that majority of these services are only implemented in Chromium-based browsers. Basically, if we look at this list, only push messaging is also implemented in Firefox browser. How to find this list? There are tools, application tab and somewhere on the left-hand side in the bottom you'll find this list. Of course, as any part of DevTools, it's very convenient and very important for us developers because this particular piece of DevTools helps us to debug these background services which might be non-trivial task at all without this special feature of DevTools. As I mentioned, it's all based on the events and sometimes it's very challenging to predict when this event will come and in which form and what kind of reaction will be provided from service worker side. DevTools gives us a chance to record all these events, which is super convenient for testing and debugging. Okay, let's briefly go through this list. I'll explain you what kind of services are those and we'll stay on couple of them to have more details. Let's start with API called PaymentHandler. It's quite niche one for the scenario when you wish to build web-based payment service even more precise, heavily web-fronten-based payment service. It's uniqueness in the fact that I believe it's the only API and only possibility to install service worker on the fly without even visiting the origin of this particular service worker with multiple limitations, but still it's quite interesting feature. And mainly we use this API for managing UI of the payment services. Background sync, I already explained the scenario when user in offline do some actions and for example they write blog post and then they hit button post or send and there is no internet connection at this particular moment. So instead of showing server not found or some other error what we can do, we can preserve the data and register a synchronization. And then at the first occasion when online is back when the connection is available the code we wrote within this particular event listener will happen and in this code in many cases we just repeat the fetch for example sending this data to our API. And as I mentioned before this will happen automatically in the background even if the user closed the tab even if they completely forgot that something went wrong. Of course I think it's a good idea to provide some UI notification about hey now we cannot send your data to the servers and it will be delivered later automatically. It's the main power of this particular API. Background fetch gives us possibility to detach data fetching process from the application lifespan from the main thread of the application lifespan. For example we want to download some large resource some mp4 video file and the download started and then user for example decided to close the application or something bad happened with the internet connection again then we have all the tools to for example pause this download and then to continue with this when it becomes possible. So we'll have the UI very similar to UI we have for let's say classical download using anchor tag. At the same time what differentiates this way to download resources from the one with anchor tag we have full control over the bytes downloaded. So it's not just the next file on our operating system which was downloaded. No, we have these bytes available in our JavaScript code and we decide what to do. In cases we just want to put them into cache storage for later usage from our front-end code. Push messaging. It's the most mature API from that list and here I just want to remind that we have possibility in addition to showing the notification itself to also run custom code. So we are closing to running custom code as a background task but in that case we must show the notification. So there is no chance for us to organize this in invisible for the user way. Okay, what's left? Notifications. On the one hand it's the best friend of push messaging API. As I mentioned there is no sense in sending push notification without showing the notification itself. But hereafter we'll talk about different part of this API called notification triggers and both notification triggers and periodic background sync are time-based activities we can run as a part of background services. I believe this is the only what makes them similar and they have very many differences. I gathered full table of this to better explain these APIs. First, what are they designed for? Periodic background sync. It's to give a chance to the developers to organize refresh of the application and data it consumes in the background. That means that it only makes sense to run this code and to send this event to the service worker while the user is online. Notification triggers is designed to give us developers a chance to schedule some notifications for the future time regardless of the connection. Okay, let's go through some of the parameters. Timing. As the name of this API says periodic background sync is something based on intervals. While notification triggers we schedule this trigger, this notification for one particular time stamp in the future. What can we do here and there? In PBS we write the custom code itself so there is no any magic way for having our application fresh and data it consumes fresh without the code we write ourselves. We just have this trigger when to run this code. Notification triggers is limited to only show the notification. Visibility for the user. PBS is completely transparent. I mean it happens in background. If we don't want to inform user about some code is running we do not inform user. In notification triggers obviously notification is intended to be visible it's native browser's notification. If we look at these three criteria these three descriptions of periodic background sync. Once scheduled it will run for more or less forever until user explicitly deregister this or maybe reinstall browser from scratch. We can have custom code there within this event handler. Hello Bitcoin mining. And this will happen all completely invisible for the user. Sounds scary, right? It's exactly the custom code I mentioned in my slide where I describe sources of the events. This is why there are multiple limitations and multiple considerations about this API about the case when this event can ever come to the service worker. First and foremost the final decision maker on send or not send this event to service worker is the browser is what we call user agent and it takes decision based on multiple parameters. First connection. I already mentioned that it all makes sense for KBS to run in online only. It was designed to run like this while maybe the main power of notification trigger is for offline mode when we don't have any chance to reach out to any particular user device for example to wake them up if we build some web front end based alarm clock. So notification triggers will work both online and offline. Pre-requisites needed to ever run this code as a part of these APIs. Notification triggers it only requires permission for showing notification like we have for web push while it's way more serious for PBS. First there is separate explicit permission required from user side to run this code in background and second maybe even more important is the fact that application has to be installed on user machine. Luckily these days it's quite simple to organize installation of your web front end application using web app manifest using the fact that browsers are well integrated with operating systems now but still it's quite a serious current. Also what I not listed here but also very important this event as a part of periodic background sync will come to the service worker only on the same network where it was registered let's say it's the same Wi-Fi network for example again it's designed like this to not break the user privacy. Let's have a look at some code. Periodic background sync it all starts from the registration of this particular sync or periodic sync to be more precise. In majority of the scenarios we do it in our main JavaScript bundle and I always recommend to do feature detection because this API might be available or might be not available under particular circumstances and we don't want to pollute our console with red lines and we definitely don't want to ruin core user experience and then we provide only one parameter called mean interval so we don't have full control over these intervals where we want to receive this event in our service worker we only can say something like hey please do not send this event more often than 24 hours for example and what's next? Next we go to the service worker and inside service worker we receive periodic sync event if we are lucky, if all these conditions are true and what's next? Next we write our own code there is no magic which will help us to update the application we have to write everything manually in majority of scenarios you fetch for example updated version of the application shell maybe you trigger some API endpoints to keep the data it consumes only also fresh and I think that if we receive this event we are already lucky but we can do step further towards best user experience and do extra checks before running our update code for example we can check network connection type and only run this update when we are on fast stable network maybe the user in data saver mode and then I suggest we can skip update at all and after all if we are talking about fetching some data maybe it's good idea to check if there is space available on the user's device if we talk about timing how often can we receive this event if we only provide a minimal interval in the best case scenario it's 12 hours and by the way it's something that's not part of the specification every browser decides themself what is this let's say timing schema what it depends on engagement score this is artificial index calculated by the browser and it's about how long and how often the user interacts with your application so the browser tries to do the best guess how popular your application is for a particular user to decide on how much resources it might dedicate to this particular application if we look at the notification triggers it starts also in the main thread and same I strongly recommend to start with feature detection and by the way here I illustrated how this notification might look like on Windows 10 operating system here we specify exact time point when we want to show this notification and then as a trigger we provide time stamp trigger and maybe in the future we'll have more triggers not only based on timing but maybe based on geolocation maybe based on some other parameters so I hope I inspired you to have a closer look on background services and maybe on these two specific APIs what's about availability and behavior of these APIs on particular browsers and on particular operating systems I have a long answer for you it really depends platform and operating system browser and its version browser settings and even flags and after all origin itself and the fact is application installed or not you can get all these details about every single browser and every single situation on the links I mentioned on this slide but I have a shorter answer for you let's just think about these features like part of progressive web application and P in this acronym means progressive in terms of progressive enhancement so let's just think about these features like about something that might improve user experience and it might happen or not and of course technically feature detection is the mechanism where we check all these availabilities let's sum up the goal of these APIs and many other APIs which are part of service worker family is to catch up with what native code can do and in some scenarios the web can do even better than native code these days let's really keep user experience in the focus and by using these APIs we give something in terms of better features more convenient usage of the application but we require something in return multiple things first all these annoying permissions second all security and privacy considerations third consumption of resources needed for running these background services so let's together with authors of these specifications find proper balance between what we give and what we require in return when it comes to developer experience I'm super happy with the current situation we have web standards we have great tooling and what I'm super happy about it's just a beginning of this great great journey I invite you to join Open Slack team I organized few years ago where we have more than 2,000 developers now discussing PWAs, service workers, background services etc and on that thank you very much


So as soon as he gives me that thumbs up, I'm going to announce and bounce. Next up, we have How to Move Mountains by Aaron Nattu. Please give him a warm Torcon welcome. Thank you. So this is going to be a little difficult because I like to move around a lot, and I'm going to fix my mic, so I'll try to just keep my upper body moving. That's it. Not the lower half. So this talk is called How to Move Mountains. It's a pretty broad topic that I'm going to be diving into, so I don't really have the chance to cover everything I want to cover. So I wanted to make note of that up front so that if you have any questions, please make note of it during the presentation, write it down, contact information's here, whatever format works for you, and reach out to me after, or talk to me at the con. I'm going to be hanging around, so I'm happy to have a conversation about this. So let's get started. Bligotory about me. I'm a senior security researcher at Security Compass, and these are all the different things that I've done. The top three roles in particular are very relevant to this topic that we're going to be discussing today. So let's actually get into what this talk is about. This talk is about something called DevOps. But in particularly, it's about how DevOps really is, or DevOps as the solution, is the problem for security and why that is. So let's first talk about DevOps as the solution. DevOps as the solution is really made to help improve a number of things in business and to really bring the business forward with through development. So I'll explain what I mean in a second. First off, it's about improving the speed of business. This is Mercury, the Roman god, who's the messenger of the god, so we had to be fast. Funny enough, he was actually the god, the patron saint of financial gain in commerce. So it's quite telling us to the importance of speed when it comes to business. DevOps is really there to equip the business with ways to start developing and iterating faster and faster. And they do this through a number of different ways of which I'm going to speak about soon. First of all, it makes developers more than just developers. So in a DevOps environment, developers are tasked with much more than just simple software development. They're tasked with being QA, operations, and security. Now at last point, if it doesn't terrify you, wait till we keep talking. So they're tasked with doing more today. DevOps is also about bringing us from the old, slow method of doing waterfall, all the way through to CI CD. So CI CD stands for Continuous Integration or Continuous Delivery. It can also mean continuous deployment, depends on what you say, but I like to think of continuous delivery because it's a little more realistic than continuous deployment sometimes. And Agile kind of fits in the middle. The way I look at Agile is it is a software development methodology that transitioned into us from waterfall through to CI CD. When I say CI CD, that's essentially the gold standard of what we want to achieve. In reality, only a handful of companies are truly doing it, but it should be something that we aspire to and it's something that, if you're adopting the DevOps way of doing things, it's really what you are pushing towards. So let's take a look at traditional SDLC. This is kind of what waterfall looks like in each iteration. So you go through the requirements, straight through to, I can't even see my laptop here, so I'm gonna look over this way. Crime is design development, testing and deployment. It's a standard, this is what we know, this is what we've been doing for years and years and it works, but it's slow. And often it ends up with this. When you're trying to merge together 50 different issues, 50 different features, all in one, all on one weekend usually, it ends up in long weekends because not only is it you're trying to merge together all these different issues and hope that it works out for the best, but you're also merging in issues and features that come from, that were developed in different environments. They weren't necessarily developed in this same environment. As much as we like to say, we equip our developers with these uniform laptops and they all have the same image off the bat, I guarantee you every single one of them has their own flavor of Vim, their own flavor of Ruby. Everything that they want and they use is usually customized. And so when you're bringing in those different environments, you're trying to put them all together in one go, it results in a long weekend and sometimes weeks of pain and suffering for developers, operations, QA, everyone involved in deploying these applications. So it's a pain, especially if you're doing six month plus release cycles, usually those drag on and become longer as you do this more and more because you experience this pain, you don't want to experience it again, so it just takes longer. So let's move on to continuous delivery. This is what it looks like. I struggle to actually find a real consistent pipeline for continuous delivery, so I created one based on how I've seen it. So this is what it looks like. But one key thing to note here is that it doesn't, look where it starts, it starts with commit. It doesn't start with, if you look at the last slide, is this working? If you look at the last slide, if this starts with requirements and design, those two things are essentially negated when you get to continuous delivery. They don't go away entirely, but they become smaller and smaller and they become something that the developer takes on themselves as well. There's a certain aspect of trust when you move to the continuous delivery pipeline. So you notice that it starts at commit and one other thing that I will point out about continuous delivery is that it ends at measure and validate, but it's not the one that you may think. Measure and validate oftentimes doesn't necessarily mean whether it's working in production because that's tested in all the steps before that. What it means is that you're measuring tests being done by the business. So think about canary testing, AB testing. These are tests that are done to measure the effectiveness of features that you're putting out, of new things you're putting into software, how it relates back to the business. So this is an important concept that really differentiates continuous delivery from waterfall, because it allows the business to see direct returns quickly on something that they want developed for the business to improve. So let's move on to the next thing I wanna talk about and that's DevOps as the problem. So this is where I'm gonna discuss how DevOps has become a problem, especially for security. Security has always been traditionally the gatekeepers. This has been our role when it comes to software development in an organization. Security is often full of manual processes such as pen tests, code reviews, even enforcing policy is very, very manual. And if we go back to what continuous delivery looks like, this is almost entirely automated, except for when you get the QA test. That is mostly manual, but there's still quite a bit that you can automate for that as well. So if we're gonna be the gatekeepers and we have all these manual processes, how do you think that's gonna scale when we're facing off against machines that do this quickly and efficiently? Does it? Especially when you come to this number right here. This is the ratio of developers to operations to security in the average technology company today. And before you say we're not a technology company, so it doesn't apply to us, everybody's a technology company. Everyone's developing software, everyone's using technology, nowadays it's not, you're not really relevant if you're not a technology company. So this applies to everyone. And if we're talking about manual processes, this is one security person holding up 100 developers as they try to release code but they have to wait for security to review it in one day, then the next day the same thing happens. The next day the same thing happens. Eventually the buildup gets too high that it just, it doesn't work and security doesn't work. This is another problem that we have when we're talking about DevOps is developers, because they're tasked with being QA, being operations, being security, they're tasked with doing a lot and it gets overwhelming. So a lot of the developers that I've spoken to, the most common complaint is I'm expected to keep up with engineering best practices, with development best practices, I'm supposed to do that really well. How the hell am I supposed to keep up with the latest security threats, with understanding how to write code securely and at the speed at which security is evolving, how do I keep up? You can't, it's overwhelming, it's too much. When developers are given the responsibility of writing code securely and they don't have a background in security, how are they supposed to keep up and get up to pace to be able to do that? It's a lot. And finally, when we're talking about all of these problems together, the way that security essentially slows down developers, if we slow down the business enough, eventually we're gonna get bypassed. This is gonna happen. And it doesn't become a problem until it is a problem, at which point it's a big problem. We all know what that means. So we don't wanna be the ones to slow down the business. So now let's talk about how to actually do DevOps and security together. This is where the title of the talk comes in. It's aptly named How to Move Mountains. It's possible, but there's a way you gotta do it and many people aren't doing this right. So I'm gonna tell you how to actually go about doing this. First off, we cannot be the gatekeepers anymore. We can't be the ones that our gating releases are slowing down developers because they're moving too fast and we're not gonna be able to keep up. So to use the same metaphor, we should tear down this gate and build a road together into the beautiful sunset that is continuous delivery. I know it's corny, but that's how we take the metaphor is we take down those gates and we help work with developers alongside them and not necessarily as these gatekeepers. So let's talk about how we actually implement DevOps and security together. There's two key things that you need to know. Two key ways that we can do this. First one is education. So the main idea of education is that because we are asking developers to write code securely, because we are empowering them with all these tasks and responsibilities, we need to make sure that they know what that looks like and it doesn't have to be difficult. So I'm gonna talk about a little bit more about not about what to teach developers, not what to educate them on, because that's been covered at length in a number of different conference stocks, number of different resources on that, and it's very organization dependent because of the nature of application security, AppSec is very dynamic. It's why I love it, but it's also why it's difficult to do because if you look at vulnerabilities in application security, they're not really universal across all applications. Think just one class of vulnerability, business logic vulnerabilities. That's entirely dependent on the application itself. So if we're talking about what to teach developers to write code securely, it's difficult to do that because it depends. It depends on the organization, it depends on the implementation. So I'm gonna talk about, instead of what to teach, talk about how to teach it. And I will start by talking about how we've taught it in the past, and then I'm gonna talk about how I've seen we can teach it as we're transitioning into this DevOps model. So first off, we have the Belt program. This is something that Adobe started and I believe Etsy's taken this on as well. So these are some pretty big organizations that are doing this. The way that it works is you can think about think about martial arts and how you transition from the white belt all the way up through the black belt and then there's different ones about that as well. But the idea is that you're transitioning through a series of belts. And the way that it works is you have it, there's at Adobe, the way they've done it is there were the bottom two, there's four belts. The bottom two belts, the way to do that is you have to go through an exam. Pass the exam, you get that belt. Awesome, it's like a certification. And then developers then display that in everywhere they're communicating, they display what belt they have because everyone knows what that means and it's a symbol of pride to have that to show that you know how to develop code somewhat securely. But when you get to the last two belts, this is where the rubber really needs the road.  So you have to practically apply what you're learning to get those final two belts, which is not easy, but it can be done. So think about writing a webhook that connects into your Git system to do some kind of linting or parsing on code to make sure it's done more securely. That's an example of it. We're doing a certificate management program or a secret management program, things like this where you're applying what you're learning to the organization and it helps overall to improve the security of the organization. That's how the belt program works. Next is security champions. So this is a program in which you have developers or people with an IT background or software development background in the organization who come from these teams and then they become an advocate for security or a champion for security. So by doing this, they get taken into the fold of security. They learn what they need to learn specific to the organization and then they go back to their team and they share that and help enforce that. This helps scale security because rather than having the 100 to one ratio that we have, you now have 100 to one plus one for each team and it spreads throughout the organization because security can't always be side by side with developers but their other developers can. So this is how security champions works. They typically take care of a lot of the more low level security implementation things with developers. So the next way that we do implement security education is your center of excellence. This method is usually built upon security champions because it can help it but it doesn't have to be. The way that it works is you have a group in the organization that is this security center of excellence and then they are in charge of looking at more forward facing issues and building upon the existing implementation, the existing operations that we have in a company. So they're looking at once you have this baseline of security, how do we improve and how do we face the next round of threats that are coming our way? It's building on that standard that we already have. And the way that security champions helps is because this information needs to be disseminated through the organization because the security center of excellence team is not necessarily doing the day to day operations. Instead, they are the ones that are looking forward and then they need to share that back with the organization. Security champions helps because those champions can be the ones to help share it amongst the teams. So this is the center of excellence. Finally, I'm gonna talk to you about a newer way of doing this that is really a hybrid approach and it helps moving from waterfall, moving from these slower development cycles where we have iterative cycle security as a gatekeeper into a method where each developer needs to do everything. They need to do operations, QA, and security. So we're asking them to do this, now we need to show them how to do this. And so this is called continuous learning. So we have CI CD, we now have CL. So it's easy to remember. It's essentially a framework to enabling and empowering developers to be able to write co-securely, to understand what that means and to be able to continue to do that on their own. So first, there's four steps to this and the first step, it starts with threat modeling. I have to take a minute to take a look at this photo. Yes, it is a man in a ski mask with gloves holding a wayboard. I didn't even care that there was the water marks on it.  If you can do it, if you can take the time to be able to properly do a threat model and get a full understanding of all the risks inherent to your applications, that's amazing. You have a much better insight into your own application security than 99% of the other organizations out there, but it takes a lot of time and that's the pain of it. The way that you do threat modeling and continuous learning is different. There's two main ways that is different. One, it's focused on one of the three layers of the CIA triads. You have confidentiality, integrity, availability. We all know this, this is the basic things that we look for in security. So you take one of those and then you work with an individual developer or a team and you go through their application that they're developing, targeting this particular aspect of security. So you're not trying to get a full understanding of all the application security issues because there's gonna be a lot, instead you're focusing on one thing. The goal of this is not to get this full understanding, the goal is to get developers to start thinking about security, but not only that, get them interested in security because everyone here knows how fascinating it is. It's really, really cool. And if you can get that spark of interest into developers about security, then you've already won, frankly. Because we know that that spark turns into fire very quickly as we all pursue the individual things that we love about security. So you get them interested and you get them thinking about security. Another important differentiator between this and the typical threat modeling is that this can be done on onboarding because it's small and quick enough, one of the main problems with developers starting in any new organization is getting them up to speed with the applications they need to build. It takes a long time, that's the hardest part. But if you do this threat modeling exercise, you're introducing them to the application so they already have a better understanding of it, but you're integrating security into that introduction. So you're thinking about security already right from the start. So this is why it's important and this is how you start with threat modeling. So we're getting them interested in getting them thinking about security. Next step is regular check-ins and I will explain this image in a second. Regular check-ins is the point where security goes to developers. So we go to them and we bring security to the top of their mind regularly. So this can be done through stand-up meetings, through not necessarily pair programming, but through answering questions, but going to them and initiating that conversation. And there's a number of different times you can do this. It all depends on how you're doing software development in your organization, but that doesn't matter. What matters is that you're bringing security to them. And you're not doing things for them, you're doing things with them. So we're not sitting down and writing their secure coding libraries. That's not our job. We are training them to do that. We are helping them figure out what that means. So this is what regular check-ins is all about. And the reason why I have this picture is because Google actually does this themselves. Where they have a team called the testing mercenaries whose sole job is to go around the organization and help developers implement tests into their development life cycle. And so I did a very poor job and I didn't waste much time on this, but these are the mercenaries from Lost. And I Photoshopped in a source code a little bit and a whiteboard. Very poorly, but it's the thought that counts, right? So hopefully you remember this. So after regular check-ins, we're going to developers. The next step is having an open office. So this is, so we've transitioned from introducing, getting them thinking about security to going and keeping security on their mind regularly to now, because of the highly dynamic nature of application security, every challenge they're gonna be facing can't be solved by one simple solution. As much as vendors will tell you that their solution will fix all your security problems, there's no one thing that will fix every security problem, especially when it comes to application security. Because like I said before, each problem is highly dynamic and unique to the organization and the application itself. So they're gonna have questions. They're gonna be thinking about security, they're gonna have this introduction of basic knowledge, but they're gonna have questions about how to actually implement what they're learning. And so at this point, this is where you invite them in to ask the questions that they need to ask. And at this point, because we've already gone through those first two steps, one thing I didn't mention about those steps is it helps build and establish a relationship between security and developers. Because as I said before, we need to tear down that gate and we need to help build a road with developers. We need to sit alongside them and work with them. So at this point, we have that relationship and they are coming to us to ask questions and we can answer questions in a more nuanced way. Some of the best, most beneficial moments that I've had with organizations when I'm working with them to improve security is when I sit down at lunch and I have a developer asking me about a problem they're having. Because then I can explain it to them in a more nuanced way and I can actually work with them in the context of the application they're building. Having that conversation is key. And so that's why we have open office. And if you think that this is difficult to scale, Capital One, one of the biggest organizations around does this themselves, where they have subject matter experts that have open office hours where people can come in and ask questions about that subject. So we will do this very same thing with security. So as you can see as we've moved transition through these three different steps, developers, each individual developer is becoming more and more capable and independent when it comes to security and we're thinking about security more often.  And so this is about maintaining interest and yes, I will play you the Jeff and I will stop so that it doesn't distract you the entire time. So maintaining interest, what does that mean? This is a way of keeping security at the top of their minds because security as we all know is fascinating. It's really interesting. There's so many different things we can talk about. There's another talk right now going on. Please don't leave right after I say this. There's another talk going on right now about hacking medical devices. How cool is that? There's so many cool things that we can do in security. And so making sure developers know that is a way of keeping that interest going and keeping them thinking about it. So to do this you can do things like doing speaker series where you have someone from inside the organization, outside the organization, it doesn't matter. Someone who is doing something interesting in security that they wanna share.  You can do capture the flag events. It's a little more intensive and difficult to set up but you can do that. You can also invite them to go to security conferences like this one. Different ways to bring and keep security interesting for developers is really important because it's not forcing security on them anymore. It's making them interested so that they wanna do it on their own. And this one is kind of maintains that continuous aspect of continuous learning because you're bringing this back to them so that they keep interested.  So these are the four different components of continuous learning and this will help bring you from the waterfall cycle where security is doing everything to the more independent way of doing software development where you're doing CI or CD or even as you're transitioning through agile. So that's what continuous learning looks like. Now the last thing that I'm going to talk to you about the next final component of building security into DevOps is automation. And this is key. However, without education, you will see that it doesn't work as well but education augments this. But when we're talking about automation, because we have that really fast automated pipeline, you need to keep up with that in security as well. And I'll talk about what that means. But one thing that you need to remember when you're talking about automation the key thing is we're trying to build an integrated pipeline with actionable results. Keywords being integrated and actionable. Remember that.  Because like I said, there's no one tool that's going to solve everything. There's no framework that's going to fix all your security problems. It doesn't exist. You're going to have to make something a little more customized. And especially at scale, this is kind of the solution because you're going to have so many unique edge cases. And so you're going to have to build this automation pipeline alongside this continuous delivery pipeline. So each stage on here can have tests, can have automation built into it. And as you build this automation pipeline, keep in mind what this looks like because you're going to be able to tie into this as well. So how do we do this? One of the ways we do this is we start differentiating between network security engineers and application security engineers. Instead of just having the typical security engineer that an organization has, differentiating between network security engineer and application security engineer is important because app tech engineers come from a development background, usually. Or they're passionate about development. They like that aspect of security and use that. So speaking for myself, I love building things. I love writing software. If I can integrate that into security as well, then I have the best of both worlds. And because you're going to be building out this custom tooling, these are the people that are going to help you do it. Your application security engineers. One thing you're going to have to remember is starting small. You're going to have to start small because you can't build it all in one day when you're even a month. Doesn't matter. No amount of time. It will almost never be done. But you're going to have to start small and you'll get there. Just remember this. You will get there. As frustrating as it may be the process, you're going to get there and eventually you're going to build this. Google themselves, and I will call back to Google because they're an example of a company that's able to do this at scale. They have over 120,000 individual test cases. It's a big number. But they didn't do this in a day, a year. It took them many, many years and they have over 100 different test engineers that help build these test cases. So you'll get there, but it just takes time. Start small. When you're building these tools into this pipeline, you're going to have to make modular tools. This is not just a good best practice for development in general. It's good for building these tools that are going to integrate security into your pipeline. Because remember, one of the key concepts is integrating your tools in into your existing workflow. And this has a side benefit of having your AppSec engineers or your developers that are working on this. Their background may be in Python, it may be in C, it may be in Java. It doesn't really matter where they're coming from. If you use something like, let's say Docker, they can build on whatever they want, whatever system they're familiar with. But as long as you have a standard input, output, or inputs and outputs and documentation, don't forget documentation, then you're going to be able to integrate them all together. They're going to be able to plug into each other really, really easily. This is a key concept to remember as well, because one of the hardest parts when you're building this security pipeline is where to start. What test cases are you going to write first? Key principle that you're going to need to remember is don't repeat the same mistakes twice. Don't ever repeat them. If you can avoid it, do that.  So if you're having to repeat a security vulnerability, write a test to catch it so that it gets, it gets stopped before it gets pushed into production. This is one of the key ways to start integrating and catching and writing these tests is to look for things that have already come up before. Not only that, this will help you, from a more business standpoint, this will help you get by because you're going to be showing immediate results. If you can stop that same thing that comes up on a pen test year, after year, after year, which we've all seen, you can stop that, then that's a huge win. And that's an easy way to get support from people way up on high in the organization that you need to keep doing this because, like I said, it's going to take time. Another key concept is unit tests. This is something that is built into every single developer's workflow or should be built into most people's workflow. Every modern software, every modern programming language has some kind of unit testing framework for it. And likely your developers are already writing unit tests. I hope they are. If they're already writing unit tests, then why not integrate security into finding those same mistakes? Integrate these security tests into the framework they're already using. So I don't mean writing tests to anticipate future security issues because that's difficult and often won't work out and we'll just slow things down. Remember, we have to keep this actionable. If we write unit tests about things that we've already seen in that application, then it won't happen again. And developers already know how to write unit tests and so it's integrated into their existing workflow. Really, really important. One other thing to remember when we're talking about unit tests is that when crunch time comes, which it will, developers are pushed to start pushing out features and they don't have time, one of the first things to go is unit tests. So this is where the importance of measuring test coverage comes into play. So there's many tools out there to do it but you can ensure that you have a certain level of test coverage across all of your applications. And then for example, one way to action that would be to fail to build if there's less than 80% test coverage. 80% is a good benchmark to look at. Realistically, 100% is not really useful. There's a point of diminishing returns. 80 is about that point. So integrate unit tests with security as well. I haven't forgotten about the systems that they're running on. I haven't forgotten about the infrastructure side of things. A key principle in DevOps is infrastructure as code. And what that means is you're storing the configurations for your different systems that are running in production the same way developers are storing their source code. So they have the configuration saved in a file and that file is being checked into something like get for easy rollback, for tracking. It makes things a lot easier. And you don't have to worry about going into an individual system in production, trying to tweak that. Instead you have the configuration to fall back on.  Using these tools to do infrastructure as code makes building the security pipeline way easier because then you can target your infrastructure as well as your applications. So it not only helps improve your releases into production, it will also help improve your testing of those configurations as well. Pentests are important. They're still important. They don't stop being important because they help you catch and find those deeper, more technical challenging issues. Those things that you may not have spotted through a unit test. They're gonna keep coming up.  You're never gonna catch everything. This is where pentests come in. But where they help you when it comes to security automation, use the results from the pentests as sources for what you're gonna test for in your automated pipeline so that like I said, we don't have the same pentests being done year after year and finding the same results year after year because that happens. Use those pentests and feed it into that automation pipeline. It's one of the best ways to make use of the pentests. So it doesn't just become a PDF that sits in your drive somewhere for eternity. If you already have these existing tools in your environment, these big scanners that try to catch everything, it's okay to have them in your environment, but there's gonna be some things that you need to do. Before I get into what you need to do with these tools, you can make use of this when you're building up this pipeline as compensating measures.  The hardest part is finding the things to test for, but until you have bigger and more test coverage, use these to try to fill in those gaps. It's fine to do that. You probably already have them. And just to check how many people are paying attention, this doesn't exist, so don't bother Googling it. I just made it up. It's a pretty good logo though, so I mean, if you see Cloud Champion LLC coming out next week, you know where to look. First thing you're gonna do is a good practice in general when you're using these tools that you're gonna have to tune in. Because remember, one of the two key aspects, two key things to keep in mind when you're talking about automation, is that the results need to be actionable. If you have 2,000 results from a Fortify scan, how many developers actually can go through that and fix each individual issue?  Or that Fortify scan will magically disappear.  We don't want false positives because it's just a waste of time. Be sure of those findings and make it so that it's a manageable amount that developers can actually do something with. So they don't have to spend half their time or more pruning out false positives or figuring out which ones are actually mitigated by something else. So tune the tools to make it easier to actually action those results. And finally, you wanna also make sure that these tools have an API. An API helps because it allows you to plug in this tool that you're using into your pipeline that you're building. Remember how I said making those tools modular is important? This also makes your tools modular in a way because it has a standard way of getting input and output into a tool. You don't have to worry about the intricacies of that tool. This makes it easy to interact and integrate that tool with your existing pipeline or the pipeline you're building. So if your tool doesn't have an API, reach out to your vendor, try to get them to put it in. I don't know what to tell you. APIs really help. So finally, after all of this, all of these things that I'm telling you you need to do to make sure you have security in DevOps, it's hard. There's a lot involved in it, but you can do it. The hardest part is knowing where to start. So I'm going to give you a few key things that you can use to start building this and it'll help you get that momentum going. First off, SD elements, full disclosure that's built by these guys that I work for. It basically allows you to plug in what kind of application you're building and then it'll tell you exactly what you need to know to write secure code for that language, for that framework, whatever you're building. It's used in the requirement stage. So if we're talking about the old school SDLC where you have requirements designed, all of that, that's gone and continues delivery. Use this to help understand what test case you need to build. So that's the hardest part is understanding what to build first. This will tell you.  That's pretty helpful. It's a little older, but it's some common software security issues that you might see. And one of the key sources of input for building this pipeline is old findings and mistakes, whether that's through pentests, vulnerability scans, whether that's just through having an incident in production, right? Look, go through all your old incident response tickets and see what you had to respond to. Those are some of the key sources that are tailored to your existing environment to try to find first. There's some other tools and I'm gonna point out some specific tools mainly because of the organizations that built them to show that it's possible to do this at the scale that they're operating at and that they are doing it. So here's a few open source tools that have been developed by these companies to do automation, to do efficient testing that they're using today. So Netflix has a couple, they're really well-known for this. Simi and Armi includes the well-known Chaos Monkey tool. Etsy is really good at doing this as well. Yelp has a tool, I'm not sure about how much into continuous delivery they're doing, but they've released tools like this that help. And Breakman, this one's actually pretty interesting. This is a security scanner, a static analysis tool for Ruby on Rails. Twitter actually went out because they used it and they found it so effective, they hired the guy that wrote it and they're still developing it. So these are some tools that you can use to A, see how it's being done by these big companies. It can be done at scale and B, to start plugging into your own environment. They're open source, you can modify it if you want and you can also see how it's being done. So these are some other tools to use. Now I'm gonna talk about AWS specifically right now because most organizations, most people in this room are likely using AWS in their organization.  But until then, these are things that you may not already know about that exists in AWS that you can make use of to help you build the security pipeline. First is AWS Code Pipeline. So AWS Code Pipeline is essentially a continuous delivery pipeline builder on AWS. It has a GUI if you want, it also has it, you can do it from the code level but it allows you to build through these different continuous delivery steps natively in AWS. This is like AWS Inspector is kind of like a Nessus Lite. It's not quite as good, it's not as full featured but it's just started. So it's very brand new, specific to AWS environments. So it's a vulnerability scanner built in AWS. Remember how I mentioned infrastructure as code? This is what it looks like in AWS. It's cloud formation. So if you're building based on these templates, you can integrate these into your testing pipeline to actually scan your configurations as well as your code. AWS Config, you define a configuration for an environment, what it should look like. If it falls outside of that configuration, it will alert you. And CloudWatch events. So CloudWatch and CloudTrail are really useful because they tell you what's going on in your environment but that may not be quick enough for you. If you need, and this is very likely that you're gonna wanna do this, if you wanna action them right away, if you wanna do something about those events immediately, Lambda allows you to plug into those CloudWatch events and do something and action those events instantaneously. So if you're not using any of those, it's worth looking into because it will help you integrate security much more efficiently into your existing environment. And so finally, I wanna leave you with this message. Combining education and automation, you get remediation. Now there's one last key point that I've kind of saved for last. Using DevOps and using a continuous delivery model is not only effective for pushing out features, but one thing that we fail to realize often is that by building this pipeline, we're not only building a pipeline to efficiently push out code to make our product better, it also allows us to push out security fixes faster than ever before. Think about it, you have a pipeline in place to push code into production immediately. That doesn't have to be something that we are scared of in security, it's something that we need to embrace because that can help us out immensely. And it's something that not a lot of people realize as they're fighting against and railing against developers doing all these things on their own in the speed of DevOps. So if you remember this and you embrace what I've talked about today, you'll find that your security operations will become faster, better, and you can respond to incidents faster than ever before. So here's some key points for you to take a picture of if you want to take home with you, but that's everything I have to say today. If you have any questions, feel free to ask me anytime. Thank you very much. Before I go, I think I still have like four minutes. If you guys have any questions you wanna ask in front of the group, feel free to ask and I can answer it right now. Yes, sure. So the question was for those that didn't hear it, is that he's a client, it's a client, right? Client that wants to integrate static analysis into their environment, worried that it takes too long to actually do these scans before being pushed into production. Yes, it does, it takes a while. As we've seen from the thousands of results for most static analysis tools. Where we can fix that is by encouraging them to build these individual tests to start finding these things in a modular way. So by making these little modules, you can call the ones that you need if you really want specific to that application and those will be done much quicker because it's small little chunks instead of something that's running everything at once. Another way to look at it is to split up the tests into the different stages of the pipeline. So you'll have different tests running on commit, different tests running on build, different tests running on push to production. So for example, you might wanna look at, when you're looking at commit, do a simple linting for, look if SQL's queries are being concatenated with user input or there's any concatenation in a SQL query. Simple things like that will catch things quickly right away so you don't have to worry about covering those simple things later on. So if you integrate smaller tests into the different stages of continuous delivery, then it happens much more quickly and much more efficiently. It takes time to build that, so it's gonna take a bit to get there, but if you encourage that as the goal of what you wanna move to, it will solve both security and developers issues because you're getting the security coverage and you're doing it quickly. Does that answer your question? So this is where, this is something that I didn't really talk about and this is kind of the reality of continuous delivery, is that if you have a very low risk tolerant organization, say financial institutions, most big public facing parts of government, continuous delivery probably won't work and they shouldn't necessarily be pushing towards that. As much as I say it's like a gold standard of development, that's for businesses that need to compete a lot in the marketplace. If they don't have to worry about that and they're very low risk tolerant, then yes, you can push faster, but it gets to a certain point where there is that trade-off of being able to make mistakes in production versus not. So I think that to a certain point, we should be okay with pushing them to go faster. If that's where they wanna go and they're okay with that, then we're gonna have to go along with them and move fast as security, but if they have that really low risk tolerance, then it may not be that continuous delivery is right for them. Is that answer? Okay, is there any other questions in the minus 30 seconds I have? You know what, if you have any other questions, please feel free to approach me outside of the conference or reach out to me anytime. This is my contact info and I'm happy to talk. Thanks guys. It's not to. Yeah, no problem.


This study evaluated the response in lung function to different treatment regimens for common variable immunodeficiency, CVID, patients with granulometous lymphocytic interstitial lung disease, GLILD. The study found that corticosteroid treatment improved gas exchange in CVID patients with GLILD, but patients often relapse. Mycophenolate mofdil, MMF, was associated with long-term effectiveness in GLILD and allowed for successful weaning of corticosteroids. Delaying initiation, or continuation of treatment, could lead to disease progression. This article was authored by Heber M. Bintalib, David M. Lowe, Gaya Mancuso, and others.


So, we've been commissioned this task, this tender, and our aim here is to identify a sustainable model and when we're talking about a sustainable model, we focus on economic and financial sustainability. Okay, so, the aim of gathering together this team is to identify a sustainable model that is methodologically consistent to what an economist would call a sensible way to manage an infrastructure. Okay, our team consists of economists and econometricians, archive specialists, accountants, and a lawyer, institutional analyst that gives us the consistency of our method and at the end of the day our proposals with the existing legal framework so that we can talk to the other team who is analyzing the legal framework. Okay, now the two main objectives is to achieve an accurate estimation of the benefits and the costs of this infrastructure and based on these benefits and costs and when we are talking in economics about benefits and costs, we are talking about how much it costs to build the infrastructure, to run the infrastructure, and the benefits of any users and the users could be individuals, could be organizations, could be institutions, any relevant stakeholder and based on this benefit and cost structure which is you should think about it as time series of benefits and costs because this sustainable model, sustainable means that it can live for many periods of time. So we need the dynamic benefits, the time series of benefits, the time series of costs that can support this infrastructure in a sustainable way. So how are we going to do it? Identify the stakeholders and measure in monetary terms the benefits they derive from using the infrastructure. We're going to identify the cost of building the infrastructure, operating, maintaining the infrastructure and then we're going to put this together in a dynamic and cross-sectional way because it will be different for different stakeholders and different repositories. Compare the two and see if the costs are lower, higher, equal, done and benefit compared to the benefits and we are going to do this over time. And once we identify whether it is something that is so increasing social welfare, that the social benefits, the benefits of the involved stakeholders are higher than the cost of sustaining or building, sustaining and investing the infrastructure, then we're going to identify who is willing to pay, for what services, how much, what is a sustainable pricing and business model for this infrastructure. We have the following deliverables. We've constructed the methodology and it's already uploaded in the Wiki and I will try to show it, to show the whole methodology in one slide. We have done the stakeholders analysis and the person who leads the stakeholders analysis is Oya from Cornell University, Oya. We have started working on the cost analysis. I will introduce the explicit method which is a survey based method that then uses the survey results, analyzes the results using conometric analysis in order to identify the stakeholders valuation of the services of this infrastructure. And we're talking about monetary valuation and then we're going to do the cost benefit analysis and based on the cost benefit analysis we will be able to identify how much the infrastructure and services cost, who values the infrastructure and services, how much they are willing to pay, what are the revenues, what are the costs over time and based on this we define a sustainable business model. Unless you do all this you cannot really identify a sustainable business model and because I knew that I wouldn't have time to guide you through my four slides I tried to put everything in one slide. Let's try to make sense of this. Okay we have the open air infrastructure, open air plus and we have it because somebody believes that it creates benefits to stakeholders but it also costs something. Who are the stakeholders? This question will be answered through the stakeholder database that we've created, through the stakeholders mapping and prioritization and analysis that we already contacted. Based on the results of the mapping we are going to introduce a sample selection method. Sample selection for both to identify the sample that we will apply the stakeholders questionnaire which is a questionnaire that we've put together in order to identify the services that the stakeholders identified in our database value in order to understand what the different stakeholders get from open air infrastructure and then in order to identify how much different stakeholders are willing to pay for different services we're going to use the so-called choice experiment methodology which is an econometric methodology that allows you to derive willingness to pay for specific stakeholders for specific services. In principle in economics if you don't have any diversions from the preferred market this should be equal to the margin of willingness to pay should be equal to price but leave this aside for now. On the other side you have the costs, you have investments, operation and maintenance which more or less summarize the different costs that are involved. These cost identification and estimated figures for different aspects of the infrastructure of operating and sustaining the infrastructure will be derived from the cost point we've put together. And then when we have the benefits and the cost we will then try to identify different sources of revenues. These are examples EU grants, institution contributions mainly up to now they subsidize in initial investment they give the initial investment but there are other potential revenues like revenues from providing access to articles from subscriptions, from contributions, discovery services, research funders and so on. And at the end of the day we need a financial net present value that is bigger than zero. These are the revenues and costs in each period of time discounted by I which is the interest rate that is the opportunity cost of capital the fact that you decided to invest in this form of capital and not in some other. So this should give you if your project is sustainable a value that is higher than zero. If it's not and you your revenues are not enough to cover your costs but you see together value of the services that is the benefits here are higher than the cost but revenues are not higher than the cost then the cost benefit analysis will identify the most efficient solutions for subsidizing this infrastructure from EU grants and institutional contributions and based on these results and only if you do or if you go through each and every step of this methodology you can identify pricing and financial schemes and the relevant business model for sustaining any infrastructure. It doesn't have to be a science infrastructure it can be any infrastructure. Our project results are all available on the project wiki okay so you can go there and you can find clicking a sustainability group you can find the proposal minutes of meetings the bibliography the presentations and then the stakeholders database the stakeholders description results from stakeholders analysis the stakeholders questionnaire that will give you identify the most important services for the stakeholders and also the cost questionnaire. Moreover the stakeholders database is also important because here we have scientists and researchers research funds research centers publishers scholarly societies and so on we have 15 different groups of stakeholders okay and you can actually go there and and suggest revisions or additions we also have a description of the stakeholder and again see again is available on the wiki and here we have categorized and mapped the stakeholders according to their contribution to the legitimacy of their claims willingness to engage influence and necessity of involvement and if I go a bit further down you can see that this categorized categorization can enable us to map the stakeholders according to their importance according to different criteria and here I put on the vertical axis expertise and on the horizontal axis willingness to pay and then what we did is that we mapped each of the 15 different categories of the stakeholders on this map why in order to understand their importance with regards to the open air open access initiative and sorry about this and by mapping the stakeholders what can you achieve at the end of the day when you are going to implement the questionnaires the stakeholders questionnaire the cost questionnaire and the willingness to pay questionnaire the questionnaire that we try to identify willingness to pay you can create a sample selection method okay your sample of who which stakeholders you are going to approach that is from the population of stakeholders you are going to select the sample will be faced on this prioritization from these stakeholders analysis who is important and who is not and at the end of the day you get a map with the different stakeholders when you then translate into weight into the sample selection a method and then once you selected a representative sample of the population of interest which is based on these weights you are going to implement the two questionnaires the first questionnaire is trying to identify the explicit cost of of this infrastructure repository annual cost coordination and national open access at this annual cost cost of infrastructure services and these are the main categories and if you go to the questionnaire these are electronic based questionnaires you can see that the questionnaire allows you to differentiate between investment cost and running a maintenance cost which are important for economic analysis and cost benefit analysis and this is the cost questionnaire and I would ask we will contact you if you are in the sample that we selected and ask you to respond to this questionnaire the first feedback that we have is that this questionnaire is is doable first of all where people can respond to it the people who are managing these infrastructures can respond to it and if you see at the end of the day we say please indicate how you expect to finance the above cost so we get an initial reaction with regards to where the revenues are coming from and these are reactions from the people who now are managing repositories okay revenue from posting articles and access to article subscriptions and so on and then from the cost questionnaire we go to the stakeholders questionnaire which is a bigger sorry about driving you back and forth but I want you to have a glimpse of the stakeholders questionnaire as well uh so let me show you part of the stakeholders questionnaire so first we introduce open open air initiative to the stakeholders because this will go to people that are not familiar with open air uh and we tell them a short introduction we give them a short introduction to the aims of scope and so on and then we start asking questions how will how familiar you are with this initiative and then in order to try to understand the the connection of the stakeholder with the open access initiative and then we ask the categories whether it's a university a library and open access and so on and then we ask questions about how does open air support your organization and what is the service policy gap that open air can feel how do you see yourself contributing to open air and in general all these are questions in order to give us a first view which we can analyze statistically on the value on the on on the use of open air from the different stakeholders so this will allow us a first mapping of the stakeholders preferences priorities benefits service important services and what they they think about the financing of this infrastructure and once we have the results from the implementation of this questionnaire we will uh uh we will okay let me put the slideshow here once we have the stakeholders questionnaire we will be ready to put together the willingness to pay questionnaire we will be ready to evaluate to estimate how much the different stakeholders value in monetary terms the services they derive from the existence of open air it has not been done in the literature before and we are going to do this using a state of the art methodology that is called choice experiment and the choice experiment is a survey based questionnaire that allows you to analyze the preferences of stakeholders it allows you to extract the total economic value different stakeholders are tribute to different services or goods and the way you do that and the total economic value includes use values and non use values and non use values might be positive externalities from having open access to scientific results that are not going to come today but they are coming they will come into the future but they are valued by the society of the relevant stakeholders today so there are many values that are involved in sustaining these infrastructures and unless we are able to identify them and measure them we will not be able to strongly strongly support the significance and even current subsidization from governments and the european union of these infrastructures and this choice experiment methodology actually uses some mathematical algorithms to present the person you are implementing the questionnaire to the respondent with different levels and quality of the different services that will be identified from the stakeholders questionnaire so if you from the stakeholders questionnaire if you identify five very important services that stakeholders use from open air infrastructure then you can present different cards to the stakeholder to the respondent with different levels of quality and quantity of these services and then use a price vehicle and asking to choose which combination of services and price they prefer and in this way you can buy you not just open air infrastructures and hold but you can put a value on the different services and on the different levels and quality of services so the choice experiment has been used extensively and has been used to pack up cases in courts for valuing public goods valuing in monetary terms public goods and nowadays it's within the european guidelines of course benefit analysis you explicitly it's explicitly says that you need to use valuation methods in order to support any subsidization of public infrastructures of public goods that are not currently sustainable but they produce important socioeconomic benefits in most for example environmental directives you get exceptions and extensions for implementing the directive by presenting results from these valuation studies that give you a positive cost benefit analysis so if the cost benefit analysis is positive then economically you should sustain this infrastructure you should sustain this public good but it might be the case that you cannot really financially currently be able to sustain it without subsidization so there's where the government comes in and says this has important social benefits for this community so i will subsidize it because it sustains these benefits at the moment what state are we at well we completed the methodology we've completed the stakeholders questioner we completed the cost questioner we completed the prioritization of stakeholders and completed the sample selection methodology the next week we are going to start with the implementation of the different questioners the analysis of data and by february we will be ready to put together the valuation questioner the choice experiment that will give us the monetary benefits and hopefully april we have the econometric analysis have the benefits already have the cost from the implementation of the cost questionnaire and be able to do the cost benefit analysis and once you do the cost benefit analysis you are able to identify the the financial model the sustainability model that can support your infrastructure or different versions of the infrastructure so financial sustainability means that we have enough cash to cover the annual project cost without running the risk of procuring cash shortages it's very easy it's very common sense and we have to we have to build this sustainability we have to build the model given the cash flow of operating costs the time profile of the interest payment lower reimbursements taxes pay which are relevant to a project and also the sources of financing financing the investment cost so based on this cost that we will identify through the cost questionnaire we need to compare this with what i call are the revenues which are the revenue cash flows of the project and and try to see if we can have financial sustainability for a long time horizon if not then we need to find additional economic tools that will guarantee this sustainability as long as we can prove that the cost benefit analysis is positive and here you can see a project which is financially sustainable with total inflows and total outflows where you have a positive cumulative cash flows and here a non-sustainable one where the accumulated cash flows are not so it's easy to distinguish between the two and at the end of the day we will come up with a model that is consistent with the willingness to pay and is sufficient to ensure financial performance and i i should leave it at that there are different business models but unless we go through the procedure that i've showed you today you cannot suggest you cannot recommend any of this because you don't know the basic cost and benefits that really one derives from this i think i'm used to my time thank you thank you very