Hi, welcome to our presentation. We want to introduce today Yaouk to you to fearless automate the installation and deployment of an OpenStack cluster. So, first we want to show you a bit what Yaouk actually is and why we think we needed to build it and how it works. After that we show you an example about that, but first we want to introduce ourselves. So, hi, I'm Robert. I work for StackEd, which is part of the Schwartz Group. Maybe you know the Schwartz Group when you are from Europe or the U.S. as the retail companies Lidl or Coughland. So, we're part of the Tau Group. We have some many other companies in that group, but basically we are the job writer for internal and external customers, for the Schwartz Group and also other ones. And currently we run multiple OpenStack clusters with more than 700 hypervisors and about 40,000 physical CPU cores and 12,000 VMs. And I'm Stefan Hoffmann from Cloud and Heat Technologies. Our version is to build a sustainable and silverware cloud stack, that is not only operated by Cloud and Heat itself, but we will also enable others to use that. And the sustainable part we want to achieve with hot water cooling, though we have in software and in hardware stack, you can be there. We use the water, for example, with integration into buildings. This needs to be monitored and operated, but also the software stack above. We use Yahoo! to roll that out. And yeah, of course one OpenStack, if customers are interested, so Kubernetes is on top of that. But enough with the marketing stuff. Let's go to Yahoo! what it actually is. Though it's a tool to install and also operate OpenStack. We want to show it to you a bit. And the name is yet another OpenStack on Kubernetes. And it contains three parts. Today we will only focus on the operator part. But somehow we also need to install the bare metal nodes. So we have also automation for that. On top of that, we want to run Kubernetes, bare metal Kubernetes, and then can deploy the Yahoo! operator. But you can run the Yahoo! operator on any Kubernetes cluster you want. Though you don't need the stack below if you have some other automation for that. Yeah, the question is why to use Kubernetes for OpenStack. I mean, the idea is not new. We saw it at the summit a lot already, but still OpenStack is really stateful and Kubernetes like to kill pots. So why to do that? Still, you can define your workload pretty nice with Kubernetes. So you can define this is how my infrastructure should look like. And then there is some magic that make it happen. And you can use some Kubernetes features like replicas. You can say, I don't want to have three keystone APIs, but maybe 10. And Kubernetes make that happen for you. That's pretty nice. Also, it comes with networking, load balancing, lifeness probes, and so on. That's pretty nice. And you can, like I said, you can easily test it because if you run it in your test environment, you have a Kubernetes cluster based on virtual machines. And you can pretty nice test that. But how do we actually install OpenStack? We could use ham charts, but no. We use another thing from Kubernetes, the idea to use control loops. So we define something and then have a control loop that regularly checks. Do we actually have this state? So we define nearly everything and keep it in Yauk with custom resources. And then the Yauk operator checks this custom resource definition. And run this control loop. Do we have a user, for example? Do we have a configuration? And then creates the deployment out of it. This is an example of how a client's deployment looks like. You don't need to understand it completely, but like you see, we define database and API stuff, like the replicas, but we also can define configuration. And that's one thing we wanted to achieve, that we are able to configure every option that OpenStack allows you to configure. But also we define the target release as we support different releases. And the operator then goes through some things to achieve that this is actually made happen that looks like then like that. I'm not going through this today, but that's an dependency graph. So Yauk knows the dependencies of a resource. So, for example, it knows if I want to have a config, first I need to have a database. And a database user. And if I have that, then I can create the config. And I need the config to generate the API deployment. And that's pretty nice because we have then a defined order of all the things. And that's basically the core of this reconcile loop. Though the loop can check if something updates in the definition before and then updates this part that changed, actually. And also, it will fail if it won't continue if one step somehow fails. For example, it won't create the API deployment if the configuration failed. But what means configuration failed? If we want to define every option that is possible, I mean, for Glance deployment, it's quite easy you have a Glance config. But for no more compute nodes, you maybe want to have different configuration for different nodes. And then you could have multiple configurations for the same or multiple values for the same option that are different. And that's not good. At Yauk, we see that because we have a Q-lung validation and then fail if we define different values for the same thing. We use Oslo config to see what options are actually there and supported and that thing is defined as an integer if it needs to be and not a string or something like that. And we have some common default set in Yauk as well, like the database connection. If we generate the database before, we know already the connections thing and the user don't need to put it in. But the really fancy thing out of the things I explained before is the day two operation. Because we know what changed, we can easily update and react on that. I guess all of you somehow operated already an OpenStack deployment and know how hard it is to train a gateway node. If you need to remove 500 routers from an L3 agent, that's not so easy. Or let's take no more compute nodes. You not want to live migrate all. We empty from the hypervisor to update that. I mean with 10 hypervisors, that's fine. With 500 or more, that's become time consuming. So you have some script for that also. Or you have Yauk. Because Yauk does that for you, it knows what to check. Robert will show it a bit in detail later how we erect the nodes. And this way we can easily update the config and introduce config changes or updates of the containers. So basically after you learned a lot about Yauk in theory, I will show you just a small practical example like how the life cycle management of a compute node in Yauk works. And just to make it a bit more understandable how Yauk can benefit you. Basically I will walk you through the whole process with some screenshots out of KNNS. Maybe you know KNNS, it's like the operating tool for Kubernetes, which is a nice UI. I don't want to make a live demo as I'm not that crazy, but I made everything a test cluster where I've already deployed Keystone, Glance, Neutron, and Nova with Yauk. And basically the Nova deployment then in the cluster looks something like this. In this deployment I stripped the most parts out of it and only like the essential parts for this demo. For example you don't see the section about databases, the section about the policy, the most config is stripped out and also like for example policies are not there. But what you can see is one for one the target release. We as Stefan already told you support multiple releases and for some services also already the automatically upgrading of the OpenStake releases. What you can also see are these two config sections. Like the green box would be a config that would be on all compute nodes. In the Yauk context a compute node would be defined by a label in Kubernetes. That would look something like yauk.cloud hypervisor true, which tells all the Yauk operators this is a hypervisor. And if you don't have a special node selector, this config would be rolled out to all of the nodes. Here I have just a few examples in there, like for example default availability zone or some other configs. And in the blue box you see a config that is selected via node selector and only will be applied to nodes where the hypervisor type QEMO label is set on. And there I just set some extra noble compute config regarding the virtualization type. If I now label a node with both of these labels, I would get this resulting config that just merged together and the next step would be extended by some Yauk defaults, OpenStake defaults and some like example database configs and so on and then implement it into the node. If I have a look at my cluster, currently there are five nodes but I will only have a look at the first one because this is my compute zero one and I want to deploy a compute node on this node. If you have a look at it now, it is empty, there are no pods from Yauk there and this is because I not yet labeled the node. In order to do that, I can just use Qubectl or edit this with KNS. For example, I will add these two labels like the hypervisor true to tell you Yauk it's a hypervisor and also the hypervisor type QEMO to get my additional config in there. In the next step, not the nova operator will go to work but the new operator because before you can install Nova compute, you need an L2 agent and this will be deployed by the new operator which also reacts on a label and basically what you can see here is just another custom resource this time not created by us but by Yauk itself. So also Yauk creates custom resource and then the next operator also creates custom resource to define our whole environment and basically what you can see here is first the name of the node where we want to deploy it then we have also some configuration for maybe the SSL encryption because all the internal communication in Yauk is already encrypted. We have some references to other services that Yauk can automatically configure them for example the keystone section in the config will be just taken from the already deployed keystone deployment and then we have some two agent specific config or OVN agent for example the southbound configuration and at the end we also have again the target release as we support multiple stack versions we have to pass that through to every service. If that is deployed we will see three new pods on our node of course if you want to deploy an OVN agent we need our vSwitch D and also our OVSTB and an OVN controller itself and also if you see in the first pod we have two containers in there so there's also the metadata agent in so we have like the whole bundle automatically deployed via Yauk and in the background there were already created all the keystone users that are needed as each service of course has its own keystone user the config objects and so on so now we have working two agent on the node in the next step after the OVN agent is already deployed the no more compute agent will go to work and we'll read this no more compute node custom resource which was deployed by the no more operator and basically it's quite analog to the OVN agent we saw before there you have also like the target name the release and so on and the keystone reference but we have also our config which was rendered like I showed you before based on the two labels and added to this definition which will in the end on the node look something like this so now we have a new pod there the no more compute pod basically in this pod we have three containers the first one is no more compute itself which just runs no more compute and we have a lip word container to have the lip word functionality and also an SSH container to enable the life migration if what is quite fancy is that we if we kill this pod the VMs would not be affected as the lip word process itself runs on the host namespace next step there may be some reasons why we want to change from configs or recreate a node and there we'll just show you how the life cycle would look like so some reasons for recreation would be like you want just to make a config change in nobody deployment which should be rolled out to your whole nobody deployment or just to one compute node or you relabel the node like you label maybe shows that the node is in your aggregate or just gets a new config or maybe we don't want to queue your config in there for some reason or you want to update your nodes or what always can happen if you're hypervisor just crashes in this case your group would also start an eviction process of the node to get all the customer workload removed and of course you can always manually delete the compute node which always would trigger this eviction workflow so if now something changes the operator detects the changes and marks this compute node for deletion and all the VMs on there would be started to be evicted and if they are gone all the that the notebook would be deleted and there will be an action as well and then the node is empty and would be can be redeployed with a new notebook compute node resource and everything starts from beginning but what is now this eviction of VMs for that we implemented a big job which would be run before the node would be deleted and in this big job I just brought in a log here we could see in the first in the beginning it will check if the node is actually still alive so we just ping it we look into Kubernetes look into OpenStack to ensure that the node is still there and not to like start an eviction of all the VMs if the nodes if we are nova if the nodes are still alive because if the node is still alive we can live migrate the VMs away so in next step the job will just look at all the VMs that running on the node and will classify them based on their state for example active VMs normally can be live migrated shutoff VMs have to be offline migrated and if they are for example in an error state they would be not handleable for you and we will just wait for something somebody to have a look at it so that we don't break it because for error status can be so many reasons that we don't want to build an automation for that and in the next step we just run our to run the evictions and migrations itself this is quite nice for a single node but the real benefit of it comes when you have like 500 nodes or more because then you only have a single point where you can configure your nodes and so on and configure your config and the versions and everything and your hook will take care of the automatic rolling upgrade of the nodes so it will take one hypervisor after another and actually you can also configure how many hypervisors from the same label set Mr. May from the same aggregate will be rolled out will be updated at the same time so for a small aggregate you can maybe one node at a time and for a big one you have like five or ten just to have this all in a nice and even fast manner so this is all about this compute noble compute example and I will get back to Stefan maybe to give you a short overview what is already in Yauk so we support the open stack versions green train and yoga yes queens will be excluded soonish we have already automatic open stack upgrades for keystone and glance from currently queens to yoga for other open stack servers not I remember right now ironic is maybe also in yeah okay thank you so much back ends currently supported our staff and net up and that's a bit tricky with the yoga release we switched to the OVN network I set up before we used neutron agents with OVS and we are also GitOps ready yeah so basically as in Yauk everything is a custom resource you can just place it in your repository and use tools like for example flux or argocd which just sync the state of your repository with your command list cluster which like enables you to just make one changes in your Git repo which we automatically rolled out to the cluster and then also rolled out by Yauk to all your workload Yauk is not or we are not done with developing it still some things that we want to include for example automatic upgrades to the newest releases for all open stack services we want to be faster to every type of visors because currently if a hypervisor dies we need to wait to keep a need to see the notice down and this can take five minutes or more that's not so nice we want to optimize the OVN setup as we use it pretty newly see all day that we could optimize something and then it's nice to put this changes also into Yauk and yeah maybe you found some or don't found some open stack servers that you would like to use on the list then either you could build it by yourself or open an issue for that that could be also included and with all this Yauk thing we wanted but it's an open sourcing and we wanted to have it really open sourced though we are from two companies but have some software owned by two companies it's a bit tricky from a legal size and also it doesn't feel so really open sourced so and we build it an association a non-profit association with other companies as well to okay I don't care now I got stuck yet for building silverware and digital infrastructures I'm stuck at this point currently but if you are more interested at four in the evening in room 15 my colleague will introduce this association a bit more maybe a short overview I guess you would see it there also you we have the educational part that's mostly about to tell others why open source open infrastructure is is important why software entity is important for them we want to tackle the community building for projects like Yauk and of course support not only Yauk but also other projects to become open source but more than in the presentation later like I said this whole thing is open source you can check it out on GitLab look through the documentation or contact us on the IRC at this point thanks for your attention do you have any questions thanks for the presentation my question is you mentioned the name of Yauk is yet another deployment tool can you elaborate on what why you decided to create something new compared to the existing initiatives in the community that were already there before and the other question is you want to expand the development beyond your own two companies do you have any plans maybe to move the development to opendev.org to host Yauk together with the rest of the OpenStack projects did you all get the question okay did you get to the first part maybe the second part together with OpenStack so we was in discussions with the OpenInfo Foundation when we thought about okay will we build an own association or not and at some point this got stuck somewhere I don't actually know what I guess it's the focus is a bit different so and we wanted to to have this project home somewhere so for now it's that but it's not now that it needs to be there I guess this can move later on if it feels right for both sides I'm regarding the first question we of course had a look at other deployment tools like OpenStack Herm or so on but what we missed the most part was like the second second day operations like keeping your cluster safe if you don't touch it like for the most part if you don't touch deployment it will break after some time and also like this what I showed you today with the live migration that you have this whole life cycle management for a compute node and so basically don't have to do some things manually we missed and other deployment tools we saw when we started mostly this operator stuff with the control loop at something at the point we started with this project we didn't found somewhere else in a way that fits our needs and also I had a look because at the same time like we started also also project started and was also presented at the summit but I think at the most I didn't found this approach which is totally fine it's the needs of other users so then they build it something else but like color ends a bit and so didn't give us this stuff it's nice to deploy an open stack deployment maybe it's even easier than yeah if you only want an initial open stack deployment but to change something it's hard for all the stuff how are you doing networking between Kubernetes and OpenStack and are you doing networking between Kubernetes and OpenStack and can you for example schedule a normal pod on a hypervisor Do you mean communication between VMs and workload on the Kubernetes cluster? Yeah so pods talking to VMs No that's currently out of scope because we would also say okay if a customer wants a Kubernetes cluster which would be then something they want and we would run a Kubernetes cluster on top of OpenStack for example in the project of the customer and then customer can put also a VM next to it which make it more easy Communication comes from the network fucking news so the Kubernetes part like talking and all of you talking to its database uses Kali code it's a bad end and everything customer facing uses OVM or OVS whatever you decide in the deployment and this runs in the host name space so the container utilizes basically every feature on the node itself and not running in the Kubernetes overlay so they basically these two flows never need each other so this has some implications so you need for example using OVS fire warning so you cannot rely on other tables because it's used by Kali code and there's conflicting stuff in there you lose if you you have around OVS in the same time as Kali code be extend you need to use different be extend ports so there are some some caveats for them but basically it's completely separate flow here yeah so so basically that the decision was okay we want to operate clouds with multi-tenants and they are connect somehow they're underlying Kubernetes to VMs wouldn't be so nice but that's what you can see but whatever port you want to have on hypervisor like to do monitor and start exporters we also run lvp the container because we don't want to install it on the node you can run everything you want but do you mind these resources are then subtracted from whatever you want to provide as a VM so you need to keep resources scheduled in mind because the Kubernetes resource schedule does not consider what opens that goes to you so you need to do it hard by C-groups or have a very good compute configurations further questions or questions regarding that you mentioned the very popular term sovereignty especially in Europe at these days can you elaborate a little bit on how you address this term here with Yauk I think it's a bit hard to put it there I mentioned it more on the cloud and heat side and so I won't go into detail about this company right now or my company right now Yauk is mostly focused on installing OpenStack and how you then configure it or what you do with it that's on the operator side I think our part for that with Yauk is that everyone can use Yauk to install their own OpenStack cluster the rest for my understanding but maybe we can discuss later about it I would say it's not really the focus but of course we don't want to block it so we don't want to block sovereignty with our but want to support it okay I don't see anything that we could do with Yauk to make it much better but maybe my understanding is different than yours okay I was just conclusively with that since we added it on your slides here it was only related to to the company and not really to this tool maybe further questions if not also time is over thanks for your attention if you have further questions you can reach out to us thank you


Okay, cool. So real quick, if anybody has a laptop out right now, you guys should copy that fingerprint down and sign it because we need signatures on the release key for secure drop. So, hooray bonus points if you can sign that by the end of the talk. So, yeah, I'm one of the maintainers of secure drop and this talk is going to be discussing how we improve secure drops security architecture. And the focus is going to be on explaining it in a way that will allow you to apply some of these other things other projects and not just, you know, laser focusing on the particular issues we face that maybe nobody else faces. So the very high level actually hands up if you know what secure drop is. Okay, a few people. Okay, so for those who don't the high level is secure drop is an open source whistleblower submission system that media organizations can use to securely accept documents from and communicate with anonymous sources. And this is something that we would stick in the newsroom of news organizations to allow them to keep the sources anonymous because do things like just not say the name of our source. We could say we don't know and just, you know, deny it until we get left alone. But, you know, we have the problem now of global mass surveillance where they'd like to say nothing is beyond our reach but and to some extent that is true they can watch the networks they can watch phone communications and whatnot. So, in order to restore the capability of news organizations journalists to be able to keep their sources anonymous we need technological solutions until we can, you know, get all those government organizations back in check. Right now it's deployed with a bunch of major news organizations. This is a handful. There's a whole bunch more, but that's not what this talk is about. So our threat model is explicitly groups like NSA, GHCQ and the rest of the five eyes. We assume that the kind of people who would want to be submitting things to secure drop are the kind of people who would be actively targeted by intelligence agencies or local governments and whatnot. So we have a very paranoid threat model we work with. The two main things we're trying to protect specifically number one is source anonymity. That's the number one thing that we would like to retain secret at the end of the day. And the second one which is very closely related is document confidentiality. So if somebody gets the documents, the source leaks they could use those to de-anonymize them even if the source is communicating over Tor. So these two things are really what we focus on during development. And the question again, who do I protect it from? Well, in reality it's actually everyone, but really nation stage corporations and local law enforcement. And we know that these kind of groups are actively targeting journalists. You can see press freedom violations around the world. There's actually a press freedom tracker that does tracks and these things. So we do know that this is happening all over all the time. And if we know those are the people who are the threat actors when we're developing this, we say, okay, well, what can these people do? Actually kind of can do a lot of stuff, but really intercepting network traffic is a huge one. If somebody can watch network traffic, they can see, you know, 800 megabytes of files move. And then there's 800 file megabyte files sitting on the server. We can kind of correlate that and say that's a lot of Tor traffic. We have a good idea who that probably was. They can maybe hack into the servers, maybe not, but it's a good chance that in a very long run, somebody may hack into one of the servers. And we also see things where they will send agents to physically seize hardware. And in this context, what we do not protect against is where somebody breaks into the office, sticks in a bad USB and malware. We're talking about an agent as part of like a subpoena picking up the server and just taking the server and running off with it. And this could include doing cold boot attacks where they, you know, drop the memory into something cold and then reboot it later and read the memory out. So we do care about the server's memory being pretty clean. And the last one that we've worked on a lot is people submitting malware to users of SecureDrop through SecureDrop itself. Because, of course, they're going to open the document on one of the workstations and kind of guaranteed phishing emails, you know, maybe not going to open it.  You're going to open it. Okay, so another thing with this complicated threat model is it is infinitely complex.  We could talk about any one little thing forever. And we're going to try to focus today on just a few of the bigger changes we made lately and some of the bigger screw ups. So the current state of things is we have one application server that runs all of our code that is mostly what SecureDrop is. And that is a GR security patch kernel to help minimize the types of attacks you can do using memory corruption. We run a Python application. We store our data in a very small SQLite database and we run all this behind Apache. And on top of the application server, we have a firewall that blocks all inbound traffic. So this server is only accessible through its outbound connections through Tor hidden services, including for administration and for sources and journalists using it. This again is monitored by a monitor server using OS sec to try to look for intrusions. And lastly, this is connected to sources and journalists both connected all this through Tor and run for the same way. And then a journalist will retrieve documents from the journalist interface and take them off to an air gap to laptop to decrypt them. So that's basically how it works. Our development is actually very simple. We write a feature. We write tests. We write unit tests. We do functional tests with Selenium. We write multi-stage tests to test deployment and provisioning with molecule. We write a bunch of documentation to help sources and admins and journalists and developers. We do mandatory code reviews for everything that gets merged by at least two members of teams. So the person submitting it and the person signing off on it. We automate testing. We do linting. This is all pretty normal stuff for development. So this isn't actually that special of a thing that SecureDrop does. But what's interesting is that at every step we actually put a lot of effort into all these things that we know that we can run end to end a complete submission set through the browser at every single pull request, every single check-in. And we do have to have some automated testing or manual testing because we're using a bunch of different USBs plugged into a bunch of physical hardware. But in the end, this isn't that interesting of a development process. But what we're trying to do is keep each of those things as high functioning of a level as possible during the whole development. That said, it's kind of like fighting little fires all the time. So as software developers, there's an update to something on the operating system and now you have to patch it because this library is no longer compatible with this library or the operating system becomes EOLed. Every couple of years you have to change your operating system. So that slide before about our development process, like keeping that so that it's very easy to go through the entire process each time really helps us with actually developing secure features because we're not getting bogged down saying, okay, we had to update one Python library and we spent nine hours debugging what broke. Now, we try to really say that library change should be quick. We just push off the CI and half an hour later we get a green light or a red light if that feature worked or not. So, you know, I forgot what I was going with this slide. There's something about like things break, but I don't remember where I was going to pretend the poop emoji is not there. But things do break and so since this talk is about improving the architecture, we're going to actually focus on the things that haven't gone well in hopes that this might help you guys in future projects not make some of the design decisions or mistakes that we've made in the past. So one of them was this is a tweet from Kevin Polson and over the summer or early fall, he got a submission to secure drop that when he untart it, it turned into a desktop file. Nautilus executed the desktop file. The desktop file ran arbitrary Python scrapes the GPG keys and then posted a signed message back to some server somewhere. And he posted this to Twitter, which is how we all found out about it. And we had to then go through and figure out what happened. And so really the root cause was that Nautilus allows an executable file with the, allows a file with the executable bit set to run any arbitrary code. And this allowed the attacker to run Python that scrapes the keys, put them in a document with a QR code and that QR code when somebody scanned it, ping the server with the data from the air gap. And so this is a very complicated attack, but this does happen. And so what really went wrong though is that the secure viewing station isn't a true air gap. We're allowing people to move USBs off of it and move data off of it. So we do know that at some point data will be leaving it. But we didn't tell journalists, by the way, be very careful about what you're taking off of it because that actually might be a trick. That's roughly equally as hard of a problem as telling people not to get phished. Like if anybody here is a security person, no. Well, your users are going to get phished all the time if you work for a big company and there's basically nothing you can do to realistically train every user to not be phished. Like it's inevitable. And so in that case, like training users not to scan certain QR codes, like it's just a losing battle. And so the issue we have with this is imperfect isolation. So we allowed arbitrary code to execute in the same environment that contained all of our GPG keys. And so the future development of this would be finding a way to sandbox the decryption code and the viewing of document code from where the actual keys are stored. And like this is something we've kind of known that that workstation doesn't have amazing isolation. It's pretty good and it's hard to get information off it, but stuff does happen. So like if this is the current workflow where you have like the source connects via Tor, the secure drop, the journalist connects to Tor again or connects to secure drop via Tor and then moves these USB sticks around. It's a lot of manual work and it's a little bit frustrating to work with if you're constantly having to like move encrypted USBs back and forth between devices. So our future design decision that we think will decrease like the manual processes that journalists and admins could make mistakes on and also increase the isolation of our system is moving to cubes to have the decryption and the document viewing and the keys all stored in separate systems so that we're not running the risk of random malware files executing in the same VM as the actual decryption keys. So this was brought up earlier in another presentation but loosely like this would be sort of the workflow so that when the journalist connects they would connect through Hoonix to have like their nice Tor circuit to get out to the secure drop application server, pull the stuff down and then each document would be opened in a disposable VM. We could then clean the documents and say okay this PDF has really just been rendered to a whole bunch of screenshots which means we can go forward and hand these PDFs out to the entire rest of the company and know that we didn't just either like malware the whole company or pull information off of the air gap. So this is we were trying to keep things clean so that when documents do leave the secure drop area of a news organization we're not giving infected files to the entire trusted internal network of the company. So another thing that went a little bit wrong is Tor did it made an update and this thing that went wrong was not Tor's fault, it was our fault. We were running a custom app armor set of rules against Tor, the Tor binaries and Tor files like files and when they shipped this it broke the app armor rules and it knocked every single secure drop off of the network simultaneously because they all updated to new Tor and we got kicked off. So update app armor rules and push those out to everybody to fix this and so that's one of the things is that we have a lot of dependencies and every system has a lot of dependencies and the more you can control your dependencies the less likely something will move under your feet and break part of your system and what we've done to fix this actually is we no longer use upstream Tor directly we actually mirror the entire Tor repository to a local from the press foundation control secure app repository and we point our secure drop servers to this new Tor mirror and actually Connor in the back over there is the one who I believe did all this Am I right? Yeah? Yeah. So Connor implemented this and this pull request again that lets us now have a little more control of our system so that external factors like our external dependencies changing still have to go through QA so that we know that we can go end to end on the system we can provision systems we know that the entire process still works before we push this out to some 60 news organizations around the world. The next major usability thing for security that we did is like a UX improvement which was we translated secure drop into a whole bunch of different languages and right now this is the handful there's a lot more done and this is up on the web and we're actively working on transiting more and more of secure drop and this is actually might not sound like a secure architecture thing but from the sense that like the system as a whole does rely a lot on administrators and journalists and sources doing correct behavior at every step along the way like this helps sources you know if we tell them memorize your password don't share your password like only connect with Tor browser if they're doing that in a language they don't really know which was only English for the longest time it increases the chances that they're going to make a mistake and in a sense because we are the stewards of this project it's on us to ensure that their behavior like is nudge along in the correct direction by a good user experience so like people like to talk about like the coolest encryption algorithms they use like for us if we say like if we tell them you know do X and they can't do that one X thing they immediately anonymize themselves that's like that's not any good if we use the sweetest latest crypto so making these good UX changes also does impact the design and turn of the program like there's little small things like every time we have a string we have to rethink about how we actually assemble these strings because you know you can't just drop random nouns the end of a sentence because in some other languages you know you have like inflection and declination and what not and so like this is only a small example but like going through the code there were a whole bunch of places where we did things like pluralize things by string concatenation by adding plus S to the end and it doesn't really work when you're internationalizing so on top of that there was a whole bunch of changes we had to make to the development process and one of them was developing a workflow that allows us to like submit all these strings out to somewhere where we can then solicit translations from the community building these back in the application and then also knowing that any feature that requires a string change we have to go through a longer process or have a like a longer timetable because we do require outside contributors to do these especially as the number of languages grow so you know making a quick patch into a release to quickly get something out sometimes can't happen anymore so we try to think about how we're going to develop in a way that gets these fixes in and allows people to ship things out without you know constantly saying oh we broke a string, we broke a string, we broke a string and then you know bothering the community another thing on the UX side is this is the old landing page and it's a little hard to read at the top in this font but it says we recommend using Tor browser access secure drop learn how to install it or ignore this warning to continue and this warning is flashed via CSS if the or no view the user agent in the HTML when somebody like hits the page for the first time and so we did some user testing and I say we as in the engineers which is to say we are probably not qualified to be doing user testing and they said oh that sounds scary like I did something wrong I'm just fizzing this for the first time so after a little bit of talking to people we switch this to being purple say okay purple is a less scary color now when users see this they won't think they did something wrong and then the downside was immediately I'm going to ignore this because it's just part of the design it looks like a purple banner and nobody listened to this anymore which again says we're engineers maybe we shouldn't be doing this sort of thing we should go out and we should find people experts in the field of user experience and user interfaces and have them really do an audit of the app say okay us as engineers we think we had this nice secure pattern but do we? what are the hidden things that are going wrong maybe this text looks pretty normal to us but to the average user it turns them away it encourages them to do bad behavior so again secure architecture might mean a lot of things but it also does include the user experience side of this another related architecture thing was we had at least one dedicated admin who would again get emails every time there was an OS sec alert and they were getting fatigued by this we don't know exactly how they were but we know that it was a complaint that there's too many emails and if somebody's getting an email every 30 minutes that says like oh this happened and none of it is actionable from a security perspective we're going to wear them down so that the day that there actually is an intrusion they just delete it with their other things or they set up a filter so that there's one inbox with 10,000 unread OS sec messages they're going to miss it and so we had to go through and really work on our rules for OS sec to say okay cool this is not necessary to alert we should try to avoid this and we're trying to shrink this down as a team so that people who are doing the day-to-day operations of secure drop aren't being overburdened and then being incapacitated in their ability to be effective admins so those are some things we've worked on in just like the past I don't know eight months or so and still working on some of these things right now but one of the big changes of the future is moving away from SQLite to Postgres so this kind of falls under the principle of least privilege we have applications that are when they interact with SQLite they can just do anything to it that means the source interface can read journalist, passwords, and user names which is there's no reason to do that and that's the limitation of SQLite right now so we have to we're going to move to a bigger database that has better access controls and it would also be interesting in the future because if we do want to do something like three application servers that all sit behind onion balance they would need to share a database and you don't want to do that on a file system so you didn't want Postgres available via PCP sitting behind the firewall and then you can have many application servers interacting with one database server oh that slide is not done anyways, so two of the major things we've done recently were we also had to get to the point where we could even add Postgres we didn't first had to add Alimbic which is the Python library and command line tool for applying database migrations and before we could do that we also depended on Flask SQL Alchemy which was not implemented in the original Flask application we also have to do a pretty large refactor of our tests and before we could even do either of those two things we'd also do a large refactor of the source and journalist applications and so even to make this one change of preventing the source application from being able to read the data for maybe like the password hashes and usernames of journalists like it's in many steps to get away from this and so it's not actually terribly interesting of a task like this is what I do day to day and it's actually quite boring but like that's a really important thing with secure software is there's just like tons and tons and tons of tedious work you need to do every step along the way a well-formatted application that's well modular that allows you to quickly add tests and verify that the behavior is correct so this isn't a fun presentation slide there's crappy work you have to do to make secure software sometimes and that's kind of part of being an engineer like talking to Connor last night about this and his analogy was that a big old skyscraper is really cool it's pouring concrete and pouring concrete is a pretty boring task so like us as software developers need to do more concrete pouring another hopefully improvement is right now we have started to dockerize the development environment to make it a little bit easier for everybody to set things up which isn't strictly a security fix at the moment but we're potentially moving from the current app server design which is just everything piled onto the application server and running all in the same user space into this is also up for debate so anybody on the team who doesn't like this idea you know you can yell at me later but this is something we're considering doing which is splitting off the different parts of the applications into their own containers and then restricting who can talk to what so the source application doesn't actually need to talk to the internet ever it needs to listen on or make itself available to nginx or the source application also doesn't have any reason to need to talk to even be able to like hit the IP addresses of the Postgres database and so by kind of removing things we get the benefit of separation of responsibilities but also we know that in the containers we can be a little more creative with the dependencies that we need across the different ones and not worry about system dependency collisions or things pinning versions and holding us back from updates we need because of dependencies and it's not a huge issue but it is a consideration that we can then control a little bit more how each of the different pieces fit together and on top of those few advancements that we're looking at which are not massive but somewhat big one thing that came up this summer or fall was there was a research paper about fingerprinting onion services which was looking at if you sit between the user and the guard node or you are a bad guard node and you watch tour traffic go in and you say okay cool I can see an encrypted stream coming in can I guess what onion service that is and can I guess what they were doing and it turns out that yes you can in a lot of cases use because there's not that many onion services out there in the world and there's not that many that get a lot of traffic so if you're trying to make a guess who's using time's up what okay anyways this is something that is an interesting research question but we don't have enough bandwidth to be able to effectively deploy mitigations against this because we have to write mitigations then test them and verify that you can't fingerprint secure drop as an onion service after these have been applied so if anybody is a machine learning expert and wants to help us or if any of these people in the audience you know come talk to us afterwards and we'll have a chat I guess maybe time for an honor to your questions cool yeah this is picture of this if you want to come join us and yeah yeah so when for a long time was kind of oh yeah so the question was do we see different threats across different languages and across different regions and for a long time one of the things on secure drop was that it was assumed that people would know like the NSA would know that some news organization has secure drop running in their office at this location and we were relying on the US legal system to protect the servers but running say secure drop in a country with a hostile government might be different so like in that sense like using Tor becomes more risky like then you want maybe your onion services to actually be more anonymous and the threat model does change as you move from around different news organizations around the world okay do we have what time is it do we have time for more questions maybe one more question anybody else can you show again the GPG yeah so oh yeah that's actually going so this is the GPG key at the beginning if anybody could take a picture oh my god so many animations cool this one if you guys could sign that one that would help us ensure the integrity of this admins use that when they install secure drop so if you want to help admins correctly verify that they have the real authentic secure drop software signing this would help us cool one more question yeah I just wanted to ask how many languages have you got your documentation in at the moment the documentation is only in English because it's pretty extensive and we don't have a plan at the moment to implement that we have only implemented the documentation on the source and journalist interfaces the translation yes yeah so I you can talk to us after that that one's a complicated question because we don't know this internationalization thing which is actually the work of Loic over here who's did pretty much all of but she asked about the documentation yeah yeah cool do we have more time maybe who is the next presenter here next presenter is not here maybe more questions cool in the grey pardon my English I just wanted to know if you have plan to integrate like provider to avoid stenographic attack like when the source is publishing pdf document maybe the hidden fingerprint in it and yeah so I think another some things like tails comes with the metadata which will strip out some metadata but if you do something like you have like printer dots or there's like some way that images are suddenly manipulated when they download them from a server like a very small watermark on them like we don't know about that so that would fall more under the training of journalists about how to redact information or release documents selectively and how to assess that themselves it kind of falls a little bit outside the scope of the secure drop software but it is relevant and I don't have the best answer for that one is the next presenter here for the next talk okay yeah just rotate cool thanks everyone


Hi guys, welcome back to my YouTube channel. This is Daniel Rosal here. I want to do a video just talking about this new diet I'm embarking upon. I've never done a diet or really anything like this before, so it's really uncharted territory for me. And the reason I'm doing this is not because I'm doing a diet, you know, who cares everyone's going on diets for, for weight loss at the moment. It's because it's a kind of specific diet for post-cholestectomy or post-gallbladder removal and functional dyspepsia and bloating and all the issues I've been having. Now, this is very much, you know, it's my perspective as somebody going through these issues. And I want to do this because if it really works, I'll be able to do another video in a few months and say, I followed all these recommendations and this is the difference that it made. So context, I'm about two years out from my gallbladder surgery. And as it seems to be the case, there is a subset of people who just get on from the surgery and everything works just fine in their body. And then there's people who everything kind of goes the other way and seems to break down and stop working. And I've been kind of somewhere in the middle, I guess, having difficulties, thankfully, you know, not being housebound. I've read some really tragic stories about people that just, you know, we're really destroyed by the surgery. So I haven't had that, but I have had basically the functional dyspepsia, which is a normal person, language, it's fullness, bloating, burping, all these, I think what doctors called upper GI symptoms. So it's not like down here in your colon, which is more like IBS, it's just all like stuff coming up. And it's very, very unpleasant and be going on for two years. So that interview with Dr. Tally really gave me encouragement to see the, see the dietician. I had this appointment set already and just give it a good shot because he said he sends every single patient of his out to see a dietician. Now again, this is just me talking here as a FD patient, someone with problems after the surgery and not as medical or nutritional advice. So I just want to say basically what I'm moving to. I also just want to talk about one supplement that a lot of people after their gallbladder removal do take and which I find kind of helpful just to give a full picture of what I'm doing now. So the diet, my diet, it's over here. I'm just looking at my screen here. Basically, it's a low fat diet. And the reason what my dietician told me is, you know, it makes a ton of sense, the gallbladder is an organ that stores bile, concentrates bile, and then puts that bile, injects that bile into the small intestine where its job is to basically one of its jobs is to help break down fat.  So when that's missing from your system and you maintain eating the same amount of fat, which is essentially what I've done, not advised, you're going to be having problems. And the problem I've been having, which I've heard a lot of people having is weight gain. I've had a really, really hard time maintaining a weight and it's been perplexing. I've been exercising a bit. I've been, you know, meant to eating a pretty normal diet and the pounds were just like coming on. So I'm hopeful that this diet will simultaneously improve all these digestive problems and it will also help maintain a healthy weight because I've kind of got back down to my previous gene size, which is amazing. And I don't want to go back in the other direction. So I'm trying to maintain at this level. So the diet, it's a low fat diet for that reason, trying to minimize that consumption of fat. And it's basically, here's what it is, it's grains, it's legumes, so it's chickpeas and lentils. It is, what else am I eating?  Now this is something that nutritionist or dietitian, I should say, was hesitant about because lactose intolerance and gluten intolerance, I'm personally trying to keep as many things in my diet as I can have and not cut out for no reason other food stuff. So I don't know that I'm lactose intolerant. I've no reason to think that I am. So I kind of insisted actually that I keep the option to eat this.  And I must say everything's been surprisingly good and tasty. So that's been a good snack for me. So beside the grains, there's eggs in my diet. There are lean meats.  So there are fatty fish. And there is less fatty fish.  Now that's kind of it's a tough one for me. And I suspect the reason I've had a lot of problems post surgery is because I'm a big fan of stuff like olives.  And these are all things that unfortunately are kind of out from the perspective of this diet. The only other thing I'll say about this diet is it's really interesting how far low fat diets have kind of fallen out of favor.   They were really popular in our parents day. And now we realize how good some fats are and don't cut out fat. So I don't know if that information is valid as a non nutritionist. But there are people with medical conditions like you don't have a gallbladder or your pancreas is creating issues that really have to stick to low fat. It's not about maintaining or maximizing health. It's about sort of dealing with a different digestive situation. So that's basically what the what the diet I've been recommended to take is porridge is what's going to be my breakfast now. Lunch is going to be this Indian curry. So I'm going to be making my own curries again. And this the only difference will be that I'll be measuring very carefully the fat, a lot of dals, a lot of, you know, Indian Indian veggie curries. And then for dinner, either more curry, or there's going to be basically those leaner meats. And that's going to be my diet for snacks. It's going to be a bit grim. There's going to be rice crackers and baked apples slices. I buy myself a air fryer for the first time. Well, obviously for the first time in my life. And just trying to make this easy for myself, make this even maybe fun, trying to see the positive aspect of this as in not just saying, Okay, I've lost my gallbladder. I can't get it back. I've now got all these digestive problems I didn't have before the surgery. I'm now bloated all the time. All these things are true and all these things are kind of sucky. But I'm trying to see it from the perspective of this is like this can be my catalyst to eat really healthy to live really healthy to just maintain a much healthier lifestyle. So that's kind of my motivator and just the mind frame shift I'm trying to do at the moment. So that's what I'm doing. I did want to mention this supplement because the other thing I've just added is a weapon to to try to try make things a bit better for myself. So this supplement is called bile acid factors. It's by a company called Jarrow formulas. And if the low fat approach is the preventive approach, this is kind of coming at it from the other direction of because you don't have as much bile after the surgery, right? So the gallbladder takes in the bile that the liver makes and then it really distills it down to concentrated bile and then puts that into your small intestine. And when you don't have that, the bile just kind of trickles slowly throughout the day as the liver makes it. So what this guy does is it's an animal derived product. It's bovine sourced. You'll also see ox bile supplements. You'll also see something called Todka. What these things are trying to do is just give you artificial bile and the way you're supposed to use this supplement is to take one pill at the start of the meal so that it'll mix in with the food and just give you that kind of missing bile that you don't have. So the way I'm approaching it and I if I can say this is I think quite a good approach.  I'm going to have my falafel. I'm going to have my hamburger. I'm going to have that pizza. In other words, I'm going to have a fatty meal. I need a break from the low fat and I'm taking one of these now only one. You do not want to take too many of these bile acid pills because let's just say you will know all about it for the next few hours in many, many different ways and you also don't want to take these after the meal because you want to take them at the start.  I've experienced this so you kind of have to keep your head on with these supplements and I am keeping them on a little kind of key holder thing that I bring with me so that if I'm out, I'll be able to take one on the go. So that's what I'm doing. That's what I'm trying. My next appointment with the dietitian is in about the end of December. So October, November, December, three months to experiment with this. I'll give you guys an update then and they really hope that for anyone else dealing with all these sort of digestive problems after colisysctomy, their gallbladder surgery that these things help so that there can be a bit of, I can provide a little bit of encouragement that there are certain things that will make a difference. Thanks for watching. Feel free to subscribe for more videos on this YouTube channel.