Hello everybody. Hi. My name is Frank and I am with Hope One Source and we're a nonprofit organization that is doing one thing and with that I want to ask you all one question. Who in this room right now knows that Drupal is being used to help and homelessness and save lives? All right. Well, that's news for everyone in the room. That's awesome. That's awesome. So take Angie, for example, a single mother of two who was living in her car under a bridge. She signed up to Hope One Source and Open Source Drupal-powered completely volunteer built text messaging platform. And as soon as she signed up she started receiving text alerts about emergency shelter and food for her children as well as later on childcare and a job. So that is beautiful because right now what I want to offer you is being volunteer based. I'm going to be handing out two tablets and if at any point you would like to get involved it can be an hour during a lazy Sunday or just one line of code. You can help the many Angie's in the community be able to get themselves out of homelessness and of course save their lives. So thank you very much and I'm going to be passing out the tablet if you folks are interested. Thank you. Thank you so much. Hi everyone. Thanks for joining me this afternoon. My name is Lauren Mifeo and I'm an associate principal analyst with GetApp which is Trip Advisor for B2B software for small and mid-sized businesses. So the idea with GetApp is that if you're a small or mid-sized business owner and you're looking for any type of software for your business, whether it's HR or CMS software, you'd go to GetApp.com, read LinkedIn verified reviews of products, filter for the features you need, and then choose the tool that's the best fit for your business. At GetApp I cover cloud business intelligence software for small and mid-sized businesses and I also research the ways that new technologies like AI and blockchain are integrated into small and mid-sized businesses. I've also contributed to opensource.com along with speaking at several open source conferences like ATO and Drupal GovCon and so I'm really happy to be with you today especially because I see some familiar faces from GovCon last summer. These are a few other places you can find me online. My link to my GitHub account is there along with my Twitter account and finally an interview that I did with the Association for Computing Machinery on the topic that I'm going to talk about with you today. I'm also giving this talk at the local ACM DC meetup on May 2nd so if you're in the city I'd love to see you there if you want to see this again. On the right is a picture of me running my first marathon in 2016 and to start off this presentation I want to take us back to 2015. Who remembers a website called HowOld.net? A couple people. So it was a website that allowed you to upload photos of yourself and the idea was that HowOld.net would use facial recognition software to guess how old you were. I had some fun with it last summer and uploaded a photo of myself taken that same month and I was flattered when HowOld.net seemed to think that I was 24 years old because that's a fair bit younger than I actually am. I was less flattered when I uploaded a second photo taken just a few months prior and the website thought that I was 38 years old so I aged 14 years going back three months and this HowOld.net made the news pretty widely as an example of how facial recognition software powered by machine learning meant well but still had a long way to go towards maturity. The thing is machine learning gaps aren't always funny. They can actually have serious consequences for end users of products that are built on ML data sets and one of the most infamous examples of this is a product called Compass. Compass is an ML algorithm that predicts defendants likelihoods of recidivism but research from ProPublica which is a nonprofit journalism outlet found that Compass has made biased predictions based on race. The algorithm was two times more likely to predict that black defendants were high risk for recommitting crimes and it was also two times more likely to predict that white defendants were low risk for recommitting crimes but there was a big problem with both of these predictions. They were incorrect and this isn't a hypothetical scenario. Compass has been used by judges in over a dozen US states to confirm prison sentences including the lengths of those sentences and whether defendants were released on parole or not and based partially on Compass's recommendation a Wisconsin judge denied probation to a man named Eric Loomis. Instead that judge gave Loomis a six year prison sentence for driving a car that had been used in a recent shooting but when Loomis tried to take his case to the Supreme Court the justices refused to give it a hearing and this was problematic because their choice not to hear his case signified that most justices condoned the algorithm's use without understanding how it reached conclusions that were often incorrect and this sets a dangerous legal precedent especially because we don't have a lot of legislation or laws around AI and confusion about AI shows no signs of slowing down. The question is did someone do this on purpose? Probably not. It's more likely that this was an unconscious mistake. The Compass algorithm probably didn't get enough training to recognize diverse types of facial variables including skin tone. It's also possible that the algorithm drew from historical data about rates of arrests between people who have black skin versus people who have white skin and it made an incorrect correlation of skin tone with recidivism. We don't know for sure because Compass is what's known as a black box algorithm and that means that we can't know exactly how it makes decisions or what went into the data but we can infer based on its flawed results that machines have their own biases just like people do and that's really what machine biases. It's programming that assumes the prejudice of its creators or its data whether that bias is intentional or not. This is a problem and it's a big one but we'll start to solve it by answering four questions today. I'll go into some more detail about what machine bias is. I'll move on to explaining why it's dangerous for end users. We'll go into the root cause of machine bias and finally I'll give you six concrete steps you can take to add bias testing to your product dev life cycles. Let's start by asking again what machine bias is. We just went over it at a low level but it's worth unpacking that a bit more because we all have bias whether we know it or not. I know less bias than you are just because I'm standing up here. By virtue of being human I have my own bias and some say that because that bias is what makes us human it's why we should outsource big decisions to machines. We all know the stories about how you shouldn't get surgery on a Friday or how you don't want to have your case heard in the late afternoon when judges want to go home. Machines don't get tired, cranky or irrational and so the thought process says that outsourcing big decisions to them will yield more objective results. But as we already learned that's not the case. Machines have their own biases just like humans do and this will continue the more lifelike these machines become and this bias manifests itself in two distinct ways. The first is through direct bias. This is when models make predictions based on sensitive or prohibited attributes so things like race, religion, gender and sexual orientation all constitute examples of direct bias. But believe it or not that's actually not the form of bias that you have to be most concerned about. The bigger issue is indirect bias because it's harder to detect. This is a byproduct of nonsensitive attributes that correlate with sensitive attributes and this is why the compass algorithms predictions correlated with race. The outcomes for one group are compared with another and if the difference exceeds in agreed upon threshold the model is considered unacceptably bias. So if the difference in incarceration rates varied more than some percentage between prisoners of different ethnicities you can make a reasonable inference that the algorithm is biased. Machine bias is also dependent on two important factors. The priority set by the algorithms designers and which methods of fairness were accounted for. If one or both of these is absent at the start of the product life cycle your risk of bias creeping into the data set goes up dramatically and because these are often absent at the start that's where problems can creep in later. Machine bias is also based on measures of fairness and this defines how to treat all participants in a system with equal respect. There's a wide range of possible measures here and they often have negative correlations with each other. That's because as fairness increases by one measure it decreases by another and just like with any product management scenario this is going to lead to tradeoffs and competing priorities especially because sometimes as fairness increases accuracy of the model can decrease. So let's go back to the criminal justice example. In this case the algorithms designers have to make some tradeoffs at the start and decide which results are the most fair. They have to ask themselves should 10 innocent people go to jail to make sure one guilty person isn't released or should 10 guilty people be released in order to keep one innocent person out of jail. Regardless of which choice the designers make their products users will feel the impact of that decision. And that leads us to the next question. Why is machine bias dangerous? There are several answers but I'll give you three to start off with. The first is that machine bias reinforces human bias. Machines are like toddlers they only learn from the data in front of them and the data that they're given. So if machines are fed any data that reflects its creators unconscious biases the machines themselves will reflect that bias and this isn't a new problem. St. George's medical school in London used an algorithm in the 80s to take a first look at applicants. The idea was that they would outsource the initial review process to machines and then the machines would decide which students were worthy of coming on campus for interviews to med school. And then it was adjusted until the algorithm's decisions matched the characteristics that the admissions team wanted to see. But four years later two doctors realized that the algorithm was biased. They realized that it was rejecting female and non-European sounding names for interviews even when these applicants had all of the other qualifications. So they were qualified to be interviewed based on merit but they weren't being flagged through to the interview stage purely because of their names. And these doctors found that up to 60 applicants were being rejected each year just because of their names not because of their merit. The second reason that machine bias is dangerous is that it doesn't account for context. Algorithms learn from the data they're trained on and that means they don't understand nuances like regulation change. They also don't inherently know if their data is biased or homogenous. And so this increases the risk that they'll either reinforce early biases or that they'll learn new biases when they're deployed. So one example of this is the practice of redlining in Portland. From 1859 through 1990 people of color were banned from buying property in certain Portland neighborhoods. That practice is illegal now but we still have more than a century's worth of real estate data based on that practice. So as a result any predictive models trained on the data set are at risk of perpetuating bias of an illegal practice.  So again the impact of these decisions are very real for the people that are inadvertently using the products. The third most extreme example of machine bias being dangerous is that it can put users' lives at risk. Again this isn't something that's inherently new. We all put our trust into the designers and engineers of these systems but if those teams are too homogenous they won't build for diverse user needs. One example of this in the past is crash test dummies. In the 60s crash test dummies were modeled after the average male. The average male's height, weight, etc. And as a result female drivers today are 47% more likely to be injured in a car crash and female crash test dummies weren't brought to market until 2011. So that decision 50 years ago had ripple effects that still impact citizen safety today. Now consider what would happen if speech recognition software in an autonomous car can't recognize different types of voices because it was trained on a narrow set of speech patterns. That sounds extreme but again it's not hypothetical. There are multiple instances of voice user interfaces that don't understand different accents and inflections because they weren't trained on enough voice variables. And Dr. Carol Riley really said it best. If these systems don't recognize people of every race as human there will be serious safety implications for the users of these products. We can't solve the problem though without knowing what the root cause of it is. So why does bias exist in the first place especially if most of it is indirect and unintentional. A big cause of the issue is black box algorithms which is what compass is. It's hard for us to know if an algorithm's results are truly fair. And this is hard even for the algorithm's creators because machines often behave differently in deployment than they did in production because they're exposed to new data in deployment. And so the result is that in extreme cases none of us can see inside an algorithm to learn how it made decisions. That's what we know as the black box problem. It's the inability to discern exactly what machines are doing when they're teaching themselves novel skills through reinforcement learning. When we don't know how algorithms make decisions we can't ultimately trust them. And so in the near future I think companies are going to have no choice but to be more transparent about their results. We're already seeing legislation in Europe that would find large tech firms for not revealing how their algorithms work. Google put an alert out to investors that ethical concerns about their products could impact revenue. So tech companies know that this is something that users have their eyes on now. And then as extreme as regulation sounds it's actually what users want. Research from the University of Chicago and the University of Pennsylvania showed that users have more trust in modifiable algorithms than in those built by experts. And that's even when the modifiable algorithms are wrong. The point here isn't related to accuracy. It's more about seeing how the algorithms work and being able to make decisions. And if you have the expertise to contribute then you could theoretically fix the problem. And so this really supports the crucial role that open source plays in public trust of tech because as the research showed people prefer algorithms when they're modifiable and they can see how they work even if they turn out to be incorrect. With that said there are some valid reasons why you would want to keep your algorithms private. I led a boss session on this topic at open source summit North America last August and I talked with developers about some of the reasons you would want to do this. The obvious one is that these algorithms are considered proprietary and revealing how they work could yield a competitive an advantage for their competitors which they obviously don't want. Another reason is that they're built on endless neurons that each have their own biases and so picking apart every single neuron is in theory maybe possible but it's just not practical for them for most teams. And the third is that these systems are at risk of manipulation by bad actors. There was a program manager at Twitter in that boss session and he said that his team is on constant guard from people who want to manipulate the systems with their own mal intent and so for that reason it obviously makes sense to keep them private. But for transparency's sake it's still a good idea to open them where possible and here's where Drupal actually has a bit too big advantages. Large amounts of tagged structured content. The volume of data and tagging data are both really essential for data sets that you can use to train ML products that are less biased and I'll get into that a bit later in the presentation. Frank Kerry gave a presentation at Drupal Con Baltimore two years ago explaining why Drupal data is ideal to train deep learning and neural networks so if you haven't looked it up online yet I highly recommend that talk and these networks can make a big range of improvements to your site from generating alt tags automatically to image captioning so again the data that is generated through Drupal can be really valuable here. And luckily we're seeing more open source tool kits coming out these days to help address this problem. One of them is a python toolkit called Lime. It's based at the University of Washington and it doesn't try to dissect every single factor that influences an algorithm's decisions. Instead it treats every single model as a black box. So what it does is it uses a pick step method to select a representative set of predictions or conclusions that it wants to explain and then it approximates the model close to those predictions then Lime manipulates the inputs to the model and measures how predictions change. So here's an example of a Lime classifier from text classification. They took two classes Atheism and Christian that are tougher to distinguish since they share so many words. Lime's researchers trained a forest with 500 trees and got a test accuracy of 92.4 percent. So if accuracy was your core measure of trust you would be able to trust this classifier. And ultimately we're here because responsibility for algorithmic choices doesn't start with regulation. It starts with all of you. It starts with strong data development and product teams collaborating to make sure the parameters are as fair as they can be from the outset. And the good news is that while bias is inevitable it's not impossible to overcome. There are several steps you can start taking with your teams to boost the health of your data sets and mitigate bias from the start. The first step is to document your priorities up front and this requires you to answer two key questions. You want to look at what methods of fairness you're going to use and how you're going to prioritize them. This so sensitive attributes should be identified and declared out of bounds here unless you've made an explicit justification for including them. You are also going to want to include a minimum threshold of acceptance and acceptance functioning before you deploy your product. This is helpful criteria both for internal teams and for choosing external vendors. So if you are a business wanting to use an ML product you want to be making sure that their product teams are looking at the data and you want to get some insight into how they're making decisions. The second step is to train your data under fairness constraints. This is tough because when you try to control or eliminate both direct and indirect bias you'll find yourself in a catch 22. If you train exclusively on nonsensitive attributes you'll eliminate direct discrimination but you'll introduce or reinforce indirect bias. But if you train classifiers for each sensitive feature you'll reintroduce direct discrimination. Another challenge here is that detection can only occur after you train the model and when this occurs your recourse is to trap them or scrap the model and retrain it from scratch. So to alleviate that don't just measure average acceptance and rejection across sensitive groups. Instead you'll want to use limits to determine what is or isn't included in your model. This expresses discrimination tests as restrictions and limitations on the learning process. The third tip is that you want to appoint someone to monitor your data sets throughout the product's life cycle not just in production. That's because developers tend to build training sets based on data they hope their models will encounter in production but many don't monitor the data that their creations receive from the real world. And this is problematic when ML systems use reinforcement learning to constantly be updating their results and refining their results. Bias should be reduced as much as possible here while maintaining an acceptable level of accuracy as defined in your product spec. And it's also not uncommon for the algorithm to be updated without the model itself being reevaluated. So to alleviate that you're going to want to appoint someone to monitor the source history and context of data received in deployment and production. And if that person detects unacceptable bias or behavior the model should be rolled back to an earlier state prior to the first time you saw bias. The fourth step is to tag your training data. Tagging refers to which classes are present in an image and their locations. But doing this at scale and with accuracy is a huge challenge that's prohibitive for a lot of teams because the tagging the cost of it is proportional to the time spent on it. And so tagging sounds easy until you think about how much time it would involve to hand draw all of the homes in a data set for example. And tagged data sets often have their own biases which can lead to bias in the set. The good news is that more products are coming to market so that they can decrease the time and the cost of tagging. One of them is a SaaS product called Brain Builder. It's from a company called Nerala that's based in Boston and they use open source frameworks like TensorFlow to help users manage and annotate their training data. Brain Builder also aims to bring diverse class examples to data sets and this is another key step in the data training. You'll want to use diverse class examples in your algorithms because training data needs positive and negative examples of classes to learn what things are and what they're not. So consider the example of homes within a data set. If the algorithm contains only images of homes in North America it won't know to recognize homes in Japan, Morocco or anywhere else once it's in deployment. And its concept of a home is thus limited. So training data needs positive and negative examples and if you want specific classes of objects you need the negative examples. Because this more closely mimics the data that the algorithm will encounter in deployment. There are barriers to this mostly the fact that a lot of data sets used to train these algorithms are too small. Again the volume of data is really important for training models that are accurate and that's why we see relatively few players in the tech world dominating the AI space because they have enormous amounts of data that they can use to train their systems. The good news is that 2018 in particular saw an increase in the number of open source AI data sets. So Sync'd has a helpful list of 10 open source data sets you can use and if you're looking for public data sets by industry GitHub has an extensive list as well. I have URLs to both of these in the presentation so if you're interested in getting them afterwards please come talk to me and I'll email the presentation to you so you can get the links. The final tip I want to hit home is to focus on the subject not the context. So let's start by looking at the training set on the left. This is an example of a biased training set that's because the wolves in this set were tagged standing in snow but the model wasn't shown images of dogs. So when dogs were eventually introduced the model started tagging them as wolves because both animals were standing in snow. So classification errors like this mean that the model being trained was either incomplete or inaccurate and the person monitoring the data should be rolling back the model at this point. On the right we have a training set focused on the subject which is dogs. So again you want to look at how much weight the algorithm is giving to different aspects of the data and make sure that it's applying weight appropriately. I just threw a lot of information at you and I don't expect you all to necessarily remember it but if you leave here only remembering one thing I want it to be this. Ethical debt is technical debt. It's not just a nice thing to build data sets that are equal for everyone. Bias leads to poor data which leads to poorly trained neural networks which leads to a bad user experience that in extreme cases can be deadly for end users. And on your end when bias is found your only recourse is to scrap the model and retrain it from scratch which is an enormous undertaking for you and your team and it creates a lot more work for you when you're already working on a very complicated technical problem. And I also want to apologize because the title of my talk is a little misleading I think. I promised that you could erase machine bias from your AI data sets. You really can't. Bias is inevitable for a lot of the reasons we discussed and the more complicated these data sets become the more likely it is to be introduced. What I do want to hit home is that while machine bias is unavoidable it's also manageable. Like every problem with product management you're going to have competing trade-offs and priorities you're going to be working with cross-functional teams that all have their own goals but that's why you have to plan ahead for bias from the start. If you design products at the beginning with the end in mind you'll know how to spot problems and you'll be able to refine those issues over time before they get to the end user and everyone will thank you as a result of that. I have some resources if you want to learn more about this topic. If you want to get specific about how to use TensorFlow for Drupal there's a great article on medium about that. My colleagues at GetApp and Gartner which is a Gartner company have also written about machine bias. We also if you want to sign up for a free trial of Brain Builder using your own data to build models you can do that here and if you want to learn more about how ProPublica analyzed the compass algorithm I talked about a link to that is here and again if you want any of these URLs feel free to come talk to me afterwards and I'll email the presentation to you. Thank you. I think we have a few minutes for questions if anyone has one. Yeah the question is at the beginning you said that some of these data sets and algorithms should be proprietary because a bad actor could get in there etc but how does that work are you telling us that the company with their proprietary thing is ethical and we don't even need to look at those things because you mentioned there without that there would be ongoing inspection and ongoing which I think there should be. So the question was around whether these systems should be proprietary or not or whether proprietary rationale should be a case for why these systems should stay private. I don't think that's the case I was actually talking about the reasons developers give for keeping them private so that was a point that was made when I led a boff session on this topic a few months ago I was talking to developers about the reasons why you would want to keep this data private from their perspective and they brought up the proprietary nature so that's a rationale for keeping them private but for the reasons outlined I think anytime you can make them open source for all of the reasons described that's a better outcome for everyone because people can see how it's making decisions and if it's an open source project theoretically someone could come in and solve an issue if they see it. Exactly because we seem to have a problem schools are requiring people to get like iPads and stuff and you know we wrote to the newspaper about it in an open ed thing and one of the responses was very scary from the parent who said my kid does not need to know what's going on inside the computer apple knows what he needs. Right no I and that's a that's a huge issue and again I think that's why the supreme court case is so problematic or rather I should say the lack of a supreme court case because the justices are who are the most powerful lawyers in the country were condoning an algorithm that not only makes incorrect decisions but they couldn't even see how it was making those decisions and so that sets a very dangerous legal precedent I think when we can't see how those systems work they do involve unintended consequences in that I wish I could give this presentation to that parent as usual that as usual the people who need to hear it most probably aren't in the room but you can try anything else okay thanks for coming and enjoy the rest of the conference


Thanks for coming out late Thursday, guys. It's been a long couple days. I'm Nick Merwin, the founder of coveralls.io. And I hope you guys have seen some of our badges around town. GitHub, Readme, et cetera. I don't know. We're kind of all over the place, hopefully, right now. But today, we're going to discuss how it went about figuring out how to get Coveralls, a SaaS app, into the hands of customers that wanted to run it within their corporate networks. It was pretty untrited territory for me, but I hope this general overview and what we'll see later with just a really basic prototype will help you guys get some confidence that you could also take your app to the next level by offering a hosted version to a customer base that maybe you never thought was possible. So in that sense, it's kind of going to be partially a business model discussion, and some just general sys adminning stuff, and then, of course, all kind of viewed through the lens of Rails since Coveralls is a Rails app. All right, so why would you want to set your app free into the dark world outside of your cushy deployment environment? So currently, let's say, your app is hosted on Heroku or VPS, Coveralls is on DigitalOcean, and if you have a subscription-based service where users are paying by the month, and maybe there's a few tiers and usage-based add-ons, and it's humming along the crewing users, and it seems like your potential customer base is covered, but you might be neglecting an even more lucrative and even preferable customer base, which could be the enterprise, quote-unquote, and this could be monolithic companies where dev teams have a much harder time getting upper management to embrace cloud-based tools, or perhaps the issue is just that setting up a $5 month service fee takes a mountain of vendor approval paperwork. So let's chat about how it can make sense to offer a hosted version of your app in addition to your cloud service. So there are three kind of tenants that I ran across in figuring this out, or as to why you would want to do this. And the first one is security. And it's pretty obvious that we all worry that when you use a cloud app, you essentially give up control of your data, and you assume that the devs on the other side hopefully use Bcrypt on your password, but you never really know for sure and just hope there's never a real data breach. But beyond passwords, it could be any sort of data that your company's security department has forbidden from leaving their networks, and in fact, it could preclude the possibility of them ever using your app. This could be maybe a medical company that wants to use your data mining tool that can't upload anything to the cloud due to HIPAA compliance issues, or maybe it's a dev team that wants to use your source code analysis tools but can't risk sharing or leaking proprietary code. There's also just general business apps, like CRMs, internal tasking, chatting, scheduling that would have plenty of info that those companies wouldn't want to share with the competitor. But by running a hosted version of your app within their networks, they can just be more confident that they won't ever have their accounts hacked or their data compromised out on the open internet. So the second one is a bit more esoteric, it's reliability, and this is just kind of a worst case thing. If your app that you're giving them is a DevOps tool that's part of the build or deployment pipeline, then they're not going to want there to be any unplanned maintenance downtime. So for instance, with GitHub or Travis, if something goes down, something goes wrong, you don't have control of it, and it blocks your pushes or your builds that could screw up your deployment. And so if you're running it internally, then your dev teams and your sys admins will always be able to coordinate to make sure everything's good to go when it comes to crunch time. So lastly is cost, and the traditional subscription model where it's just pay by the month doesn't really work when you've given over your app to a customer. So oftentimes those potential customers are also less price sensitive and could be willing to pay a premium for your host's service and some sort of extra white glove support service. But once you deliver the app to them, you have to assume that it won't be able to phone home to check the subscription status. So instead of a time-based subscription, like a monthly fee, you probably are going to need to do some sort of seat-based usage or something else that's beyond just the monthly or perhaps yearly. And then once you're thinking about seats, then you can do seat packs as tiers, as in the number of active users that can access the app as it's living within their network. You could also build some sort of self-destruct or shut down mechanism that will force the customer to purchase or renew the license after a set amount of time, like six months or a year, just to make sure that they come back and they keep a valid license. So there's a lot of different opportunity to work out business model kinks beyond just the traditional SAS model. So before we go into the how we figured this out, let me just mention coveralls and what it does and why we took it to the enterprise. So coveralls, really quick, coveralls is a tool for code cover tracking and notifications. We can annotate your pull requests with successor failure statuses based on changes in your coverage percentage, or just send messages to your chat or email with updates. This helps dev teams make sure they don't deliver on tested code, especially in a production. And on site, you can see line by line coverage of the code base, the cloud version for open source is free, but we charge for private repos. And the rub comes because while we don't actually store source on our servers, we do store private scoped OAuth tokens for GitHub. And so for some companies, that's just a non-starter. They can't allow us to see within their private repos. And then some of them are already using GitHub Enterprise, so they're not using Cloud GitHub to start out with, and the two are not interoperable. So what we did is gathered interest from a potential customer base over a matter of months. And it became clear that there was enough of an interest that we should try to do this and try to make it happen. And so that's what brought us here. So there are some pretty big main hurdles that I found in converting your general Cloud Rails app to a Fog one where it's living within, below the Cloud layer. So delivery and installation, let's talk about delivery first. The best user experience for your customers would be to have the least amount of work to get your app up and running in the environment, obviously. So three download files would be, I think, is the most consolidated way that we can achieve this. And that's first the virtual machine that the app is going to run on, and that could just be a Linux box, as I'll show you, a virtual Linux box. And that's typically between 800 and a gigabyte size download. The next one would be the packaged up app files itself. And that's a decently high number, sometimes between 50 and 100 or more, depending on how many gems you have, because all the gems need to be bundled and rendered into the app package itself, as we'll see. And then lastly, a license file that's specific to the customer and is generated for every customer. And so out of those three, your customers would probably only need to download the VM once in a blue moon when you do big upgrades to the underlying system or have new dependencies. The app package is something that they would download whenever there's updates. And we'll get to that in the license file would be around the same amount quicker, like when they go to trial to active or what. So actually, let's talk about those incremental updates for a sec. With the standard deployment, obviously, when your code is hosted up on Heroku, getting features and fixes out to your users is as easy as get push. Not so in this case when your app is running completely out somewhere in the wild where you have no access to it. So that's where the app packages come in. It's a smaller download that has just the most up-to-date bug fixes, features, et cetera, that is a much quicker download than having to re-download and configure an entire virtual machine. So also for the license files, one last thing, it makes sense that this is the part where you'll probably need to build a secondary app, in addition to your main Cloud one, that will just service those customers and charge them subscription fees and have trial accounts and all that stuff. So that's where they'll download the license file and then download the other two in the secondary app. All right. So installation-wise, the networking config is the first thing that they're going to see. And since they're not, you don't want them to have to log into the VM themselves, the best way to get around this is to build a tiny little menu-driven Ruby app that gets presented on boot. And I'll show you guys how we did this. And that way the customer won't need to know any of the specifics about Linux networking. In addition to that, external access just has to be assumed as not going to be a possibility. So hopefully if they're, the reason why they're purchasing this hosted version is that so they can be completely confident in the security behind it, they won't give it any external access. It'll be totally walled off in a sandbox in their network. Hopefully. So the assumption of there not being any way to access it from the outside is probably valid. Last one is just process management, and that's how are we going to get background jobs and the server itself to stay alive and start at boot. All right. Couple more of these hurdles. So product support, generally you can see immediately when a user hits a 500 error, you're going to get an email and you're going to be able to go on to your error tracking service, air brake, whatever, and take a look. In this case, there's no way to know when that happens. There's no way to get pinged. So you're just going to get an email from your customer saying something's not right, what's going on. So you need to provide them a way to be able to send you details about the exception. And so we're going to look at how to just keep those exceptions somewhere that we can then send over or let your customer send to you and attach to an email or something. So that goes the same for logging. If something's really funky on it, you're going to want to be able to see if there's any weird parameters coming through the routes, especially when they're not causing an exception, per se. Lastly, there's resource management. If it seems like things are running slowly on their site, there needs to be some way to address that. I'm not going to go over that in this talk because I feel like that's more of a sysadmining issue, but that's something for homework, I guess. So lastly is the intellectual property question. And this is a big one. And because you're giving them a VM, they can unpack the VM and mount the disk and look at everything you put there. So even though it comes in one nice file, it's still extractable, and you can just load up another VM, a virtual box VM, and then mount the disk that you had delivered them. So that means they're going to be able to look at everything as though they were a root user. So I feel like database access, unless I don't know, there might be some way to get around this, but you should just assume that they're going to have access to the database itself. And it doesn't really matter. I don't know. That's the bigger question here, along with code obfuscation. Since Ruby, there's no way to really, really fully protect your source code once it's being executed in somebody else's environment, and it has to actually work itself, you can write a code obfuscator as a deterrent, and that's what we've done for CoverAll's Enterprise, but I think, I mean that prevents people from immediately reading it if they just mount your disk and look through your source file tree, but it's probably safest to just cover this with your license and say something like, the license you're purchasing from us covers only the use of the software modification or redistribution or not permitted. So it just kind of comes down to legalese as being your last line of defense here. And if this keeps you up at night, then maybe the best thing to remember is that your customers for your app should be most interested in getting updates, getting bugfixes, new features, and support rather than breaking into your source code, reading it, copying it, spreading out on the internet. So that could be its own talk about how to actually achieve some level of code obfuscation, but we're not going to go over that really today. OK, so let's get into the nitty gritty here. So this is a pretty hectic diagram, but it's basically showing the general architecture of how the app is running within our VM, which is Ubuntu. And we chose Ubuntu because it's just widely used and relatively easy to configure. So the main components that live within it are the Network Config app, which is the first thing that customers will see when you boot it up. And that lets you do things like select static networking or DHCP and then set the name servers and reboot, shutdown, et cetera. So it's just kind of a simple little starter app. And then behind our web server, we'll go into why we chose Passenger for it. There's going to be a pre-installed app that I just made in Sinatra. It's a really simple app that facilitates the unpacking of the package file that contains the actual app, or the app updates, and the license file. And so within that, behind the web server, we'll live your Rails app also. So the Rails app itself, some of the things that these three components I think are the main ones that set it apart from your standard Cloud hosted app, or where you started from to begin with to get it to this point. So the license file reader is going to read and cache your license at the boot when your Rails app boots. And then it's checked on every page view. And then you can do things like enable the disable features, or lock the app down. Second one is data import and export. Because when you need to download a brand new upgraded virtual machine, say it's got new dependencies for new features, then you're going to need to give your customers a way to get their data out of the old one and get it into the new one. So that's things like dumping the database. If your app has uploads, then you're going to need to pull those together if uploads are living on the VM. And then anything else that the users might have uploaded or changed that has a state on the VM. The third one is support package generation and downloads. So we'll show how we can use the Rails rescue from in the most general terms just to collect all the exceptions that happen in controllers, at least. And they can be archived into a temp directory. And then when an admin for the app, as it's living on the VM, needs some support, they can hit a link that generates an archived file that we can encrypt also. And then email it or dropbox it over to us. OK, so setting up the environment first. For development, I think it's probably easiest to use VirtualBox. It's free and pretty simple. And it's really easy to export and appliance from your machine called an OVA file. And that's just a nice little, it's a tar or zip or something with a .OVA extension. That's just something easy you can link to and serve from your CDN. Also Ubuntu, just because of how widely it's used, hopefully we can trust its built-in security settings and standards. So we kind of get that out of the box and our customers get that confidence out of the box. So when you provision it for your Rails app, you obviously want to use the minimum amount of dependencies and touches to it so that you can just keep the download size smallest. If you don't need to install something like image magic, then just don't do it. There's no point. And it's pretty easy to end up with a two gigabyte or more virtual machine, whereas you probably could have kept it slimmed completely down to a one gigabyte. So also, when you're doing development on the virtual machine, it helps to have two separate versions of it. One that you're going to use just for packaging up your app because you're going to need to have the full app sitting somewhere on the virtual machine so that you can run a bundle to vendor the gems into the app itself to get ready to be completely packaged up. It's not something you should do in your local development environment. Say if you're developing on OS X and you vendor your gems in that environment, they're not going to run on Linux. Or once you're in Linux. So all right, next let's talk about networking configuration. That's just a little screenshot of what a user will be prompted with when they, or a customer, when they load up the virtual machine the first time or every time really. You can still SSH into it.  But this is just sitting within a file in Ubuntu called the TTY, the teletype.  And really you're just, to make this work the bare minimum is just having some system calls that write to the file, the interface file, or the DNS file. So there's also shut down reboot options. So here's some, I don't know if that, can you guys read that? It's pretty small. I have it on a text editor too. So yeah, like I was saying, it's a menu driven app. And it's just basically collecting information from system calls, and then displaying it, and then letting you input things just using get s as a very simple Ruby app. And then letting you just reboot the entire computer. So it's pretty standard, simple, no rocket science here. So the next part of the puzzle is the server itself. And I chose passenger for it because it just seemed the most dead simple to put your app in a directory and have passenger just start serving it. Once you hit touch temporary start, then passenger will load the new code, and it doesn't really care if the code isn't there to begin with. It's not going to completely blow up. It's just going to 404. So that means the loader app that we'll talk about on the next slide can easily just extract the packaged up Rails app into a directory that's already been preconfigured for passenger to serve the app from. And as you can see, sort of online 14, that's where we're running the loader app. So as soon as you boot up the VM for the first time, those error pages will actually, if the Rails app is not found, it will redirect you to the setup page. The setup page is where you can upload your package file and your license file, and it'll be extracted into its eventual living place for good. So the installer app itself is just a simple Sinatra app that takes the package and the license file, puts them in the place where they'll live. And this one, you want to have pre-installed on the VM for distribution because it's going to be doing the work to get Rails up and running. So we can go over a bit of what is actually happening in the Sinatra app. I'm using a simple encryption library called gibberish that just is a nice abstraction over, I believe, open SSL. And then this horribly insecure shared key up there, ASDF, ASDF. And if you do have your code obfuscated, then it doesn't really matter that the key is right there. And it's just best to assume that if people are going to be looking into your code, they're going to figure out everything about it to begin with. So you don't really need a huge key. So what it does is it first will decrypt the license file to make sure that it's valid. And or it will just use that license file. Let's say if you're upgrading your license, it can just put that in your Rails app. And your Rails app will start picking up the new license. Maybe it's gone from trial to live. So if the package itself is present the package file, then it'll decrypt it and place it in the directory where the passenger is pre-configured to read it from or to load it from. And then it'll just touch the restart file and redirect you straight to root. And so you'll be good to go. So that index is just how simple it can be. I mean, this is like the bare minimum of what it takes. So tweaks to the app itself that we discussed, the license file, in a regular customer subscription app, there's all the data, or the customer specifics in the database. And that's checked on every page load. But we can't do that here because there's no way for the subscription side that you're selling them on your sales app to have any effect to the Rails app that's running on their side. So when they get a new license file, it's just an encrypted JSON file. And it'll live in the temp directory and get loaded on boot every time. So we also want to add some of the Rails secrets, all of them, and for the simple example app I'm running Devise also. So the secret token and device secret can be read from the license file on boot. And you want to do that because you don't want one customer to be able to tamper with another customer's cookies. It's a really kind of extreme use case or possibility, but still just best practices. So this is a really basic module that demonstrates how to read and write the license file. It's using the same encryption key as the loader. And this would obviously be obfuscated, hopefully somewhat, just for a deterrent. But it gives you an idea of how simple it is. The first thing to get loaded in the initializer is so that the rest of Rails can use it when it's booting up. And from here we can have calls out to it all over the app, checking the trial period, disabling functionality, displaying messages, to link out to the web if they need to upgrade their license. And another big one here is the seat limit. And the seat limit is something that can be checked, say, in a user validator, like validate seats and just do account against the license seat number. In that case also for administrators, you'd also want to provide a way for them to manage the users, of course, as though they were using your cloud app. So they would be able to deactivate or delete users so it's a free upseats if they needed to. All right. So next is the support package. And this is the really simple implementation of an exception tracker. And it's going to put exceptions in little encrypted files with a cleaned backtrace into your temp directory to be ready for download by the admin when the time comes, when they start hitting 500s. And that just encrypts it back up and tars it and just is able to be emailed to you or dropboxed to you, depending on how big it is. All right. Lastly is the data import and export management mechanism. So this is just a dead simple PG restore and PG dump. These are Postgres commands, as this would be in a controller. So the first one is purely just taking whatever you spit at it and attempting to shove it into the database. And export is just giving you an entire dump. So nothing really tricky here, but it's definitely an important part of being able to upgrade between virtual machines. For background jobs, I'm not going to get into it too much, but we used form in export, which allows you to take your proc file and generate upstart processes. And those can be used by Ubuntu. And so that will let them be executed on boot and stay running, hopefully. I think the next step here would be for your app itself to show the background job status within the app itself in some sort of dashboard. And of course, that wouldn't be something you would show on your cloud app, but just something to think about. So lastly, when all of your pieces are in place, what's the best way to make it actually distributable? Hopefully, you can pare it down to a single script, because it's just like a Git push, Heroku. This would be something that deploys to your CDN, and maybe updates your users that there's a new version of it ready for download. So some basic things to do here. This is a super basic rake task that just grabs the version, does a bundle, a vendor bundle that throws everything in your vendor directory, and then tars everything up that's pertinent, excludes what's not, and then encrypts it all using your shared key. And that's it. So some of the things that it could do are obfuscation. At this step, if you needed to run a post-processor on it, then it would happen here. And then last one would be actually uploading it to your CDN and then notifying your users. And I keep a version file in the root just so that all these scripts and random things can use it. Because each download of the package is going to have some sort of version attached to it that your user is going to be able to want to go back and forth or be able to identify. So that was pretty much it for what you can get up and running as a prototype. And beyond that, once you actually have it in customers' hands, if they're already using AWS, they might want to see it as an Amazon image, which can be spun up without actually having to download anything, it's pretty trivial to convert an OVA file to an AMI. Amazon provides some command line tools to make this simple. So you could also put that into your packaging script. Actually, no, this would be a different script that would run purely on your VM when you're ready to cut a new version. And then once you have an AMI listed on Amazon, it's pretty trivial to just hit the public button, and then it's searchable in the public registry. So some more things that are going to be taking the next step beyond what we've covered. Resource management, there should be an easy way for your admin to see how your VM is performing. And this can be part of the support package also. Clustering, perhaps you could set up a mechanism to run multiple VMs at once for better performance. Mail server, which is not that tricky, but could be important since in your Cloud app, you're probably using send greater mail gun or whatever. Those aren't going to be accessible from the VM. So your admin should be able to specify a mail server. Incremental VM updates. You could also include bash scripts to change pieces of the VM around when you import a new package file. So when that gets exported or untarred and migrations are ran, those migrations could also include, say, new dependencies that were stored inside the package because you can't just run an app to get install from there. You have to assume that your VM doesn't have any access to the outside world. So we talked a little about the SAS app that would run in the cloud in addition to your main one to sell license files, facilitate that. And last one is enterprise sales, but that's just a joke because who knows about that? That's the big question mark, which I have no idea about either and is definitely uncharted territory. So I think we're running out of time. So maybe we should just do any questions. Yeah, the question was, what are we doing for obfuscation? That's still kind of a work in progress. Right now it's a Ruby C extension. And the way that GitHub does it is they've compiled their own Ruby to do it. So it's at a lower level than just a jam extension. And so figuring out how to do that right now, but it's not something that's widely discussed. Yeah, well, the question is, how do we reduce the amount of patches that we need to ship? Because there could be maybe a bug that's specific to one customer, but we don't want to have everybody else come and download it all at once. Or maybe you can just do a flatwall. Yeah. Yeah, there's really not a good way to tighten up the feedback loop beyond having to bundle and or to package up and then release brand new versions for every little bug piss. I mean, I think that's more of a customer relations question, where it's a small bug. Like maybe it can weight into a bigger point release if it's being fixed just for their environment. Or if it's something that's business critical and it doesn't work for them and they're paying customer, then of course, no matter how small the changes, a new version has to be released. And everybody will see the new version up there. Yeah, it could be incremental like that, where the package in the migration will actually do the updates. But probably that would be for smaller things. Like maybe you can include a .dev that can be loaded in that's not too huge. But for OS level, that would necessitate cutting a brand new VM and asking your customers to go download a new one, then do the whole import, data export dance. Scaling support teams. Well, we haven't really had to scale ours yet because we haven't had so many customers that it's become overwhelming. But I mean, I feel like we're going to just need to scale in traditional ways, but it's yet to be seen what extra sort of hurdles we're going to have with supporting multiple versions of the package and the VM and having this asynchronous support flow where we get emails or drop boxes of just packages of logs and 500 errors. So it's definitely uncharted territory also. So there's two pretty simple little apps up there that are just prototypes.  But it shows off some of the license file reading and pretty much all the screen grabs from the presentation were from that app except for the app loader, which is the little Sinatra app. Anything else? All right. Thanks for coming, guys. Thank you very much.


Hello, I am Dr. Himali Tekani, I am a consultant obstetrician and gynecologist at Manipal Hospital's Old Airport Road, Bangalore. Mothers should initiate breastfeeding at the earliest possible. The first breast milk which flows out is called Colostrum and this is the most nutritious component for the baby. Women should start wearing a supportive bra. They should know about all the nursing positions and use the one which suits them best. The idea is to make yourself as comfortable as possible and the baby as comfortable as possible. Ensure adequate burping after every feed. Feed the baby on demand. The baby is allowed to room in that is to stay with the mother close to the mother's bed. Once every two hours the baby should be fed. One should look for cracked nipples which cause sometimes pain during breastfeeding. Use of emollients or oils would help ensure pain free breastfeeding. Some women do develop breast engorgement and sometimes abscesses in the breast. So if the breasts are very heavy or start becoming painful and tender, you should visit your doctor immediately. One should ensure adequate hydration during breastfeeding and calcium supplements should be continued throughout breastfeeding.