And our next speaker is Philip Cren and he will talk about SecComp. Just welcome, Philip. Thanks. Hi, everyone. Who has heard of SecComp before? Okay. About half the room. Good. So let's see where we can take this. So SecComp, the secure computing model, is something that has been around for quite a while, but is still not as widely used as you would wish. And generally, security-wise, often this is us, and we say everything is fine until something, whatever that is, happens and then nothing is fine anymore and everything is on fire and everything is terrible. And then you always wish that you had something else to fix your security model. And SecComp might be one piece in the big puzzle of security solutions that you could use to help fix stuff. Obviously, there are no silver bullets and there is no one tool that you want to have like more layers and an in-depth security model. And SecComp is exactly what comes into play here. Generally, it's just one piece of least security. So it's trying to take away things that you don't actually need that might just expose you to more risk. And it basically just blocks certain system calls that you know that your applications do not need. So if you, for example, know that your process never needs to fork another process or you never need to call another binary from your port process, why not just take away the possibility from your application? So even if you have a remote code execution and somebody can run whatever they want in the context of your application, you still cannot do some stuff that's just generally not necessary for your application. And that's really just the idea of SecComp. And it's basically just instrumenting the kernel to do that for you.  But it's application-driven. It's not like generally, it's not a Selenux, for example, or app armor. It's really application-driven because the assumption is the application developers know what their application needs and then they can register the right profiles for the applications and then filter out all the unnecessary stuff. It was introduced like forever ago, basically. It had the so-called strict mode. The good thing was strict mode was very strict, and the bad thing was also that strict mode was very strict. With these permissions, you can barely do anything. Yes, you can read and write a file, but there is no network access at all. You cannot even change the system time. Nothing can be touched here, basically, except these four system calls. That was a good first step, but it was way too restrictive for the average application to be actually useful. So it had to kind of develop over time. And a couple of years later, the foundation was added for system call control. And in 2014, SecComp was actually added to make that configuration easier. And I hope that everybody is on the kernel version that supports that, but that's a different story. Anyway, so you can generally, I guess everybody knows, man, sis calls, so you can see what system calls does your kernel version support. And by the way, there are new ones being added over time, and you can see that in sis call as well, which ones have been around for how long. And then with SecComp, you can basically see how you can filter those and not allow specific system calls. The way you write that is BPF, the Berkeley packet format. Where do you normally use BPF? Probably TCP dump when you do that. When you're pretty desperate and you try to find something on the network interface, there's one of the places where you might use BPF. And those are not the most human-friendly ways to write anything, but you maybe get used to writing BPF at some point. So the minimal setup would be, for example, here, we need these two header files, and then here at the end, you basically have a structure where you register the system calls that you want to allow, for example, or deny. And it could look something like this, and this is pretty small. The most important parts are, first, you want to check the architecture, because different architectures, different system call numbers, which can add confusion, and you might allow the wrong ones then. Here, we basically check which system call is coming in, and then you decide, do you allow that? So here, we're basically having an allow list, like everything we allow will pass through, and everything not only allow list will then run into the kill process. And just shoot down your process. Every call that you do, once you have system call set-comp enabled, will then run through that. The question that sometimes comes up is it very kind of expensive because it's for every system call, not really because it runs in the kernel, so you don't have to go to user space for all the calls. So it is pretty cheap to actually run those. The results that you can have is a call can be allowed, that's probably the good case. A threat or a process can be killed. You can also return an error and log any problems if you run into those. Is anybody using set-comp? Are any of the applications that you're using on a daily basis, are they using set-comp? Probably yes. Some of the more widely used ones are these here. So you probably touch set-comp somewhere every day. Especially Docker has used set-comp quite a bit. And if that hasn't been changed in the meantime, 44 system calls out of 300 plus are being filtered. And that used to be quite a lot of manual work for them to figure out what you can block and what you cannot block, but they found a more or less secure setup of what can be blocked out by set-comp. So for example, some stuff that they are blocking is you cannot change the system time because that's not namespace. Time is global on the system, so you cannot change that from within a container. You cannot clone a namespace. You cannot reboot the host. You cannot unshare or copy a namespace either. So these are system calls that are blocked. And you can also check that out. So for example, what you can do, which is obviously not recommended, but you can disable the set-comp filter on Docker and then you can just clone a namespace and then see that you're running here as root. If you use the default settings, then this would not be allowed because unshare has just been disabled. So this is where Docker uses set-comp, for example. Is any of your applications using it? You can also filter that out if you look at proc whatever status. And that number tells you zero means no. One means the strict filter, which is probably too strict and nobody's really using that anymore. So you will probably not have any processes with one, but the ones using set-comp will have a two there. And that's how to identify which processes have actually registered a set-comp filter for you and which ones have not and probably should. And then you can just check out the process from there and see what's being filtered and whatnot. So why am I talking about that? I work for this little company called Elastic. We have some products, and we also use set-comp heavily. If you've never seen any of our products, maybe you've seen this one, the good old elk. Or we've added another component called Beats, and then we kind of had to redo that elk because there is no bean elk and then we tried this, the elk or elk bee. And you can see it flies and everything, but anyway. So in our stack, we have two components that use set-comp heavily. That is first, Beats, which is like a library in the agent or shipper written in Go, since that runs on pretty much all the systems where you want to collect information. And then Elastic Search, since that stores your data and probably you want to add some extra protection there. Lockstash has an open issue, and Kibana as well, they don't use set-comp yet. But Elastic Search and Beats, for example, do that. I'll give you an idea how to do that. Elastic Search is a Java process. How do you set set-comp filters from a Java process? JNA Java native access is the thing what you do. This would be the relevant part from the source code. First, something that is not strictly related to set-comp, but something that every process should do is this one. And people mostly accept that you should not run processes as root, especially server processes. So we will just throw an exception and we try to run Elastic Search as root and exit. And secondly, we will check or we will try to register system calls there. Then, depending on your operating system, set-comp is really Linux-specific, but most other operating systems have a similar concept, even though it might be called differently. And we try to register those on all the available operating systems that we support. And otherwise, just throw an error and say, we don't support that one. There are different ways to do set-comp, but we'll stick to Linux and then it's just set-comp. And this is basically what we do in Elastic Search. So the joy of BPF at work, the main ones that you're interested in, for example, you cannot fork a process and you cannot execute another binary. So even if we had a remote code execution in Elastic Search, which hopefully we don't, you could not fork another process or execute any binary on the system because we filter those calls out. So that just wouldn't work. Then we have beats, which are this lightweight agents, and they have basically their own syntax to filter that out in Go, or they wrote their own library because they didn't want to write BPF itself. They wanted to have YAML instead. So basically they created something like this. Here you can see this allows all the calls by default and then filters out specific system calls. So this is basically a denial list. Is this what we're using in beats, actually? No, because beats, since it's shipping data over the network, would need some network access and this doesn't allow any network access, so this is just a sample of what we do. Actually the real rules are many more, but we do use an allow list. So we basically have whitelisted all the actions or calls that we need. This will be a very long list, and everything else will be blocked. So every time a new system call comes out that we need a new one, and we want to whitelist that one, we will need to update that list manually. But as always, allow lists are more secure than denial lists. Because new system calls, you might allow something otherwise that you don't have intended. So we could just try that out very quickly. For example, if you run good old Netcat, and then you could just open a connection and say hello, and ideally my hello should arrive. Where's my hello not arriving right now? No, this is, Netcat doesn't use second, and also, but we'll get to that in a moment. Let's see. Ah, this is the wrong host, my bad. Because this is a demo, and I'm already using this domain for a demo. Because, and well, if you use the right one, this works. You can use it as well, it's just online, so you can start chatting. Now the main questions are, which system calls are involved here? What are we using to figure out the system calls? S-trace, yes. So we can just run this with S-trace. Oops, S-trace. And that will show us a ton of system calls that will be needed here. The most important one is at the end when, okay, I think somebody said yay here. And that then killed, with a bad file descriptor, my process. But you can see all the calls that we are doing here. And that is, by the way, one thing that is kind of relevant. So yeah. If you run S-trace, and for example with dash e-bind, you can just filter down on the bind action, because that's the one I want to monitor here. So you could just see, this is just a system call for the binding. That's how to monitor that. And if you leave that out, you see a lot of system calls. And this is, by the way, a good way to figure out which system calls does my binary even need. What do I need to whitelist the other way to figure out which system calls do you need to allow for a binary to write the right second filter. There is under this address, you can find a little C utility, which you can add, which will then collect all the system calls that you're doing. Otherwise, you can always use the S-trace output and just grab the system call, which is always the first thing in every line. And you could just aggregate those what you need to allow or what you can deny. And then you can simulate what would happen if you would deny a certain system call. And for that, there is a little tool called FireJail that can basically run or deny specific system calls. And you can run it like this and let me paste it in. I think I've copied that because nobody wants to see me type that. So we have FireJail without the profile. We drop the bind permission and then we just run S-trace on my binary and I tried to bind to port 1025 again. And then it just tells me well my system was called my process was killed because you can see I tried to open a socket, I then tried to do the bind and exactly then does FireJail kill my process just as expected. It's not allowed here. And yeah, that's the output you will get here. One thing that is sometimes interesting is what happens if somebody takes over your binary and then tries to change the filters. Any guesses what will happen? Will it, you can only, I'll give you three options and you can decide what you think is most likely. First you can only set second comp once. Second you can drop the permissions to change anything in second comp. And third euro. The third, no it's not euro. So what you need to do is when you set up second comp the first thing you want to set is set no new privileges. This basically means you can always tighten down over the life cycle of an application. You can always tighten down the rules but you can never loosen them up. So once you've taken away some permission you cannot add it anymore. So even if somebody takes over your binary you can allow a fork or execute or whatever because you can only tighten it down. So this is the first thing that you should set to make this secure. Are we doing that? And I was kind of curious how we were setting that up. Yes we are. So for example here you can see the system call since Linux 3.5 for set no new privileges is 38. Then we try to set this one here and afterwards we actually take and if setting that filter did not succeed we would actually arrow out on the process. So we ensure that this has worked and you cannot change the permissions afterwards. Beats has since it has this YAML syntax just has this no new and you're done which is maybe a bit more readable but the library in the background that does all the YAML parsing is just the same with VPF to set the right filters. And then you probably want to figure out that somebody has kind of like caused some second violation. So Audit D, the Linux auditing framework can also collect those. We have another component in our stack called audit beat which can collect any violation of those and then you can just collect those and see where it's going. So for example you can look in Kibana, you could just say like give me all the violated second policy actions and then you would see these were all the kind of like things where you violated the policy. And then you would also see like which user which binary which system where that happened and then you can try to find out like do I need to change my rules or did somebody break in and try to run some filters or system calls that they were not we also have this new scene component which is more drag and drop oriented and then you could drop that in here and this shows you this was when I was trying that with Netcat and you could see for example you want to use on the what the fuck host was trying to use Netcat and that was actually a violation and I filtered that out here and then you could would know like which binary is affected which user might be affected and you could filter those down. So to wrap up it's always kind of Lego security so there's not just one block yet you need to put in but there are many blocks that you need to combine to have better security and SecComp is just one of these little building blocks. Sometimes people ask like how does it compare to Excel Linux or AppArmor. One is or both of them are or all the approaches are kernel level filters or interceptors. The main difference is that SecComp is where the process actively sets that and basically the application developer roasted out with its application whereas it's setting it mandatory before the application is even being started which is what Excel Linux and AppArmor are doing. So it is widely available hopefully all your kernel versions support that you can use SecComp and hopefully have more secure applications. If you want to write that more easily from your own application code there is a library called LibsecComp and they have a lot of samples of how to write the filters from various applications. So that might also make your life a bit easier to apply SecComp in your applications. And then people always ask me about Windows and I haven't used Windows in years but I googled and this seems to be the most similar system called filter that Windows has and it has I guess everything is as terrible as the name is there but this is what you get and if Windows is your thing look into it but I have no idea how it works and I don't really intend to look into it too much but this is the closest equivalent that I could find and that's pretty much it. I think we're right on time for a couple of minute questions. If you want stickers grab the stickers afterwards wait for the microphone so everybody can hear the questions. By the way before we do the question I always try to take a picture with you so I can prove to my colleagues that I've been working because we're a fully distributed company and nobody knows where I am today. Can you all wave? Very good thank you. Question? I was waiting to and wondering why you didn't mention the SleepsecComp and it was just on the last slide so the question would be why aren't you using it actually because as you mentioned doing that work manually is quite hard you have to have those defines when, which kernel version, which architecture, all that stuff and the SleepsecComp is actually doing that for you and making those ugly defines macros not necessary and all that stuff right? And also gives you a much easier way if you because that thing you also I think not mentioned that you not only can filter which system calls can be used or not but you can also check the arguments, so for example you can only allow certain syscalls if the arguments are right like you can allow read or write or open on some file descriptors or things like that. So the main question is why aren't you using it if it's there and you know about it? To be honest I don't know because I've only been with the company for four years and it has been there before me so I kind of don't know the history why we started on our own path with with our SleepsecComp. I will try to find out, this is actually a good point, I don't know why we maybe we didn't just didn't know about it I don't know but it was a long time ago so hi, great talk, thank you. In Docker when you look at syscalls there is a SecComp that is involved and also capabilities and I was wondering could you please describe how they both fit with each other like is Docker generating a SecComp policy relative to which capabilities are enabled or like I feel like there is two moving parts that I don't know how to fit together. With the capabilities you mean like NetAdmin for example there is a Capsys Ptrace that controls the Ptrace syscall but you can also block it in SecComp. Yes, I don't know how they join together but you can override basically the SecComp filter with that setting and you always set so I know that for example if you want to run a container that monitors the network calls from other containers you need to set that NetAdmin that must override the filter because otherwise you wouldn't get to those. But how that is implemented in the background I haven't checked the source code of Docker for that. Okay, thank you.


to do is split this into two things. First of all, I do sales for a living. I've done it for a lot of years. I could talk about the, well okay, I could talk for hours anyway if you know me, but I could talk about this for a long time. I think I've covered most of what I wanna tell you, but I'm sure I'm gonna miss holes. So this is not about me being unprepared, but this is about me trying to divide this into two sections. One is a talk, one is Q&A. I gave this at MountainRB, had a question that led to one of the best things I wanted to say and forgotten. So the person that gets me to talk the longest with their question gets a bag of coffee. And that is not hard, again, if you've met me. So I'm here to tell you about sales, to talk to you about why it's done, how it's done, what you can do, and I'm secondly here to tell you that Dave Thomas was wrong in his keynote. I do not want you to spend the next year learning a new programming language. I'll tell you about that in a couple minutes. So what is sales? Anybody? Oh, come on, you guys have got to start talking. Seriously. Making people give me their money and giving out candy, right? So sales is not necessarily strictly financial. Sales can have a financial aspect and that's a large part of it. Sales is also not, well, we'll go into that in a minute, but sales is about finding common ground. It's about education in a lot of ways. It's about people understanding that there's something out there that they don't know about, that there's something out there that can help them, that there's something that you have and possess that would help them get better at something, make their lives easier when it comes to DNS hosting, or when it comes to invoicing people for BlinkSale, when it comes to doing project management with Basecamp. It could be product-based, it could be person, it could be a person that's actually telling you about a company, selling services. I started the company Edgecase four years ago and I've been out there telling people what we can do for them. So it can be a job. It's about understanding how to frame discussions, how to put things in a certain mindset to help people, about understanding people, and it can also be about making friends. But most of all, it's about positive persuasion. It's not negative, it's not tricking people, so it leads into, yeah, so I'm getting ahead of myself. So I'm actually trying to do this whole thing where I follow an outline, which again, if you know me is really weird, so forgive me if I jump ahead. But what is sales not about? Well, first of all, it's a not a zero-sum game. Anybody know what that means? If there's a winner, there's a loser, exactly. By the same amount, great. Sales is not a zero-sum game. Sales is not about if I gain, you lose. People think that that's the case because a lot of time there's a financial aspect. You're trying to get them to spend money. Well, obviously I'm getting the money and they're doing away with the money. It's not about that at all if you do it correctly. It can be about that, but it's a positive-sum game. Everybody can win and everybody should win in the end. If you're not, you're doing it wrong, okay? Sales is not about tricking someone. That feeling when you, and I'll hat-tip to DNS Simple here, that feeling you get when you go to GoDaddy and you buy a new domain and you accidentally buy a used car, right?   Right?  But no, it's not about tricking someone. It's not about getting them to do something they don't wanna do. It can be, but it's not. That's the wrong part of sales. It's not about getting the most from somebody you can, milking them for everything they have. It's not about walking away going, I convinced them to spend $10,000. They should have spent two. I'm good. Again, people can do that. Oracle's made an entire company out of it. And it's also not just one job. It's not always one person's role. Sales is not always about putting on the suit and being the person in front. Sales is a part of everything. So, show of hands, consulting companies. Okay, decent amount. Product, who's working for a product company? Who would define themselves as other? Okay, okay, not both. Who would define themselves as other? Okay, okay, okay, great. You make content and sell, okay. Okay, awesome, okay. So, everybody here in some way is dealing with sales. Or has to be dealing with sales. If you're a consultant, it's pretty straightforward what you're doing, right? You're promoting your services. You're promoting what you can do for somebody for a certain rate, right? Sometimes in products, it seems a little different because, well, I'm just the programmer behind the scenes.  If you can't get behind what you're doing and what your organization is doing, you shouldn't really be there. Think about that for a while. And we'll talk more about why. So, sales is, it's about persuasion. It's about education. It's about all of these things. But in the end, it's also about getting in the right mindset. In order to do sales, you have to be in the right mindset. Because there's a lot of baggage that comes with the word sales, okay? We picture the sleazy guy in the suit that comes in and convinces you to do something and you walk away going, how does he have my kid and why is he walking out the door, right? It's not about that. It's not about that, but it's what we bring with it. It's the baggage that comes with it. So, part of that right mindset is understanding some things. First of all, the business world in general allows us to eat, right? We are working during the day because in essence, we have to. Most of us are lucky enough to be in jobs where we get to do what we really love. And there's a certain aspect to be said for, in our industry especially, if I won $3 million tomorrow, would I be doing my job? Well, maybe not your job, but quite honestly, I think half of us would be hacking on something, right? So it's harder in this crowd to say, you're doing this because you have to. But truly in essence, you're doing your day-to-day job because you have to, because you need to eat at some place, right? But business is allowing us to do this. Business is allowing my kids to grow. It's allowing me to go buy that guitar for my daughter that I wanted to buy and sign her up for lessons. And it's about being able to pay the bills at the hospital, unless you live in France. At the hospital, if something goes wrong, right? We all have to do it. A lot of us have this mindset that if we convince somebody to give us money to do something, it's a dirty thing. But truly we all have to do it. They have to do it just as much as you have to. As you understand, it's just a part of what we're doing. Your mindset starts to change. We carry a lot of negative baggage around with us when it comes to sales, when it comes to business, when it comes to business transactions, when it comes to marketing, when it comes to these kind of things. But truly we're all doing this because we have to. And other people are doing this because they have to. Other people need services. Other people need things. So it's okay. You have value. Another thing to think about, you have something to offer. No, this is not one of those Stuart Smalley, I'm good enough, I'm smart enough, and dog-gun it, people like me. It's not that, but you have financial value. If you don't, you really need to quit. What you have, the skills you have, help people in a meadow way, right? We build things that help other people do jobs. We build things that help people get much better at their jobs. So looking at proposing for an application this past week, ran up to a city with Dan, our sales guy, and was doing some business analysis. And realizing there was one thing in particular where the system was set up in a really terrible way and they had input these numbers and she couldn't find these other numbers. And you realize she's spending on average about 20% of her time on this one little thing. So if we approve this application and they're paying her state employee, I don't know what, let's call it 60,000 a year, you could save them somewhere between 10 and $20,000 a year simply by doing this one thing. And that frees her up to do other things, to find improvements in what they're doing, to help them in other ways, right? Everything we do has this financial aspect that has a value to what's happening and you're helping them. Understanding that you have that value will help you later on down the way, because again, it's not a bad thing. Asking for money is not the root of all evil, like most of us like to pretend. So what should you do instead of learning a new language next year? I want you to go out and learn psychology. I want you to think about this. We love to hack on machines. We love to hack computers, we love to see how they work. We like to find their little intricacies and work around them, build tools around them to help them. But every single one of us has one of the greatest computers out there sitting on top of our heads and we have no idea how to use it. We have no idea how it works. We don't take the time to dig into it. We'll dig into the innards of any other machine. Some people here will go down to source code levels I didn't think I don't ever want to see personally, just because they're curious on how something's working. I love Ruby, I'm not going to read parse.ly, but I know other people that have. But then these same people won't sit and think about how we interact with other people. So there's a great starting point that we have in the gig community and it's pragmatic thinking and learning by Andy Hunt. And no, I'm really not paid to talk about this book. If you've seen me talk in the past year, you're going to start to think that because every single talk I've promoted the book, but he does one of the greatest things I think and that's give us a starting point for understanding how the brain works because he gives us what's called a mental model.  That's what the brain is. And what's fascinating is that you understand so much more about what's going on because certain things are processed on one processor and certain things are done on the other because it's better at it. But it's a shared bus. There's communication errors that happen all the time. There's certain things that happen because of this. Is this deadly accurate in medical terms? God know. Although it'd be kind of cool if it was. But it helps you understand certain things. Once you read that book, start looking into the field of social psychology. That is psychology and the brain and how we interact with people. Now, about four fifths of the room out there, the introvert's going, oh God, right? Interact with people. But truly, the more we understand about how other people are and the more things that are going on, we start to realize a lot of our fears are simply misunderstandings about what other people are thinking. They're also misunderstandings about what we're thinking. So if we, yeah, yeah, cover that. One of the things the brain does is pattern match. It forms patterns out of things in order because we're taking in an insane amount of information. Every second, we've got tons of sensors that we don't think about that we don't know are there and the brain is taking them, shaping them, putting them together and putting and abstracting them. Giving us abstractions, just like we do, right? Understanding that this happens, how this happens, why this happens, helps us understand some negative parts of that. That's an interesting out of order problem. The first part of this is in stereotypes. If you understand what the brain's doing in stereotyping, you'll understand a lot about what's happening around you. Now, stereotyping is interesting because it's one of those words that, again, is loaded, right? We think of it in bad terms. I say stereotype and some people think white hoods and back in the 60s problems, right? We start thinking of sexist comments. We start thinking of these things. But realistically, stereotyping is something the brain does because of the quantity of information we do. It's not going to stop. But if you understand that it's there, you can let it not affect you in a negative way. You can harness its power at times, but you can also step back and understand why people are interacting the way they do. One of the ways is in understanding what matters to other people. Now, I've just talked about stereotyping, and I'm going to make some gross stereotypes here about us. Who here was at Rubicon for Orlando? Who here remembers the bar scene on the last night? Anybody remember why the bar scene was so awkward? There was a suntanning convention in the same convention center that started the day we left. That was a hilarious scene of social psychology right there because you had on one side of the room a bunch of us, geeks, laptops, and backpacks, right? And on the other side of the room, you had lots of suntans, lots of built bodies.  And what was funny is the whole, we've seen the geek thing played out in TV over and over again about how certain people feel towards nerds and that kind of thing, revenge on the nerds, right? But what's interesting is our reaction to them as well. And them being, yes, a big group, right? We automatically see people who go and get suntans and suntanning beds or spray on tans, right? As having less intellect, geeks tend to very quickly apply a value to people who value different things, right? An intellectual issue. We automatically think of somebody less if they value how they appear and what's going on on the outside. It's not true, right? It's a gross generalization. Maybe it's true for some of the people you've met, but does that mean every single person applies that way? Some people think the same thing about us in showers. Does that mean everybody here doesn't take a shower? Thank God that's not true. I wouldn't be standing up here, right? Not in a closed room, at least. But understanding what matters to other people can help you in a large way. How does this tie back into sales? Well, one of the examples I like to use is sitting in a meeting. You come into a meeting and I've seen this happen time and time again. Programmer comes in, person having a problem comes in. She's sitting there in a business, okay, yes, I'm using a sheet, I'm just pulling this out random. Sitting there in a business suit, Franklin Day planner in front, very collected. Penn's put in the right position, this and that. Guy comes walking in, hasn't shaven in a couple of days or maybe he's taking part in November or something. Has a t-shirt on, it's got a picture of Han Solo. Says who shot first. Flip flops, you know, Coke or Starbucks or something. It sits down, puts feet up on the chair. Instantly there's a failure to communicate on both sides. If I was up here giving a talk on sales, but I decided to wear a suit. Well, okay, that's a bad example.  I go into a user group and I want to start talking about something. I show up in a suit. The reaction, if they don't know me, the reaction is phenomenal. We instantly have a lot of distrust. As a group, we have this weird distrust of, wait a minute, what are you trying to trick me over on? What are you trying to do? What are you trying to sell? What is it you're pushing? Which is interesting because when we show up in a Han Solo t-shirt, other people instantly start thinking, Jesus, is this boy ever gonna grow up? But we know that who shot first is a really critical argument that happens, right? This is a big discussion. This is funny. Somebody walks in in a t-shirt like that to an interview. Half of us go, wow, he's gotta be good. So understanding that we have these biases is really important. Not because I want you to stop, but I want you to understand that they're happening. Because we value things differently. We don't value them better. Right? Just because they have a pseudon doesn't mean they're any less bright. Some harder ones to think about. What do geeks think of people who have right-wing political beliefs? Don't answer that please, I don't wanna start an argument. What do geeks think about people who have strong religious beliefs? I know a lot of people who are not big fans of Uncle Bob since he started Twittering. It's pretty sad, because it's a different opinion. But we do that, right? And if we understand that we do that, conversations can be much more productive. So this whole thing is simply to tell you, look into it. Read some more about it. Understand some more about what we do. So when you're sitting in a meeting and you're talking to somebody, or you're sitting there and you're trying to tell them about something you have, it's not a matter of value. It's not a matter of being smarter than somebody else. Well, okay, you probably think you're smarter and nine times out of 10 maybe you are, but that doesn't matter, right? What matters is that we need to communicate and we need to come a step in their direction. I'm not saying put on a suit, but I'm saying understand why the suit matters to other people and the conversations can be more productive, okay? The other part of understanding how the brain works as a computer is you can understand the fact that we have bugs in there. We have bugs in our brain and we have a defined list of, that we have an errata for it. They're called cognitive biases. There's actually a wiki page on them. They're fascinating to read. They're common bugs in the brain. Now, we stereotype all the time, right? I'm telling you this, we find abstractions. Well, there's this, and I didn't write it down on the card and it's gonna bite me in the butt. I was hoping it would just come to me at the last minute. But there's a symbolic reduction fallacy. Thank you, it did. There's this one cognitive bias about when you reduce something to a thing of symbols, you miss a lot of the larger part of the scale of what's going on. There's an intense need for, there's another one that's an intense need for closure.  Well, if there's an intense need for closure, does that explain why we sometimes have trouble convincing people about how good agile development can be, right? Maybe that's why we're having trouble communicating some of these things. We sit there and we get frustrated because we're like, this is the way it should be done. This is what's there. And they're sitting there going, but I don't, are we jumping off the end of a cliff? I don't know. We also need to learn, not just psychology, but also spend some time learning how to debate. Get some books on debate. There's some interesting aspects to how to frame arguments and what happens. There's some really interesting exercises that come out of this too.  No, I was terrible. I got slaughtered. But one of the interesting things we did is when we first got there, we filled out this big, long questionnaire on a lot of our personal beliefs on things. It was really hard to answer some things. I mean, there were questions on abortion, there were questions on, and as a teenager, you're there trying to, you don't really wanna write this down, you don't wanna think about a lot of this stuff, but how do you feel about the death penalty? How do you feel about these things? And not knowing what was happening, going into the class, I spilled out my heart, right? You get into the class and you immediately had to do debate from the other side. So at the time, it's been an interesting progression of my life about where I stand nowadays, where I used to in high school, but I got to argue for abortion because I did not believe in it at the time. And so having to do that from the other side was fascinating. So even though you don't believe in it, do it. Doing this helps you. Doing this helps you think about what the other side's thinking about. So if you're out there selling your product or your service, start thinking about why you shouldn't use your product or service. Start thinking about where you fall down. Who here can tell me what a straw man argument is? Okay, exactly. Exactly, you're both right. You're setting up a weaker argument.  Understanding how this works can really help us understand how we're framing arguments and things. So hold that thought for just a minute because the next thing is understanding when you're in sales, when you're in a role of persuasion. Understanding the differences between features and benefits is one of the most critical things. If you take nothing else out of here, this is what I want you to remember. Understand the difference between features and benefits.  I see closure talks all the time where they go through and they talk about all these little features of it. Ask yourself the question, so what? For every feature, there's a benefit. If there's not, forget about the feature. I'm not talking about product planning here, I'm not talking about your product in general. But I'm saying every product, everything that we use, every service that we use has a defined list of features. But the people that are coming in to use them are going to be unique. Your job is to understand the match between this feature and this person. So setting up your arguments in strawman form might make you feel better. You come in and talk to me about Git and you start beating up on SVN branching. Okay, you're going for the easy win. If you're telling me about beautiful code in a keynote and you're showing Python, yeah, okay, you might get a laugh. Are you really convincing anybody of anything? Right? Understand the arguments you're making, understand how to frame them, but understand how they match. One of the funniest examples was, I was in a sales training for, so a little bit of my background. I started off as a geek, went in for one of my first interviews, and it was the most deflating thing I'd ever done. The guy barely talked to my feet. He goes to show me this desk I'm going to be in and we go through this 10 minute walk of mazes of cubicles where they're like nobody.  I'm looking around, there's not a soul around me. There's a phone on the desk, I'm kind of like, oh, at least there's a phone. There's a gab, but it doesn't ring that often. And here's a computer you're gonna be working at. I'm going, oh my God. In case you haven't noticed, slightly extroverted.  So despite loving computers, this was not for me. And I looked, yes, I looked at one data point, made an incredible generalization about the entire industry and went, I'm not doing this. So I left and started trying to find my way. I ended up in hotel sales of all things. By series of accidents, I was a secretary at a hotel.  True story. But anyway, so I find myself in hotel sales. I go to this training and it was on the differences of features and benefits. We spent three days talking about this. It was phenomenal. Really made me look at things differently. Well, one of the funniest ones they talked about was this lady and she was the one that did it. She's given me the presentation. She's like, yeah, one of my favorite ones was, I'm giving this proposal for this association. And I go through and they had some special requirements that they were asking about. And I looked into it and I knew we could do those and all this and they come in. The leaders of this association come in for a tour.  We spent all this money on rebuilding the pool and it was great. It was so beautiful now. It was this beautiful deck there. There was this hot tub. It was big and I'm in there and we're probably in there for 10 minutes and you can see the lady kind of getting antsy. And finally she's like, Deborah, you know we're the paraplegic association of America, right? She'd spent all this time on, do they have enough ADA rooms? Do they have the ability to get around hallways? Can they do this in the meeting room? Never made the connection that the fact that you have a pool might be okay for some people, but in general kind of lost on the whole conference. Spending a lot of time talking about this when it could have just let it lie. It was a phenomenal example about not understanding your audience. The other one was I've spent as much time selling Ruby as I have selling edge case. Come from the Midwest, Columbus, Ohio, not exactly Tec Mecca. I'd love it to be, but it's not. So there's still a lot of people that just don't know about what we do. Being the only Ruby shop in town makes it a little bit easier, but in general the reason we started edge case was my fascination of I love for Ruby as a language. My fascination with doing things right. So we start the company and I go out and the more people I can get to use Ruby, if I can fight on the supply side instead of the demand side, I think we're all gonna win. That's been my theory from the beginning. So I go out to conferences and I go into days of.net. I go into conferences where it's mainly Java. I go into these places because I wanna talk to them about what we're doing. And I saw this interesting thing happen where I had forgotten my sales training. So I'm in there and I'm giving them talks about blocks. I'm giving them these things and I see these blank stairs. And I couldn't figure out why people weren't as excited as I was. But I wasn't thinking about it from their perspective. Sat down one day, completely rewrote my talk. Showed them like four gems and four features. Had more compliments about my talks. Had more people communicating with me afterwards about it. And I realized a couple things. And Nathaniel touched on one of them in his talk the other day about something he and I discussed, which is first of all, we're at a conference. I'm not gonna teach you anything in an hour. If you walk away with one or two points, I'm extremely happy. I'm here to steer you in the right direction, to talk about something and hopefully inspire you. That's it. I realized I was not doing that in my language talks. So I talk about blocks and I kinda show them. Then I talk about, and then what do I do? I bring up builder and I start showing how builder works. Feature, benefit. Does everybody use builder all the time? No. But builder is something they've not seen before. It's thinking about things differently. I go in, I start talking about message passing. I start talking about our small talk background. What do I do then? I show dynamic finders an active record. Do we use dynamic finders a lot in active record? Not usually. But you know what? People who haven't seen that before, people who haven't seen that possibility before start thinking about it in terms of differently. So understand the differences in features and benefits. And then a couple smaller points. If you're in the sales process, you're in the right state of mind. Couple things you need to learn. One, I've had trouble learning. I still have trouble learning if anybody knows me. And that is the art of shutting up.  The sales, the hotel sales trading. We were talking about this. And he says, yeah, perfect example. I'm sitting there out there. Got this new client. Only been doing sales for about a year. I'm trying like hell to win his account. He's there talking about the hotel. And he's like, he goes out front. He's like, about this parking thing. And the guy looks up at him and goes, oh yeah, we'll take care of that for your group. And he stops and he looks. He goes, well, I'll take you up on it. But that's not what I was gonna ask. He was worried that they could fit enough cars in the front at the same time. Now 17 bucks a person for valet parking meant for a 300 person group. He just gave up a lot of money because he didn't shut up for a second. The second thing was an example where Adam rode with Ken and I, my business partner, down to a sales meeting. This is a few years ago. And we're sitting in a meeting and we're talking about what we can do for them. And we're talking about how we can manage this project. And apparently, and in retrospect, we realize what happened. It's hilarious. But Ken and I started negotiating with each other. We're on the same side. They're not saying a word, but we feel the need to fill the silence. By the time we left, we probably had $20 less of an hourly rate than we should have. We probably had given them way more than we wanted to.  I started thinking about him like, he's right. Like I feel this need to fill silence, right? So little thing to take away. Let people talk. Don't talk yourself. And if you know me, that's actually a hilarious statement coming from me. In the end, though, everyone needs to know a little bit about sales. You need to be able to sell the product you're working in. You need to have buy-in to what you're doing as a company. I find that extremely important. You need to understand what you're doing. Sales is not always about going out and convincing somebody to spend money. Maybe it's about convincing your boss that you really need to go explore Mongo because it's a better option. Well, why is that? Don't set up a straw man argument. Don't give him a list of features.  It could be about convincing your wife why you really need an iPad. Features of benefits are gonna be really important there. But understand that we have, as sales, it's the best job you could possibly have. It's a meta job. Feeling the first time you ever wrote some meta code. I do. I remember the first meta program I did. It was in Pearl, yeah. But it was so amazing to write code that turned around and wrote code and did something. If you're a parent, if you're not a parent, it's hard to understand. One of my proudest moments, I always thought would be in accomplishments I do. But as a parent, I realize the best thing that can happen is watching your kids accomplish something. Having a meta job is amazing. Being able to go out and convince people to use your services, there's a feeling like none other. I'll leave you with one last story until we take questions and answers. It was in a hotel sales. These people come in the door, it's a slow afternoon. These ladies come in and they're like, look, we just need something very different. They were a law firm out of Chicago. They were coming down to have a six week long court fight. Federal court, six weeks or more. It was gonna be very long. Lots of documents, lots of lawyers. Federal courts were in Cincinnati, Ohio. They basically needed a temporary office. They didn't wanna go rent office space because they didn't know how long it was gonna be. They come to us. They'd gone to three other hotels who gave them, yeah, you can use this conference room, da, da, da. Now imagine working in an office like this. That would suck, right? And they're basically saying, what can we do? Is there anything we can do? And I sat with them and talked to them and thought about it. I remembered having a conversation at the time I was smoking. I was in the smoke break area. Talking to one of the engineering guys about a bunch of suites we had that we weren't renting because the hotel didn't have the budget for the furniture to completely be able to rent these because you have to have all the furniture in place for your inspections.  Let's go take a look at those rooms. So I did something in sales which you're never supposed to do, which is take a client into a room that's not ready. But we start looking. I'm like, how would these work? We start looking. They were fascinated by the idea. The rooms needed a couple wall plates and a couple fixtures hanging from the ceiling. That we could have budget for. So I'm talking to the engineering guys, talking to them. In about three hours I had this contract worked out. We had 15 lawyers come in and stay in suites for six weeks or more. We rented out these rooms that were not renting anyway. And we found a solution. We found a positive solution. That was kind of cool. It was one of my proudest moments as a salesman. My first real proud moment as a salesperson.  So there was not a lot of times of pride at the time. A couple weeks later they start, they come in. It's the first day there. Our catering staff worked on a top down, it was a union. So they had this top down list. First person most seniority always got the meetings and then they would come in. And so you'd see the certain servers coming in and out, walking around all the time. And then if there was really busy, you'd start seeing people you hadn't seen in a while. Nights that we had weddings and things like that. You'd see these people. Well, sitting in a smoke break room and it's during the day which is usually our slower period. We had probably three servers on. I see 10 servers walk by. And the last two I didn't recognize at all. I hadn't seen them before. I sat there and realized they're there at work on that day because of something I had done. That was a cool feeling. Since then I've started edge case. And if there's anything cooler than raising your hand in the other room that says you get paid to do rails. It's being able to raise your hand and go, because of me I'm actually able to pay other people to do rails. That's pretty sweet. Thanks. Questions, tomatoes, comments, anything. Yeah, thank you, throw something. Actually I shouldn't say that at all, please. Jake, so what was the thing that helped me turn the corner to become more of a salesman in the programming world? It was the mindset. It was the understanding that I'm doing this because I love it. I'm doing this because I feel like we're developing software in a better way than a lot of other places out there. And that I have something to offer them that's gonna help them in the end. And the third thing is just understanding that, maybe it was fourth, I lost count, is understanding that we're all in the business world and we need to be doing this. I'm not tricking them into giving me money. I'm not convincing them to buy a piece of plastic or anything they're gonna regret later. I'm getting them to buy services that I know they're gonna love. And so that was it. Anthony. What you're talking about was direct sales, but there were some things that it sounded a bit like marketing to me, and I'm curious on your take as to how much of sales is real sale, like pure sales, hands on, how much of it is marketing, and do you find yourself really doing both in your role? Yeah, so you can project, everybody heard that. I do find myself doing both and I find them to be one and the same. You can't do one without the other. You need a message to be able to talk to people. You need to understand what it is your product has to offer. And I think marketing to the masses of just, here's my features is terrible. I think marketing to certain subsets of individuals about what they need to know about your product is what's there. The most powerful messages we hear are about one feature of something for one particular group that cares, and so it's combining that. So I find myself, because we're also a small company, I don't have separate marketing and sales, but yes, there's a lot of direct sales to this, but being able to do direct sales helps me be able to do marketing on a meta level as well because you're literally hearing from people, you're not trying to guess what it is they're thinking of what you do. So I don't think you can separate them, truly. Does that answer your question? Pretty quick. You just wanted me to talk more. I know. Okay, that doesn't count. Yeah, what's up? Second question is, you talked about shutting up, right? And the point of shutting up to me is listening. That's where you listen to what the other person has to say.   Not usually. I don't know if there's legal implications with it, but as an honest person, if I throw something out there, I have to stand behind it. I want to. So a lot of times it's dealing with the consequences of what you've done. Other times, it's fumbling and trying to figure out what's there. I don't know how to answer that as a whole, like here's the magic answer, but there's different ways of dealing with it. Would you admit to it and would you say, or would you try to work back into a line that is more favorable to where you want to go? So if it's about persuasion, would you try to persuade them back into the where you want to deal with when you go, or would you admit to, you know, oh you know what, I overstep my bounds on that or I say, you know, how do you approach that? If it's something that I can't stand behind, then yeah, I would, you know, you'd have to go to them and just be honest with, I overstep my bounds, here's what we need to do. But otherwise, learn from the mistake you've done, and maybe that's right there, something you can correct, but maybe that's something where you have to give up on what you did here and just know next time. So some key elements that we do is, A, I don't talk about rate until it's absolutely necessary in a conversation, because not everybody that's calling us is first concerned about what rate they're paying us. Number two, there's always a timed element to everything we do, and that was a lesson that I had to learn because we had some people paying rates where we decided we were not where the market was. In fact, we were off by a large percentage, and so we increased our rates and we didn't have a timed element to our contract. So there's that ability. And then yes, being honest with them, going to them and saying, we're all in business together, it's been two years, we need to talk about this. And usually people are willing to have those kind of conversations. How do you sell to an opinionated audience that has a solution in mind but you don't think it's the right solution? Well, first of all, not thinking that's the right solution is good, but make sure you frame it in the positive.  Spend a lot of time showing what your solution does and talking about your solution. I don't spend a lot of time talking about why waterfall programming is terrible for you.  I don't spend time telling you why I think a cobbled together Drupal like WordPress-ish solution is gonna be a bad thing. I spend time talking about what we can do, but first of all, believing in it. Because the other thing is being able to walk, understanding when you need to walk away.  You need to, I mean, not here, but you're in the wrong conversation, right? And so understanding, having a passion for what you're doing and truly believing in it are the first steps. The rest of it will fall in place. And then if they don't use you, try to ask them why and learn from it next time. In sales, you can iterate the same way you can in software. With people, you can iterate the same way you can in software. I didn't know what I was doing when I built the business. When we started Edgecase, I had no idea what I was doing. Still to some degree, don't know a lot of what I'm doing, but we're iterating on it. Understand where you fell down, understand what you did wrong, and correct it next time. How do I convince people to go to hourly without fixed features? Much larger conversation, but it does start off with, first of all, understand that you're not trying to trick them. You're not trying to win one over on them, but truly this is a better solution.  I will do some business analysis in the sales cycle to kind of try to get an idea if it's bigger than a bread box, right?  Is it gonna be $200,000 solution? Or is it somewhere in the middle? But second of all, identify the big risks. If you can point them three areas where this could be a can of worms, that's usually enough to show them what's going on. And then second, and then last thing is, I try to tell them, I'm not asking you to spend $200,000 with me. I'm asking you to spend X amount of dollars at a time. And there's a lot more things we do that go into that too, being remote and all that, but that's a lot of that. So I'm out of time. If you have any other questions, find me. Thanks for coming.


Charlie has a condition called listen carefully, which means smooth brain, so basically his brain's not developed the way it should, he can't walk or use his hand really. But as you can see he's a very happy little boy. I think when people buy a lottery ticket you don't necessarily think about places like Clarehouse. The pajama party today is all about celebrating 25 years of the national lottery. By playing the national lottery you're able to support children and families going through the toughest of times, but also making the very best of memories. And I hope that we're able to convey some of that today. It feels like a happy, special, positive place, but also full of support, medical, emotional, but also giving these kids a childhood and making really special memories for the family. The support that Clarehouse gives us is huge. It gives us a little bit of time together, away from the caring role and looking at it as a bell. It's just massive, it just helps us to function really, doesn't it? It just feels like we're not on our own as well. There's been over 10 billion pounds raised for charities across the whole of the UK. You're walking around any town, any village, any community, and I'm pretty sure you'll see a sign there saying that this has been funded by national lottery funds, and it just makes you happy to see that.


Hi, I'm Mike Ostley, research agronomist here at the Carrington Research Extension Center. So today I'd like to take a few minutes with you to discuss one of the most complicated things we can do in our northern climates, and that's grow a cover crop with our full season corn crop. This has been something that more and more people have been interested in with the increased adoption of soil health practices. But because we have very few days without corn during the growing season, it's been a particular challenge. Now one of the main reasons that we're interested in this is not only for the soil health, but also for the livestock component. It's a great opportunity to have dual use of your acres, if you not only have that corn stover out in the field, but also some green nutritious material for the livestock to graze either late in the fall or early in the following spring. So one of the reasons why this is a challenge is, yeah, we have this great looking corn crop out here, and this time of the year when a lot of cover crops are being seeded after spring wheat, we still have all this shade in our corn crop. And so we have a situation where there's not a lot of growth potential because of the corn that's overshadowing our cover crop. Now there's a few different things we do to try and overcome this. So one of the things that we've been looking at recently is the idea of a wide roll corn, where we plant our corn intentionally on something like a 60 inch spacing in order to allow more sunlight down into the canopy later on in the season. By doing that we get much better cover crop growth, and ideally we're still studying it, but we hopefully don't lose a lot of our corn yield, but we're still kind of on the beginning stages of that front, and so we'll find out as we go. So here's a little bit of what I'm talking about. So when we plant our cover crop in our corn, we're putting this in around early July or late June, kind of in the V4 to 6 leaf stage on the corn. And we're planting strips between each of our corn rows or some people might broadcast and hope for a rain. On the edge of the corn here at this time of the year, you have some really nice looking cover crops here, something along that looks like this. But in the center where you don't get as much sunlight, we only have something that looks like this. If you have a nice early fall, we may be able to harvest the corn and still get a lot of late season growth, because the turnips and radishes and some of the other cool season crops will still grow pretty late in the fall, even after the first freezes. But if we have a late fall, like 2019, you're never going to get anything like this in the middle of your standard spacing corn crop. So there's a few alternatives. We can either move our grazing from the fall to the spring. So in that case, we'd be focusing on having winter annuals as our cover crop. So our winter annuals, like our rye or maybe even something like a camelina, will still come back to you the following spring, even if it's seeded in the middle of the summer, the prior summer. And so this would be a great opportunity to still be able to take advantage of the cover crop growth the corn year, but grazing the cattle the following season prior to your next crop. Right now that's looking like one of the more likely scenarios. However, if you do have a wide row corn, allowing that more of that sunlight to get to the bottom is very realistic that you should be able to have a full season corn crop. And then maybe November still have a window to graze, which should be a lot more biomass than you would get with a narrow row. So I hope this helps to clarify some of what we're doing out here and what some of the options might be for cover crops and corn up here in the Northern Plains. Thank you for your time today.