Here at NoSQL now 2013. The focus of our panel today is going to be on Enterprise NoSQL, getting NoSQL into the enterprise. What's it going to take? So we'll look at that. We'll look at what's next in the environment. I'm William McKnight from McKnight Consulting Group. I'll be your moderator and I'm loud, so I'll move that down. We have three panelists here today. I'm going to let them introduce themselves. Dave, you want to introduce yourself? Use the microphone, please. Rosenthal. Dave Rosenthal, one of the co-founders and engineers at FoundationDB. Which launched this week? Which launched yesterday in this room? My name's Dave Rubin and I'm the director of development for the Oracle NoSQL database group. I'm Pete Avan. I'm a senior engineer at MarkLogic. Great. OK, so I'm going to ask them some questions. We're going to try to get answers from all of them on the questions and as we get towards the end, I'll ask you all if you have some questions for them. So I'll be thinking of your questions as well. But I'll start with something really simple. Cloudera CEO, Mike Olson, made some bold comments recently, Dindi, about Hadoop replacing ETL in the data warehouse. So my question for the panelists, start with Dave here. Do you agree? I guess absolutely I do agree. I don't know how bold the statement that is. I think Hadoop is definitely replacing the traditional data warehouse. And I talk to people every day who are thinking about replacing their data warehouse with a Hadoop cluster. I was talking with somebody earlier who said there was sort of an air of inevitability about Hadoop. And I said, I think you said it just right. Nobody's using it yet. But when you ask somebody if they think they're probably going to be using it, they all say yes. In terms of ETL tools, I think there'll still be ETL tools. ETL is a batch processing function. I see somebody building that on top of Hadoop. Sort of is a piece on top of the Hadoop stack. I don't know if Hadoop replaces ETL tools. But that's my quick take. Dave? So I disagree a bit. So I think there's a really interesting place in the space for Hadoop. And in my view, it's all about scale. And so prior Oracle, I actually built a big data system in online advertising. And when you're asked to run a simple query, like select count distinct group by month over 400, 500, 700 terabytes of data, Hadoop is your only answer. But there are other requirements that the business asks you to do. Can I get real-time reporting for our users? They want to know revenue. They want to know how campaigns are performing. They want to know what audience segments are performing and not. Hadoop cannot do interactive reporting today. So down the road, maybe interactive query at scale, ad hoc query at scale with the advent of maybe Impala, Dremel, those things look interesting. But today, I believe Hadoop has an interesting place but not replacing the data warehouse. Yeah, I think it's a place with ETL, maybe, and flexibility with the information with Hadoop. And maybe that's the vision for Hadoop. But you're going to need a database to manage millions and billions of objects. And when you're working with a more operational store, you're going to want a database. You're going to be able to drill down and look at that information and validate what data is driving the decisions you're making. Hadoop's not there, right? It's a distributed file system with a compute layer. There's some flexibility, but I'm not quite sure. It's a bold statement, but it's a bold vision, and maybe it'll get there, but it's not there yet. There you go. OK, so guys, we've been through data revolutions before Walmart had their first terabyte database 20 years ago, first one. So why is NoSQL any different from these revolutions, Dave? From Oracle? Oracle, Dave? Hello? There's an eventual consistency problem. I don't think it is different. Like many other technologies, some things are driven out of pure necessity. NoSQL was driven out of the necessity to solve problems at scale, basically in online advertising, either search or online display, where no other solution could actually solve that problem. I think it's growing in acceptance and widening its scope to different industries, but I think the problem domain remains very similar. OK. Dave, repeat anything? Are you assuming those data revolutions succeeded or failed? I don't know if it was skeptically. No, no. Yeah, so I think that the thing that's driving NoSQL is two big things. Number one, the economics of traditional databases. And number two, being able to scale up data to huge, huge sizes. And that's not a revolution that I don't think that it's turning around. So I think we're on a long path with it. Yeah, I don't think it's a revolution so much as an evolution. What we see a lot of is there's a lot of the whole story, about 80% of the world's data is unstructured, and now that number is growing. In this network world, the amount of information that's just drastically, the velocity and the volume, is just ridiculous. And it's about gaining insight and driving decisions from that data and being able to handle that data. And you can't quickly get it into rows and columns. And the tools don't exist for that. And you want flexibility in your models. You want flexibility in your system to be able to ingest that information and start doing something with it. And this has been going on for a while, right? Mongo started out a double click. People actually were trying to solve these problems, and they couldn't solve them with rows and columns. They couldn't do it with a scale-up solution. They needed a scale-out solution. And so I think about that also with terms of MarkLogic. I mean, we've been doing this. We started solving these problems like 10 years ago. MarkLogic was founded a couple years after that. It was the first release in solving these types of problems that don't fit well in the rows and columns, more unstructured information, giving it semi-structured information. Yeah, so I think it's an evolution. So obviously the questions are coming from the perspective of what exists in the enterprise today, which is a lot of rows and columns, a lot of tables. But I think you started to answer my next question. If you want to hold that mic, why now? What forces have created our current data-rich environment and the tools now available to process, share, and analyze all this information? Why now? Why not? It's an evolution, right? It's the evolution, but I think now you have this. It's kind of exciting to see that everything is happening with all these different databases. But basically people have been struggling to do something with this information have not been able to, and they've hit the wall. And so now they're exploring alternatives. And people are driven by necessity, like David mentioned, it's not something like, oh, let's create some WYSIBang new technology. They're actually trying to leverage the data they have they're not able to use today. And so there's different, depending on what problems you're trying to solve, it's more like right tool, right job. Because in the no-sequel space under that umbrella, there's quite an array of different technologies, from Key Value Store to a document-oriented database like MarkLogic to et cetera. And so it's not just like, oh, no-sequels this, one size fits all type solution. It's what problem are you trying to solve, right? And so, and people are trying to solve those problems. And what we see at MarkLogic, it's all about the, you have your operational data, but now more and more it's, I have this world of information. If I'm gonna have data-driven decisions, I wanna be able to take in those external data sources and marry that with my operational data to be able to do more with the data that I have. And within the data I have, I have, I'm sitting on content that I don't even know what's there, I wanna look at that. And then I also wanna pull in these external sources and start to remix and analyze and, and so that's, I think that's where we're at. Dave. So, I think there are a few things going on really converging.  So people have come up with interesting algorithms to analyze unstructured data where that, those kind of algorithms could not really be computed on hardware 10 years ago, right? So that's going on. So being able to take in an unstructured stream of text like a Twitter feed, you know, a fire hose, open that up. If you're a financial services guy, do sentiment analysis on that text and understand what's going on from a macroeconomic perspective, you know, wasn't, you couldn't even contemplate doing that, you know, five years ago, 10 years ago. So I think that's going on. I think the other piece is that people have looked at, looked at these schemalists types of stores and have found great use for them.  But clearly the ability to be able to, you know, model your store for your application or change it very quickly without going through that very, very painful, you know, relational modeling exercise, whether it's, you know, a star schema or a normalized model, it's painful. So there's that going on. And I think, you know, there's a lot more solutions out there in the market today. And so people are just looking at these things out there and they're making a choice in terms of what problem they're trying to solve. But I think you have a number of different things converging. Okay, all right. If you like. No, no, let's pass through to the next one. Okay, do it. Very well. So let's look forward a little bit now. So we've heard about some of the use cases of NoSQL here this week. What are some aspirational use cases of NoSQL for the enterprise that we may or may not be thinking about that we probably should be in our enterprises? So I've been trying to figure out at this conference whether NoSQL includes analytics or is mostly focused around operation or whether it sort of implies operational or not. I still don't know. Maybe somebody can tell me after this they've figured it out. But I think the obvious aspirational use case for the analytics side of the equation is the real-time analytics. And sort of what you're getting into is the limitations of Hadoop. Hadoop is a batch system. I hear people about trying to like micro batch Hadoop, get that batch runtime down to 10 minutes or something. It just sounds ridiculous, right? So I don't know how it's gonna happen. I think you gave a couple of examples of companies that are pursuing that. But I think the real aspirational analytics use case is getting to that real-time goal. I think in terms of the operational stores, to me the aspiration is to try to be able to actually consolidate the many disparate NoSQL databases and to some extent eventually traditional databases that companies are now using down into one engine. And I talk to companies every day that don't just have one NoSQL database, they have four or five or six. They've adopted all of these databases, each one providing them a different data model. But I think a lot of them have an aspiration not to have to operate and run six databases to get that capability for their developers. Anybody else? Go ahead, Dave. So I'm gonna give you two answers. One is a selfish answer and it's something I've been wanting for years and that is search over the enterprise repository of unstructured data. You know, whether that's PDF, Word, source control system, I wanna know, give me everything that has to do with network latency that we've done in the last year. I can't answer that question today. I'd be surprised if any of you could answer that question in the data that you have in your enterprise today. It's all unstructured, it's all raw text and it's all basically unsearchable. So that's one thing I'd like to see. I think ultimately when you look at real, useful use cases, I think really ad hoc query at scale is probably, you know, is the holy grail of where NoSQL is going. Aspirational use cases. Yeah, it's aspirational, that's kind of interesting because Mark Lodzik, we do have search at scale over this unstructured information and Mark Lodzik, we're investing heavily in semantics and that's not just because we think semantics is cool, that's actually what people are asking for. It turns out there's facts within the data, there's facts about the data and there's the facts of the world and if I can take my unstructured data, throw it into a repository and do this search over petabytes of information but then I can model relationships among that data, what kind of new powerful searches are gonna help me to make those data-driven decisions in my enterprise. And so that's something we're very excited about. As well as the real-time analytics. And so aspirational use cases I think are also getting tools on top of the NoSQL, a lot of the NoSQL stuff that's out there is very infrastructure-y right now, right? And your standard BI tools are still bolt-ons to a relational model. And so like Tableau, Cognos, and some others have started to make strides in converting their applications on top of NoSQL repositories, which is very exciting. Their first pass, I noticed with one, when you have a real-time repository but you have some single-client memory-bound application that basically takes in information and you're looking at a static view, keeping those applications in sync takes a little update on the part of those applications, right? So I like the real-time analytics as well as the semantics. Okay, there we go, there's some good categories there. Now, we know that it's hard to replace something that's in place and working in the organization but are there any cases inside of organizations that are using traditional data management infrastructure that you see moving whole-scale to NoSQL in the next few years? Start with Dave from Oracle, if you have anything. I didn't catch it. All right, so I think there's a couple of things where, or a couple of use cases where people are moving existing functionality or business solutions. And it's around storing of image data. So when you look at applications, for example, in security, people that are doing image recognition, fingerprint, facial recognition, there's metadata associated with that data that they clearly wanna be able to look up very, very quickly.  So there are cases like that where we see people getting interested in using NoSQL to do that kind of stuff, especially when they're looking at doing scale out. Peter, Dave. Yeah, I'm gonna say that I see this all the time and maybe it's just by virtue of the people that come and talk to us, but every week I talk to people that are considering replacing an existing database infrastructure with NoSQL databases, absolutely. Somebody once came to me and said, well, isn't this kind of thing only gonna happen when somebody's rearchitecting their whole software because isn't that a big transition? And I sort of said, well, how often do you think a product needs to be rearchitected every five or 10 years, something like that to stay relevant? And how long does the rearchitecture of a 10 year old product take? Couple years, right? So by my math, at any given time, probably about 20 or 30% of all the software projects in the world are being rearchitected, right? And that's a lot of opportunities for people to think about their data infrastructure. And from my perspective of the people that I talk to, virtually 100% of developers are considering NoSQL as at least part of the infrastructure for their next generation application. Good point. Pete, anything? Yeah, I don't think rows and columns are going anywhere and so maybe it's just the playgrounds that I'm existing that I see as it's more augmenting that structural, what exists in the operational structured world with the unstructured, and basically augmenting those systems gracefully and providing interfaces for developers to quickly, like the agility, basically adapt and integrate with those systems. It depends on the use case, but I by no means think that NoSQL is a rip and replace for existing relational databases, right? And so it depends on the use case and the particular NoSQL being considered. But generally, there's a reason why those rows and columns, if it works well in rows and columns, keep it by all means in rows and columns. But if NoSQL's here to solve the problems that you aren't able to solve with those traditional technologies. Okay, so if we're talking about the enterprise, we have to talk about IT. So where are these budget dollars coming from for NoSQL? Are they coming from the business or are they coming from IT? And just how are we dealing with IT in organizations that are fairly entrenched with their SQL? Pete, you wanna start with that? Oh, it's the, who's driving these decisions and dealing with IT. Okay. So driving these decisions, I think it depends on the vertical. You know, I think we see definitely technology, the IT is the early adopters and the technology is driving business decisions and we see IT more embracing these technologies and then they're delivering that message up to, you know, the C-suite who's been calling, you know, wants to know more about this information, you know, about these new technologies, but the purchasing is coming from the business. You gotta be able to show you provide value, really. You know, are you solving the problem? And even if I solve the problem, what's the actual business value? You need to be able to measure that. And if you can do that, right, then the purchase will come, right? That technology is worthy. Yeah, but then, you know, but it's not so much with IT, I was just trying to think about IT. Like I said, they're driving these because they're exploring these two technologies to solve these problems for the business that they're unable to solve today. Central IT? Central IT. Okay. Yeah, so that's what I see, you know, but then so that's like in Hollywood, that's what we see some of our customers in Hollywood IT is really pushing the adoption of these new technologies, right? And then in others, like in research or information publishing, that's where more of the business, right, is more familiar in making the decisions with regards to the NoSQL solutions that we see more, at least that's what I see a more logic. How about you guys? You know, it's interesting. If you would have asked me that, excuse me.  And the developers are looking at that question and trying to answer that, you know, over many hundreds of terabytes of data and are driving the decisions to buy the technology so they can actually meet the needs of the business. So roll the clock forward. Now we're actually seeing, you know, people at the CTO and CIO level that are basically saying to their development groups, you know, go out and figure out how you're gonna use NoSQL. So we're still seeing it being driven from the development folks who actually argue for the budget to solve the business problem, but we're now, we're also seeing it from the C level folks who are saying, you know, go figure out how to use this kind of stuff. Solution in search of a problem then. So the money is an interesting question, but money often follows pain and companies are willing to get rid of pain with money. And the pattern that I see a lot is that developers are really eager to adopt and work with NoSQL databases. It's an exciting thing for them. And oftentimes instead of trying to figure out sort of the way to integrate some existing, some new functionality into their existing data infrastructure, it seems easier to download and try this tool that, you know, is gonna give them a task queue and only has seven API calls in the whole library, right? Now, the problem is that you can download and learn 90% about a NoSQL database in a weekend in terms of the API, but you definitely cannot learn 90% about how to operate one in production in a weekend. So the pain comes from the operation side. From the ops people are left sort of holding the other end of this decision to adopt these NoSQL databases. And from my experience, the money flows from that. Okay, very good. All right, so in my talk this morning, I talked about some of the enablers of NoSQL in these enterprises. I talked about, you know, cloud computing, being accepted and having a strategy, open source being embraced and accepted into the organization. What else? I talked about data governance a little bit, but what are some of the things that organizations need to be doing? What sort of investments do they need to make in order to embrace NoSQL? I mean, like you said, you can just go out and download it, but how does that work its way into the enterprise and then into production? What are some of the things that should be in place? Anybody? Go for it. So I would say probably one of the most important things is you gotta give your developers time to get their hands on a NoSQL solution. So they can really understand what it's good for. There are trade-offs in every single development decision you make, right?  And like anything else, using NoSQL database has specific trade-offs. And so if you dive into a project and you get, you know, three quarters of the way into developing a solution and you figure out, wow, I'm building a financial solution and eventual consistency isn't gonna cut it. You know, you're in big trouble. So I would say that is the most important thing. Really understand what it is you're doing and allow the developers to get their hands on these things so they can figure it out. Do you think ultimately the rules that apply to what enterprises are accepting today will continue to apply to NoSQL? Like governance, you need governance, you need return on investment, you need these sorts of things? Well, you know, it's really hard to make a general statement on that, right? So there are different parts of the business that require different levels of governance, right? You know, if you're building a credit card processing application and you have PCI standards, you know, that's a different thing than building a solution that's gonna take, you know, open up a Twitter fire hose and compute sentiment analysis. So really it has to be looked at case by case. Pete, you wanna come in on this? Yeah, what was the question though? What's the investment? What investments should an organization have already made in order to more fully embrace NoSQL into their environment? Yeah, because what came to mind for me was like awareness. I don't know if you know, it's more like people, what I tend to see is people see the worlds through a certain pair of glasses and that's a relational scale up pair of glasses with certain tools. And so this awareness that there's a new set of tools that can actually solve the problems that they're trying to solve and some that are in production actually, some of those problems that they're trying to solve have been solved. You know, it's a lot of great stuff here. One of the aspirational cases just real quick was a cloud, which I didn't even think about. And then Greenfield development is always wonderful for a developer, because when you're in Trinch and IT and you got this legacy system built over 20 years and your job is to modify, it's a huge pain, right? So to go play with a new technology for a weekend is fun and all, but it gets back to awareness and business value, right? That's the driver. It's not like, hey, cool, you play with this new technology over the weekend. How is that gonna add to our bottom line, right? How is that gonna help us to extract information out of here and drive our business decisions? Or how can we leverage that data to, you know, are we leaving money on the table by losing that information? Will our algorithms that we have be better if we stop dropping certain feeds on the floor because we just can't model them fast enough to get into our relational database, right? So everyone's wondering. So Dave, you wanna come in on that one? Sure. So I'm thinking about what these guys are saying and I'm actually still sort of formulating an answer. Investments are, the investment question is a really tricky question. What do you invest in? I'll tell you one thing that I see people investing a lot in and that's benchmarking. Okay, I see a lot of people spending a lot of time benchmarking different NoSQL systems and from my personal experience, the benchmarks are usually wrong and they're usually not very useful, especially to the actual applicability of the business. So I wish I could tell you what to invest in but one thing that you might wanna think about not investing as much in is doing benchmarks because a lot of people that I talk to spend a lot of time doing that and from my perspective, it's of dubious value.  Yeah, it's one of these things where NoSQL databases, it's a new market. There's a huge number of dimensions of differentiation among the products and so it's easy to sort of latch on to some things that you can measure about them because everybody loves numbers but I think I like what you said a lot about investing and learning about the systems and not trying to sort of boil it down into this one's 10,000 or this one's 20,000. Right, right. Okay, so how about tools and products? We've been talking about sort of the demand side. What about the supply side where you guys are? What newer tools or products do you see as having the most impact on the NoSQL ecosystem? It can be your own, I suppose. It's our own. No, you can go ahead. I'll think about something else. I would say supporting or getting closer to supporting something like ANSI SQL. So really today most of the NoSQL products are API driven which is great. That typical workload of I got to get data in as fast as possible with single digit or double digit millisecond latencies and I got to scale that thing out is really wonderful. Now you start collecting data. The first thing to business ask you is, I need some aggregation. I need to answer some questions. I need to start analyzing the data along these three different dimensions aggregated by this timeframe. And so I think probably the most productive thing we can see on the side of the tools is some sort of structured query language. And I think the juries come in and the verdict is SQL. You can see this at Google with BigQuery and Spanner. You can see it with Impala, Dremel. And so I think that's probably gonna be the biggest productivity gain on the tool side. I know it's not a tool, but of course once you have SQL you can take all the tools that live on top of SQL today. And now your world is wildly different. So do we have to rename our industry once we get to SQL? Well, I think it's inevitable really. Unless somebody out there can come up with a better query language. And I think there's no mystery why SQL has been so successful for the last, I don't know, 20 to 30 years. There was an object database initiative, I think in the late 80s, early 90s to try to topple that. It failed for one reason or another, but it's hard to see another query language that's really gonna become as popular as SQL. I mean, look at Hive, right? Why did the Facebook guys actually put a SQL processor on top of Adoop, right? It's just people wanna talk SQL. It's what they know how to do. It's quite a nice language for doing the different aggregations and data access that you wanna do. So Pete, Dave says SQL. What do you say? Well, I was thinking Agility, you went multiple. I think for no SQL systems is a benefit to having multiple points of entry into your data. There's something to be said for standards, definitely. And so, but right tool, right job. We are a document database for, we're a database for trees, essentially hierarchical information. We model that information in XML, so we support Xquery. We also support SQL. SQL is a wonderful language for rows and columns or structured type information. These higher level REST APIs though, there's something to that too for, when you go into an IT organization and people are used to programming in a procedural language like Java.net, usually you embed SQL inside one of these languages. And then you ask them to learn something functional like Xquery, that's a mental leap. It's also a new tool set, right? And the enterprise, it's all about making information as actionable and operational as quickly as possible. So we tend to wanna, you wanna give people the interfaces for the languages and the idioms that they're currently, the tools they're currently using. And so having those types of interfaces, I think is, there's something, it's very useful. But like, there's something to be said for having that SQL interface, because like with MarkLogic, we can put Tableau, we can put Cognos, we can put Excel, anything with an ODBC driver now can connect to MarkLogic. But there is a lack of application, a dearth of lack of applications for no SQL databases, because we're doing something different. It's no longer rows and columns. It's structured and unstructured, right? And what's that, I don't know what the killer app is, generally in the enterprise, everyone has a different definition of what that app is. And so all apps are pretty much custom built. So then it becomes, well, how quickly can you custom build that application on top of this no SQL system? How quickly can we get to where we wanna be? And what's the level of investment, right? But so there's probably a wide range of tools and apps. So far it's been on the infrastructure and also building, I think Dave mentioned tool for ops, tools around monitoring and integration, making it easy to integrate in that infrastructure. But we're seeing more development in that arena as well. But yeah, that's all we got. Dave, last word. So I'm gonna disagree with Oracle Dave a little bit. And I think that one of the, I'm Foundation Dave, he's Oracle Dave, we decided before. I think that one of the biggest lasting legacies of this whole no SQL little adventure that we're on here is gonna be that developers want different APIs for different problems. Though the idea of polyglot persistence of wanting a simple API to do a task queue if you just have a simple task queue is something that's gonna be, I mean that's the legacy, I think. And I don't think that things are gonna converge to SQL. I think that SQL is gonna continue to exist because it is an excellent query language for a variety of tasks. But there's a huge demand for things other than SQL. I think that the most important thing coming to the no SQL world right now is transactions. I think that's been the biggest thing lacking from the no SQL products that have existed over the past few years. And I just spent four years at FoundationDB building a product to attack that problem. So I think it's the most important thing. All right, great. Well let me check in with you now that you've heard the panelists a little bit here. Do you have any questions or follow-up questions for them? Let's start right here. Factors inhibiting no SQL coming into enterprises and into the cloud in specially regards to security. Anybody wanna pick that one up? I think one of the keys is the lack of rich security models in the products today, right? You know, you just can't build that kind of stuff overnight. You know, the relational database folks they've been around for ever. You know, that technology and those features have been built over the last 25 years. You know, cell based security, you know, rich authorization and authentication models. So I think the no SQL products will get there. It's just gonna take time. Go ahead. I think you partially answered your own question as you asked it, which is the cloud is a big problem for sensitive information. And one of the big challenges, especially, I'll talk about a class of enterprises that we speak to sometimes ISVs who are developing an application. And one of the big challenges is that they've been used to delivering a product on premise to their customers. And the backend was part of that product. Well, they now have this challenge where half of their customers are saying, well, keep shipping it to us because we wanna own the information in our own data center. And other half of their customers are saying, could you host this for us? Cause, you know, we don't wanna host this ourselves. It's sort of annoying. And this is a huge challenge in terms of backend cause now you need to have a backend that can both run in the cloud in a multi-tenanted infrastructure and also run on premise at a customer. And so finding a good solution to that is a real challenge and I think it's preventing a lot of people from adopting it. Pete, unless you have something quick, I was gonna go on, go ahead. No, I was just thinking, some of these NoSQL solutions are relatively new and MarkLogic has been around for several years. So we've had transactions. We're asset compliant. We have role-based security, support fifths, 120 SSL. So I just wanna bring that to attention cause I just don't wanna be, we put everything into the NoSQL bucket, right? And it's not all are, have the same history or investment that's been put into them. And with regards to the cloud, there's a sensitivity of information. There's also the elasticity. That's also who's providing your cloud. I work with Amazon. It's not the fastest machine of cloud infrastructure out there, right? So there's different costs, not even the technology, but your Amazon investment that you're willing to make there, right? And yeah, I think what we're seeing at MarkLogic is people have been looking at the open source solutions but now they're coming to MarkLogic because we do have high availability, disaster recovery, backup. These are the things you expect out of an enterprise database, consistency and different ways to, when it comes to governance and compliance and your single source of the truth, there's certain things you expect in an enterprise database, right? And maybe that's why those other technologies aren't being adopted, right? Is NoSQL precursor to a whole lot more open source coming into the enterprise? Anybody?  So, and there's been a lot of open source. It's a very, very challenging business model, as you could imagine, right? You know, developers aren't cheap. People want to get paid and, you know, do the math, right? Yeah, I think it's inevitable. One thing that open source is great at doing is copying, right? When a product is laid out in front of you of what it should do and when a product stagnates, when a product category stagnates, I think that provides an especially big opportunity for open source to come in and catch up because the product roadmap's laid out. I think, and maybe I'm bad mouthing open source here, I think it has a little bit harder time innovating and there's a lot of innovation happening in open source, you know, SQL projects right now and wherever things go, though, whether it's closed source or open source or whatever, I do think it's inevitable that there'll be an open source effort to build a version of that functionality, but I think it might not be likely to happen until things have settled down a little bit more. Let's go with you. No, you, yeah, yeah, Spark, Sparkle. Anybody on Sparkle? It's a great, great idea. So, yeah, so actually, so the next release of Mark Logic, that's where we're investing into semantics and so that means the ability to load triples, model an RDF with a triple index and getting back to standards, one of the language we'll support is Sparkle 1.0 with some aspects of 1.1. So we don't have inference, that'll be some external service to start with, right?  Really excited, like I was saying, because right now you have separate triple stores and document databases, you don't have them all integrated in one system. So what kind of queries can you drive when you can look at those relationships in conjunction with full tech search over documents, right?  Are you a plant? Because that was a great question. I love it. No, but, oh, nice. So, but we're very excited about that, but Sparkle, but that's part of a, you know, we have a roadmap for the next three years where we're gonna be building where, this is a serious investment. We're gonna continue to invest in semantics over the next two years. Great. Right here. We touched on that a little bit. The question is about data warehousing, anything more to add, data warehousing as a real-time platform of analytics. Okay. Okay, how's it gonna morph? I would say yes or no, right? The perfect answer, it depends.  If you look at the Storm architecture, basically real-time streams-based, you know, parallel computation framework, very, very compelling for doing real-time analytics, you know, on a stream of data being produced, you know, by machine resources, right? So there's that, right?  Predictive analytics, easy to do, no, you gotta, you have to find a data scientist that really understands a lot about statistics and data analysis to make that happen. So I think strides are gonna be made. I think, especially on the real-time side, when you look at things like Storm, because I think that's very interesting, but I think on the other side, you know, taking data that's part of your enterprise, that is in your operational data stores, classic transaction data, and being able to run data warehouse type queries, and especially, you know, operational data store as well as any kind of NoSQL data stores, I think that's still gonna be around for quite some time. Okay, let's get one last question, one last question. Who's got a last question? Okay, any more DBAs? NoSQL systems right now are pretty hard to run, so the DBA is the person that's running the database, I think they'll be around for a long time. Guys. I'll actually answer your question with a question. Will we ever be able to solve the out-of-disk space error? DBAs will increasingly work for the cloud providers though. Pete. I think it's the evolution, I think there's always gonna be a need for DBAs, but what those DBAs do, you know, I think because we're seeing the evolution of the data warehouse. I was thinking of a paper by Greenclumb called, back to your question, it's called Mad Analytics, and it talks about the characteristics of a next generation data warehouse, and I think a lot of those NoSQL solutions have those attributes of magnetic, agile, and deep, and so if anyone Googles that paper, it's an excellent paper, I'm not sure I agree with their solution, but I like the characteristics they describe at the data warehouse, but the DBA job is not going anywhere, right? Someone still has to, I just think there's actually gonna be more work because what we're seeing is more right tool, right job. Enterprises are more willing to bring in, you know, multiple systems if there's benefit, right? So they'll be learning new systems, it won't just be one with multiple databases, now there'll be a mark logic, and an oracle, and a foundation, and a Mongo, and a whatever, you know, depending on what problem they're trying to solve, at least for the time being, till we evolve to the next level, right? And with that, we're out of time. Thank you all for your great questions and interest. Thank you, panelists.


This is the ScoreCast X XL Rig. It's the larger uprights for the taller athletes. If you're like me and Tim, you can stay with the standard one but this is 2.5 meters high rather than the 2.25. Still intakes the stem that's put together. Stim that's busy away spanner in now. We'll get this upright in no time at all. The structure and how it's put together exactly same as the standard rig. It actually comes in seven pieces so there's four pieces for your base and then three pieces for the upright. It's easy to take apart and put back up together but one thing we like about it is to separate into two parts. If you just at home you want to create some space lean the two up against the wall and then it's easy enough just to one at a time bring them out so it's easy enough to pick up and store away just slide the top bits in to make the upright. XL Rig up it's like I say 2.5 meters high. It gives you that high clearance on the pull-up bar or you're pulling and hanging and muscle up movements. We've got the human flag handles on the side for if you want to do any human flags. We can also hang rings off of that top plate so you've got all of your pull-ups, you dip on the rings. Tim's doing his nice body weight row. We've got a few other little attachments that we can also add on to make it a complete home gym solution. We've got the dip bar attachment which goes you can put it on the outside but then you've got to worry about that counter balance. If you put it on the inside of the rig we don't have to worry about that. Quick goes through safety bar on and then you've got that nice tail position for your bar dips. The final thing we've got is the jpegs. So we can put the jpegs in. If you've got a bar and you want to do a massive leg workout as well we can set that up there so you can get under that bar. You can do your squats. It is everything that you need in one simple portable home gym solution. The Scorecaster next rig. So this is the powder coated version which comes as standard. We can leave it outside for up to five years but at some point eventually if you're just going to leave it out in all rain or in any type of weather it will eventually rust so either take it down and bring it inside during bad periods of weather or we can do a galvanized version that you can leave outside for over 20 years and it will be absolutely good as gold. So there you have it. The Scorecaster next rig. A complete home gym solution. Everything that you need for a workout home is a favourite on this rig and on our website so get yourself kitted out now.


After about an hour of strenuous exercise, long-distance athletes can really start to deplete their glycogen stores, the body source of quick energy. Studies dating back to the 30s found that by hooking athletes on a treadmill up to an IV drip of sugar water, you could delay fatigue, and that drinking sugar water could help as well. So the sports supplement industry has come up with an array of energy, drinks, shots, gels, bars, chews, even sports jelly beans. Used, what a coincidence, by the Jelly Belly cycling team. In fact, the Jelly Belly candy company paid for a study that found that sports jelly beans could shave four or five seconds off of a 10K cycling trial compared to sports drinks or gels. But what about compared to raisins? Given that professional, collegiate, and recreational sporting events are supplements centered, heavily marketed to, athletes at all levels may be left with the impression that specially designed supplements are essential for optimal performance. Yet these products are often expensive, and cheaper natural foods that may provide a healthier alternative are often overlooked. There are low-cost, natural food products rich in carbs, such as sun-dried raisins, that have the potential to improve performance to a similar degree. Raisins are nutritious, convenient, typically palatable, and are a cost-effective source of concentrated carbohydrates. But do they work as well? They work exactly as well, exact same time. Same thing comparing raisins to a commercial sports gel. Same respiratory exchange. Same carbon-fat oxidation. Same energy expenditure. In fact, the only significant difference between raisins and jelly beans was in, quote-unquote, hedonic scores, pleasantness scoring, and raisins beat out the jelly beans. Compared to jelly beans with flavors like extreme watermelon, there was a greater preference for just raisin-flavored raisins.


I just dropped it in the private chat and we're not streaming we host DEFCON music and it takes a hot second to switch away from it. There we go and you guys are live. Welcome DEFCON safe mode. We're here at the DEFCON 28 safe mode do no harm a healthcare security conversation Q&A. If you haven't had an opportunity to check out our pre-recorded video it's on the media server and it's also on YouTube. We'll go for about an hour talking about some various topics that we'll probably touch on again tonight. Your opportunity here is to ask us questions in the Discord text channel and then see us live on Twitch. If you have any questions or anything go ahead and feel free to put them in there and we'll get them in no particular order. All right I'm going to kick it off and let everyone introduce themselves briefly and then we're going to go ahead and get to the first question. I'll start with myself and I'm Kowati. Replicant here hope everyone is safe and healthy. Hi my name is Ash Left. I'm an embedded software developer from Starfish Medical. I have no idea what order we're going in so I'm Jessica Wilkerson. I'm from the FDA and I'm a cyber policy advisor. Hey I'm Vidya. I am from Medcorp. We are a cyber security solution targeting medical devices and VL turned to you. I'm a goon. I am a patient, a hacker and a researcher and that's a lot of love of me. Back to you Kowati. Awesome so I'm accepting some fantastic questions tonight. We're going to go ahead and kick off the first one with a user called Razzie's which has a multi-part question but I think the the heart of it is at the end here and it's a question to everybody. Please don't be shy. Do we have the resources to adequately detect and respond to attacks against internet connected medical devices? So right now in the status quo can we even detect these attacks as they're happening specifically on the devices? Who wants to take it first? So I'll take a first pass. I think I think while there are several solutions that exist in the IoT space their application to healthcare is what creates this problematic scenario where we're relatively inadequately equipped to sufficiently respond right now. I think we think about what's the worst thing that could happen when you put security on a thermostat and we don't really think about well what's the worst that can happen when something happens to a medical device that's attached to a patient and so being cognizant that while there is a potential to detect us from a technical perspective there are ramifications for implementing that for real patient safety and actual patient care. I totally have to agree there and it becomes even more complicated when we talk about implantable so to do any detection on those devices is very hard and not very robust at the moment but the problem is as well as Lydia said the implementation of traditional cybersecurity tools in the healthcare space is problematic due to the fact that this is not your traditional cybersecurity space. Each hospital or healthcare institution places their own challenges. So it's almost about having something that caters to your hospital in your healthcare institution. Yeah I just want to chime in here for a second and say yeah I don't think many hospitals especially rural critical access hospitals under resource hospitals have robust detection methods. You'll be surprised how many of them lack intrusion detection systems or have good endpoint security you'd be surprised how many of them get regularly penetration tested or go through that entire process and just to plug again HHS and the FDA the 2017 Health and Human Services Task Force report estimated that a good majority of hospitals in the United States lack a single full-time security professional on staff. So with that's like a perfect storm to have a lot of attacks probably go undetected. The other little part of that I just want to just reflect on what V said I completely agree the home space is this complete black box. A lot of people who get medical devices especially in the era of COVID are given home monitoring equipment or they get an implantable and it has an accompanying base station and when they're discharged from the hospital they say go plug this into your router at home and there's really not a lot of work done to secure their home environments and we don't know much about that space it could be fine or it could be a nightmare I think what we lack is the ability to collect that information and then share it with people. I think that's a perfect point to raise around ownership right like who is actually responsible for securing this is it really reasonable to ask someone to go home and secure their home network not knowing their their technical skillset or whether that's really part of getting care right and then you think about hospitals like is their core competency really going to become to get cybersecurity and to the point of not even having resources we have to really think about how how we can think about where the ownership really needs to originate and what is most efficient and effective. One thing that I would say on that I think is like the cooking government person on this panel is I do have to plug all of the work that is happening at some of the the national level policy issues because I promise we are aware that these things are are issues and they are being worked on and so there are a lot of the industry bodies industry bodies government bodies and then bodies where the two come together that have documentation and just general conversations about this on a recurring basis so I know as little respect as everybody can have for the myriad best practices guides and other things that you know white papers that the government puts out that sometimes just sit on shelves and never get used but these are things that have been looked at and things like the medical device sectors joint security plan they've been looked at in for a health care delivery organization something called the hiccup and I if you paid me money right now I could not actually tell you what that acronym stands for but it's like the health care version hospital version of their best practices guide and in the hiccup in particular I want to say it is or maybe it's the JSP one of the two they actually have like it's separated out into what large organizations can do and then what small and medium sized organizations can do and they they do different things for both or recommend different things for both so this is an ongoing conversation that is happening at that national level but we we certainly don't claim to have all of those answers another thing that I think is important is we're moving into a thing where we want to do early detection and monitoring but most of the instances where there has been hospital breaches we've been left on the back foot right we needed to become aware of the breach before we could actually act on the breach so I don't think there is that early utilities within health care at the moment I can use an example of a health care attack in South Africa itself they only became aware of it once the systems were encrypted and once you know the damage has been done so often we have to rely on the cyber criminals to make us aware of their you know presence in our networks it would be awesome if we could detect it early and stop it I mean that is the golden standard that is the dream but I don't think we're there yet I think the real dream is is being proactive about this right and getting the security on the device so it doesn't even get to the point of there being a vulnerability that has to be detected if we if we can just reduce the end of things that need to be detected there's there's a better probability of actually coming up with it but but is that going to actually be you know something that we're going to get because I mean these devices last for long so inevitably there is going to be vulnerabilities always it's it's it's going to be there because as time progresses research is done and libraries become vulnerable or hard to become vulnerable also it's it's about the threat landscape right we're talking about risk here and just like everything in life there's nothing there's no such thing as risk free we can't just have a device that has security or doesn't there's always pros and cons we can't think of every possible thing so it's always it's it's this combination right of you know building the devices and making them as secure as we can in a reasonable amount of time with the resources we have and then once they're out into you know the healthcare ecosystem or the consumer ecosystem in the case of you know medical devices that maybe have to be on a home network there's so many pieces to that threat landscape right I like I'm not sure how we can fully secure it like there's no there's no perfectly secure network right I also want to throw one other wrinkle into this which is that there's actually some systemic systematic issues when it comes to healthcare usually in hospitals or what's traditionally been the paradigm is that the network is owned by IT and that's where all the security folks live and they're able to detect kind of the attacks that they're used to and they can control some of it but as soon as it comes to the medical device side of it they kind of wash their hands or are told don't go there that's bioengineering and they handle all the medical devices and so what you often have is a dearth of cyber security knowledge in the bioengineering space by no fault of their own they have a lot of other things they're fantastic people but what we have as a result of that is this something we call the discovery dilemma where if you don't have people thinking about these issues and you have malfunctioning medical devices you are unlikely to find a problem even if there was some random malware that spread over your network it affected some of your medical devices and as a consequence they stopped working unless you are thinking about that as a potential etiology of your problem you're not going to go and look the the one and two steps further to figure out if that was the issue so we talked a little bit before this panel started about the discipline of forensics you know I've asked many bioengineering organizations and groups in hospitals across the country and asked how many of them do that type of forensic work on malfunctioning devices that are just really out of the ordinary and none of them said that they did something we have to change so it's because the structure of these healthcare systems in such a way where there's a bright line between who handles medical devices and who handles the network and because those two groups don't commingle and have equitable cyber security knowledge where we might have this problem where this might be a very common thing but just we're not detecting it so I don't know if you want to move on to the next question here but one thing that I would point out on that that I've certainly experienced especially over the last couple of months is we have experienced I would say less and less medical device specific vulnerabilities where it's like the pacemaker has a vulnerability or the MRI has a vulnerability what we're now seeing are like higher level protocol vulnerabilities swen tooth was a great example there was the all of the the whole run of bluetooth vulnerabilities there was the ripple 20 there was the track which I think it was was the one part or track I tcpip was one part of ripple 20 or whatever it was but anyway these are things that are not medical device specific but well we're starting to see and it goes to the detection point and then I wanted to hit on something that that video had said um the the almost the nice thing that's happening about the fact that we have general it vulnerabilities that are now making their way into the medical device space or becoming relevant to the medical device space is other sectors are detecting them for us so even though there's maybe a little bit of a lag in detection and I don't know identification capability within the sector itself because that expertise may exist in other sectors and because we're all relying on the same software the software that's in a medical device is the same that's in a consumer product we're ending up still be notified and still being able to notify others because it's happening in other sectors but the other thing you know video had a had a good point about like we need to we need to lessen essentially the attack space and certainly acknowledging the fact that um being ash are both right like things are always going to be vulnerable another thing that we've noticed is that we have an increasing number of manufacturers who will tell them about vulnerabilities and especially with the bluetooth ones we have certain manufacturers who are like uh yeah but like the way we architected our system like our device doesn't trust the bluetooth that's built into the device so all of these bluetooth vulnerabilities don't impact us um and I think that that's the sort of what video is getting at is like we can build systems that are not impacted by certain types of vulnerabilities we just have to actually put in the work to do that but and I think sorry but um actually I had this conversation right security the the way that you change security and manufacturing ash was right we said it needs to be cheap and it needs to be easy and it needs to be accessible and it should be the easiest thing to do it shouldn't be the hardest thing to do that shouldn't be the most expensive thing to do and when I refer to expensive I don't necessarily um mean um in terms of cost it could be time it could be design it could be research it could be cpu it could be battery life right but the thing is at this point I don't think you know at that end of the pipeline that is very easily accessible for the smaller companies that is I'm not talking about your big companies but your smaller ones that are doing some innovative work all right we have a new question uh from judo and I think that this is going to be one we can all kind of chime in on uh it's I think well down to what are the steps that industry and I'm going to add regulators in the space are taking to formalize stricter risk and controls around device development that we use in patients uh they assume that a larger device manufacturer formalized basic security features just like the payment brands have done and set standards for the rest of the industry so um I think there's a lot that's already been done I just want to quickly reflect on that to say you know the conversations that we were having 10 years in the space we're asking for this and I feel like so much has changed between now and then where we we do have a lot more standards and I think that the industry has been moving forward in a lot of ways now we can still talk about some of the criticisms but what I really think is a lot of people that have been paying close attention will say yes we saw the promise problem legacy devices but we you know I said it in the video we don't have this year to my knowledge and there's still some time left um a story of a device manufacturer threatening to sue a security researcher for example or um something like that uh like want to cry happen in 2017 so um can everyone reflect on kind of what the industry has been doing and we can take it back a few years and give examples of some of the landmark documents that and guidance that has come from the industry as well as the regulation space well I can kick us off as the regulator um so I you know I think uh I was not at FDA let me purpose this by saying when some of these documents came out but um the two in 2014 I want to say there was uh you know one of the first drafts of a cyber security guidance particularly cyber security guidance came out at FDA uh a few years later there was the post market cyber security guidance and then in 2018 the most recent one um you know there was the the the new post market drafting this was the one that that talked about coordinated disclosure it talked about cyber security bill of materials it talked about a lot of um the advancements that had been made in the sector over the past couple of years from the time that the guidance had been most recently updated um and actually let me just really quickly for those of you who don't know guidance is essentially how FDA tells industry the best way to meet the regulations because if you try and read FDA's regulations they're pretty bare bones it's really much like you must do risk assessments but then like that that's the end of the sentence and it doesn't have any other detail so the guidance allows FDA to elaborate a little bit more on what precisely that means in given context um and so you know credit to to Suzanne Schwartz and Dr. South Carmody who um who's had worked on a lot of this when he was still with the agency uh you know they actually like fought the good fight and got cyber security specific guidance out from the FDA to industry and um I think you know obviously I'm biased here but that was just a hugely forward I think in many ways for um or not only us but for the industry yeah I think um to that point I think there the the conversation that's come from all of that initiative right you you have you have this relationship that I think didn't exist before even if we go back four or five years like there wasn't such a collaboration or willingness to have manufacturers work with security researchers work with security vendors right it was more of okay who can get what off of eBay what can we prove doesn't work and then try to try to get the best headline that we can get out of it and I think the if you listen to the the recorded version of this there's a lot of hope that we all have for for where this conversation is going and that there are going to be solutions coming out of this I think on on all sides that are actually sustainable and efficient so I can add a positive story so three years ago I had my first f1 came as a patient had some concerns about my own device it was as I would say beautifully broken and wonderfully flawed and I made Suzanne Schwartz at that one but except for that before coming to DEF CON I actually reached out to my manufacturer and had a discussion with them when I was met with legal you know it was like running into a wall it was a cold unloving situation three years later I'm working with that said manufacturer on things that I bought to them three years ago it took us since December to find the footing where you know the discussion is not incriminating it's not me pointing a thing that's not them pointing a thing that's finding a unison's and what they bring to the table what I bring to the table and working together but except for that I mean we had conversations I've been on the patient I don't know what the word is that you guys that Jessica that gave patients the opportunity to give their their experiences right so generally in three years that I've been involved in this we've made leaps and bounds is that necessarily enough yet no I don't think we should lose momentum I think we should kick ass and keep going I think we should push the envelope I think we should have the hard conversations and I don't think we should stop now yeah I I would agree I think that the the landscape is shifting really quickly when I started working on medical devices which was over five years ago already the technology has changed so much even in the last 15 years if you think about you know smartphones and you know 20 years ago I don't think most of us would have predicted that ubiquitousness of it and the impact that it would have on our daily lives and so you know historically medical devices were electromechanical and you know as medical device manufacturers we come we have a system about approaching things from a risk and from a safety perspective like how do we make sure that people are safe and a lot of medical devices didn't have software in them and then we started putting software in and then we started connecting that software to the internet and then all of a sudden the risk just blew open because now we can we have we have iot devices we have clinical iot devices we have medical iot devices we have implantable iot devices and so everything has been changing so fast and I think you know getting regulatory and and getting standards in place um takes time and you know especially from a software perspective how many standards exist like we have so many standards in software that are competing standards if everyone would just follow my standard it would all be fine but then there's 20 other people who also have a standard that they want everyone to follow and so it's you know it's tough it's like the wild wild west for software right and um so I've seen like a lot of these different groups coming together you know um obviously FDA is a part of it but you know you know most of our customers they want to serve American markets so we actually talk about FDA a lot every single day but there is the rest of the world that isn't governed by the FDA and so there's a lot of different bodies working together and yeah I see it as hopeful I know IEEE is working on international standards for clinical iot and medical devices and interoperability and yeah I think people are working on it but I don't think we're there yet certainly I actually have to jump on that because if I think if I don't one of my bosses is going to find me at home but um to the point about international regulation there is a body called the international medical device regulators forum and speaking of you can never get government to do anything in any reasonable amount of time that's just sort of a default law of the universe but that group which is a group of it's like a nested group of regulators so you would have thought that this would have taken significantly longer than it did but um they actually just came out with a cybersecurity guidance document and so um you know that's that's another thing that's been really valuable for us uh and and I would actually encourage everybody to go look at it it's got a really fun definition of legacy that uh you know you all might have thoughts on so anyway I am dear up guidance the only thing to add is that we're never going to be done right so as much as we've progressed I and absolutely take a moment to recognize the accomplishments here there's certainly a path that we need to continue pursuing in order to even maintain a baseline of security across the board I think what I want to see and this this would make me like as a patient super happy is if we had uh in the manufacturing we had security standards for secure manufacturing because it's one thing having a cyber security framework right or a guidance in terms of that but it's a different thing and having a standard for you know how we are going to securely manufacture these devices because often they're none you know hardware is the constraint because when we talk medical devices we are designing you know firmware to run on hardware um so I think we need to you know completely start at the beginning um when they conceptualize the requirements is that we need to build security in there right it shouldn't be this you know after the fact kind of like let's put it on the end and you know hope for the rest and that's you know normally done because we didn't consider it in the beginning and it hasn't been in that best you know that practice but there are companies doing that and that is changing um but you know it's also been an industry that's been around for a very very long time and that's got some bad behavior and bad things that it's been doing but it's also got good things right I mean how many lives have medical devices saved I mean we're always focusing on the negative I mean I wouldn't be yeah I would have died at 19 I'm now much older I'm not going to tell you my age but you know so I mean so V are you saying are you saying push left is that the summary of what you're saying yes I think you and I said let's push all the way left right let's not do this at the end um I've worked with the developers and the engineers right you know everyone thinks like security is the answer you want to change this shit get them to start doing with security in mind and you know you're not going to deal with future problems I literally have found that security and forensics forms part of hardware engineering and forms part of software development and firmware design you know it just there's this synergy when we start working as a team and as a collective instead of you know silos and I know these people that are going to argue the silos are good because everyone needs to focus on the independent thing but when we bring different minds together that's when we start seeing things come together in a different way and I think that's going to be the answer is working together instead of against each other and you know that multidisciplinary approach because we're not all experts I mean I shouldn't be doing any hardware design it will break you know that's from my spirituality but I can tell you from a security in a logging perspective or forensics perspective what could work um so I think we need to get that multidisciplinary approach all right um Iona go ahead and get our next question kicked off by Ken in the DEFCON Discord it says there's been a lot of progress in the last five years I think we've all talked about the optimism that we've had you know guarded optimism but there's so problems with smaller device manufacturers there's problems with smaller hospitals how do we motivate them how do we get them to do the right thing and I think Ash on the recorded talk you had you know talked about how that's a big part of medical device development are the kind of these smaller boutique consultant shops like your own yeah I wish I had a I think that's a fantastic question I have that question every day um how do we how do we motivate them how do we make the how do we get there right because we've got constraints on lots of sides and you know at the end of the day you know most of the people who are working on this we are really passionate about the the end patients especially you know some of the smaller shops like everyone that I work with actually cares about the patients um and a lot of our clients are doctors who have an idea for some medical device that they think will save lives and they only have so much budget so yeah how do we motivate them I think I think everyone is motivated um or at least for for some of the smaller ones um we are motivated how to actually get there how to implement how to how to communicate to clients and even to other engineers about what the risks are with software because sometimes again we're so used to see thinking about things as you know um electromechanical device you know we think about safety is like the lock on the door you know people who aren't in software all the time people who don't come from a security background they don't think about all the creative ways they just kind of think well this is secure we put a lock on it um you know it's like I put my phone into a little like box and then I put a lock on it how could it not be secure well because I can connect to your phone through wifi and then exploit it somehow right like so it's not actually immediately obvious to a lot of people so then when you're trying to you know try to set up a project and talking about budget and how much time we allocate and you know what what sort of security things do we care about and you know everything is pulled in all these directions so I I don't have an answer for how to balance all of those perfectly I think we're trying to do that every single day um but I think it's a really great question and I think we should keep asking it um yeah that's what I have to say I think the phones are a great analogy sorry just two seconds here right I think the the fact that you bring up this notion of okay people understand security on a phone and they say hey I am comfortable that my iPhone is going to do this and I know B&I had this conversation but how do we translate that into the expectation that a medical device will be secured and of itself independent of how it's going to connect to whatever the clinical care is and I think part of it is we have to be fair to all these clinical innovations that are really trying to bring something to the patient experience and change it for the better and and not try to force them to become cybersecurity experts that there has to be a way for us to to enable them to get there without having to become pros at cybersecurity I will say though and I have to say it sorry everyone I just I think I disappeared for a moment my computer decided it didn't want to be part of this panel anymore um so that there is there is pre-market guidance and the current version is is not nearly as detailed as I think you know some of the things that we're working on now but um you know to get on the market medical devices have to go through the FDA in most cases you I'm not going into the class system none of us need that but um you know if they're if they're gonna have software and other things like that chances are they have to get pre-market approval to even go on the market and and so to a certain extent they have to demonstrate through FDA whatever the device is however small the manufacturer is that they have provided reasonable assurance of safety and effectiveness and we check for for cybersecurity concerns I mean I think I made some clever comment about Matt being my elementary school best friend but like Matt is the lead um for a lot of the cybersecurity reviewers at FDA and like we're like I said in the recording like we're getting better at that every day but this is something where you know FDA is rightfully by statute by law the gatekeeper of what can and can't be put on the market and and so I think you know for better or worse you can you can argue whether or not it's good or bad that we're doing what we're doing um the smaller manufacturers still have to go through us um so there is still going to be that part of it I also just want to say we had talked multiple times about how you have to make security um the easy thing to do and for some of these rural access hospitals smaller organizations are just not going to be able to do the job right and so I think a collaboration we said this a couple times there shouldn't be a competitive advantage in cybersecurity when it comes to hospitals right so you shouldn't see a billboard as you drive down the street that says you know when you have your heart attack come to hospital a because we didn't get hacked last year like hospital b did right so that shouldn't be like that and I think luckily there's been a lot of momentum towards collaboration information sharing um uh through a lot of these organizations and bodies we've already talked about but you know what was an alien concept to hospitals five years ago is probably not that crazy now which is hey if we get hit with ransomware you know we're going to go through our protocol of instant response then guess what I'm going to call the CISO at competitor hospital b and let them know this happened to us and did you see anything collaborate uh what normally was a very uh we're not going to share we're not going to talk we're not going to collaborate in the space has clearly gone out the window uh for some of the more forward thinking organizations now let's take that information for example all the great work that larger organizations have done like Mayo at looking at particularly secure devices compared to others share that information with other rural hospitals so they don't have to go and reproduce all that work from scratch and they're doing that uh being able to share in that space is so so key so it's not reinvent the wheel at every single hospital let's not try to build gigantic teams that do vulnerability testing penetration testing on every medical device and reproduce that work time and time again instead it's a community we got to recognize that it doesn't follow the same competition that you normally would where you're not sharing this is a different world this is about patient safety now we all have to be on the same page yeah but I mean I'm going to say something that's going to be very unpopular right if you take a medical device um functional requirement is not security functional requirement is keeping someone alive or healthcare right we need to understand that because we are super imposing security onto these devices then that's not their main functionality yes as they advance they are connected and they need to be secured but I don't think we should solely focus on having such secure devices that might just never see the you know market they might never save their life you know that one patient's life that they can change I think we should never lose sight of what the purpose of these devices are when we try and attempt you know the security implementations onto them awesome all right we're going to go to the next uh question here uh this is from from uh synapses questions about where the industry is going uh insofar as what are the new uh and perhaps we can even talk about the pre-market guidance in this regard like what types of requirements are we going to be putting into into some of these new devices as they go for approval uh to push that forward to perhaps of course always respect that these devices are security or sorry that was not a slip I promise their medical device is meant to help patients they have to be secure are we going to be deploying things like stack protection ASLR DP and other things like modern like language changes that have better uh security kind of baked in from the start are we seeing trends in this space towards utilizing more secure tools and development of these medical devices so that they are less prone to some of the scary vulnerabilities we've seen in legacy devices so I think uh you called me out there a little bit with the cyber security guidance so um yes uh you know I think I had made the comment earlier that the FDA the actual regulation parts FDA's regulations are pretty high level I really would be amused if any of you were like that motivated to go look at literally the code for federal regulations to go look at our regulations they're very dry um and then you have guidance and then you have what companies are actually doing to meet the guidance or to meet the regulations and I think from the FDA standpoint it would help no one if we picked like the one way of doing security to rule them all and then told all the manufacturers that they had to do that one thing so instead essentially what we've we have been communicating to them and we continue to work with manufacturers on continue to work with our own internal processes on is essentially saying like you need to have a secure architecture we're not going to tell you what the architecture looks like but you need to it needs to be secure and you need to be able to produce the documentation to us that proves and validates that it's secure um you need to have security controls that are the same thing you need to be able to not only have them you need to prove that they work um and so and then but but then at that point it's sort of like and it's now the ball is in your courts manufacturers you tell us what your security controls are you tell us what your secure architecture is um because I think like going down into the the slr level and things like that you know to to these and ashes and and vidyas and some of the other points that have been raised each individual medical device is very unique you know the the things that you're going to do to secure a drug infusion pump are not going to be the things that you're going to do to secure an MRI and so we do need to have that flexibility built into the regulations built into the guidance so that um you know we can we can allow the the manufacturers to figure out you know what is the best most efficient way to secure the device to these point that we don't you know impose such um rigid constraints on it that you know certain certain devices just can never meet them you know maybe they can do something else so that's what I would say didn't mean to call you out just think it's such a great segue all right who's up on that question so I mean you had mentioned ash like how much things are changing and do you see trends towards more kind of baking in these controls from the start and do you see exciting trends emerging um I mean I think I agree with what Jessica said I think um I I'm not sure you know I come from uh software and engineering development background and I'm learning more and more about security but uh so traditional security controls I don't have a huge history with that but I think every device is really different and from security perspective or from a safety perspective um we make sure that the device is secure so that it provides safety so that patient outcomes are are successful are positive right um every device is different I think that more people are learning more people are like me there's more engineers who care about the security part and you know obviously with FDA and guidances and um with regulatory we have to meet standards um that's absolutely correct how we choose to meet those standards is up to each manufacturer so you know we have an internal process of how we approach that the end of the day um you know it's it's literally different every time we make a device so we're we're different too because we uh our consulting firm so we make a lot of different devices we have no we have none of our own in-house products that we are continually sort of like building um so every every device is unique and it has a unique solution has unique architecture and as technology is changing as Wi-Fi is in everything now more and more and more uh yeah I mean we're using different approaches yes we have to make more uh we have to have a bigger focus on security because of that but um but I think it's just we're just following the trends uh with technology itself yeah the only thing I'd add there is I think there are um there are a variety of ways to approach it and we have to absolutely keep patient care at the the center of it to make sure we don't introduce new problems by trying to solve the security problem um but I I think there's there's a fundamental expectation now that there's security built into this and it's not something that can be built on afterwards where we're seeing the narrative more and more that it has to be part of the initial requirements discussion it has to be part of how the device is going to connect how it's going to be patched if we don't think about it from the onset of these fundamental architecture decisions I think we'll we'll continue to have this problem of trying to to solve the quick regulation question that came back and and try to get it through so it so it can get to the hospitals no definitely I don't think we can get to a granular level that we can start dictating architecture because again those things depend on the type of device and and the requirements that you need the device to fulfill um but I am looking forward to better guidances and yes I was that boring person that went and read that because I wanted to understand our pre-market post-market book but um after some discussions with Jessica and Suzanne I'm excited to see the new ones coming out in the more um robust nature of them because we are moving towards a place where we are going to be enforcing better manufacturing so let's add transition to our next question into a little bit outside the medical device space and really more into the hospital uh Ken brings up another question about who's who's keeping the hospitals um in line meaning uh the joint commission is uh and there's a couple other bodies are the ones that accredit hospitals so when you go buy a hospital and they can be a hospital some type of regulatory body has to make sure that they are uh to meet certain standards the joint commission is probably the most famous one of those and so a lot of what we talk about on the hospital side actually would never be FDA jurisdiction those are medical devices and their approval this is the joint commission's probably space or other organizations that accredit hospitals so the question that Ken poses is you know has the joint commission jumped in here and how do we make sure that hospitals do what they're supposed to do we've talked at length about how we can do better with medical devices but I think we need to start moving also into the space of like hospitals need to be held accountable in some cases and in accordance to what they're able to do but we need to move that needle as well because we've all heard stories of devices with certain security controls from the manufacturers that get turned off or aren't deployed appropriately or deployed on horribly secured networks at hospitals and so part of the responsibility has to be this multiple people right how do we get hospitals to do the right thing knowing that especially in times like COVID then it's not going to be the place they're going to be putting their resources in right they're worried about their ventilators in the ICU they're not necessarily worried about this problem right now is there a way that we can move forward and make sure hospitals do the right thing to be flippant but like do we I feel like there's there's such a burden here that we're trying to historically pass on two hospitals that that maybe was unfair right like yes hospitals if they're told and then a user guy that they need to have certain controls in place and they didn't sure there's there's something there that there's a flaw there on the plan but at the same time is it a fair expectation for a device vendor free device vendor to say okay we want you to have these 32 different controls in place times the hospital having a bunch of different vendors like I think there's a balance there so I think yes you have to enable hospitals to be more successful in having some of the basic security in place but I think it's also somewhat unfair to place the onus on them so much yeah I mean I think I could just share something you know we talk all the time about shared responsibility and I actually question thank you so much for pointing out that FDA does not actually regulate hospitals because I think sometimes people forget that we actually have a very small slice of the of the pie and we like to think we're very effective whether or not we are as I suppose up for for debate for from others but you know that's not something we have any control over but you know there there are multiple federal agencies and state-level agencies and actually that's part of the problem is the sheer number of regulators who are active in this space who may or may not have harmonized regulatory requirements that hospitals are expected to meet you know that's that's challenging enough as it is but I something that Vidya said I wanted to hit on because we were one of these you know public private partnership bodies that I'm a part of we are recently talking and one of the core sysos was like I have a spreadsheet of 400 different websites that I have to continually check to figure out what updates I need to make to all of the different devices that I have and it's not something where that you know they're not always getting active notification from the vendors as like you should go check one of these 400 websites and there's just it seems you know almost an infinite variety of procedures that each individual manufacturer can try and impose on on a given hospital and that just can add up so fast um so you know in this in that sense like there are certainly things that we can do better um I don't know necessarily that um punishment is is always the way to go with that I you know I don't know how many of you know about the the office civil rights wall of shame um but literally if you have a breach of more than 500 people or at least this was the way that it was a couple of years ago you go up on a portion of HHS's website with like your name and how many records were breached and how you got breached and etc etc um and it's not exactly the most helpful way to do it because you also get fined in that case um which is then that's that's resources that you have now lost that you then cannot put towards your security because you just had to pay it you know just somebody else and so um you know there's I think in this is these are conversations that happen at the federal government level all the time is like how do we how do we regulate holistically how do we regulate in such a way that um you know we are we are helping the industry move forward rather than um you know continuing to perpetuate some of the problems that we see I want to touch on you because this is something that is really near and dear to my heart is is hospitals and the one thing that I've realized is we've created this the shameful thing of finger pointing when a hospital gets breached right um when we make it this negative connotation that they don't want to come out and say hey we've been breached we need help we've made it impossible for them to turn to the security industry and ask for help because we shame them when they have been breached right when they are having to deal with medical care and then sophisticated attacks from cyber criminals you know and on top of that implementing different patches and different software updates with different systems and you know how they're going to connect um all these things up I think we are expecting them to take ownership and accountability for things that aren't theirs to do and then still we we punish them for not being able to do it it's very negative connotation to you know we need help um I uh I agree I don't think uh I don't think punishment is necessarily the right way but I think that they're they're not being punished the only hammer that really they pay attention to is HIPAA because it's tied to a fine and an embarrassment on a wall um there isn't really for example rural hospitals uh A's gets owned and there isn't a HIPAA breach no one knows about that and there is likely going to be little remediation efforts uh probably because they've been owned for years and they don't even know what I'm trying to say is that I don't think we should shame hospitals but the status quo is likely um is untenable for a long term because there is an increasing attack surface there's less resources being put into this and then on top of it there's just it's going to have to be looked at in some way because we can't have the most secure medical devices on the planet with phenomenal protections and then then put on networks that no one cares about um or that no one's doing a good job at securing because there isn't that pressure so how do you incentivize the hospitals do you say for example we're going to give you 0.25 percent more on your Medicaid reimbursement if you meet these cybersecurity requirements right are you going to say we're going to cut your reimbursement from uh from insurance claims or this if you don't meet these requirements it's a joint commission going to say hey we're not going to credit your hospital unless you meet these requirements these are types of questions we should be talking about whether or not it's a a carrot or a stick this is how we're going to have to figure it out but what I do know is you can look at many different health care organizations across the globe and know that that's just we can't keep going in that realm because the patient information is just too valuable and the risks of compromises to these networks are just too great for patient safety and then so we have to have that conversation sometime um again I don't want to shame anybody but there's we have to do better in all fronts of this because if you just secure one side our patients are still vulnerable on the other side it's really got to circle the wagons everyone's got to get on board I think I think to your point right I think cybersecurity in healthcare was defined as HIPAA for the longest period of time and nobody ever connected it to patient safety so I think that's that's why there is a disincentive but in absence of an incentive for a hospital to to really build security beyond just what the U of C are is going to find them for it's a change of of how they perceive security right cybersecurity is always the stick that's beating you that's saying you need to X Y and Z it's never the thing well if you don't do this this is what's going to happen and it's going to impact your patients I think we need to change the narrative and you know the way that we approach this is by getting the hospitals to believe that this is the best for their patients um this is not just something that that needs to be done because HIPAA says so or any other organization says so I don't know the joint commission and I'm gonna I'm gonna be honest with you I don't follow that because I'm not from the US um but I do feel that if you change the way security is perceived within a hospital and you make it more patient safety you will see people change the way that they approach the problem you're absolutely right and I think there's a balance there right we don't want people declining clinical care because they now understand the potential risk but I but I absolutely think you're right because here it should always be about the patient safety that we're trying to ensure collectively I mean COVID's kind of broken the perimeter right it's I mean they are ascending home monitors I mean which angle do you defend from you know where is our perimeter at the moment I think it's it's just we're gonna have to rebuild healthcare I think after COVID globally and that's an opportunity to do things different right we now have a way to move forward positively without having to reinvent the wheel because the wheel's kind of broken down now um so I see the positive in that as well and I'm excited for it personally all right everybody we're gonna have to wrap up here soon but I want to give everyone opportunity to just say where they're gonna go party after this Q&A so I'm gonna start off with Jeff where you're gonna party his internet is having a seizure he told me so I'm not sure if he'll be able to rip on he's gonna go party with me I bet somewhere yeah somewhere with that or somewhere didn't go to Starbucks and steal some internet ash where are you gonna go party after this Q&A I'm gonna open the new record player that I got that arrived in the mail today and set it up from here V where are you gonna go party after this it is six a m so I'm gonna go party at club duvet it is a very very very warm toasty club and that's where I'm gonna be spending the rest of my morning nice Vidya where are you gonna go party gonna hit up the beach all day every day all day every day and Jessica take us home I was gonna say I was going to club BED with DJ Pillow because that's exactly well on that note well on that note we want to say thank you to Def Con the CFP board for giving us another shot at this and everyone out there watching us on twitch to say thank you for attending our do no harm panel again stay connected with this this is really important work let's keep our patients safe and everyone take care


Today we're going to be looking at a list HW command, now this command will list pretty much all the hardware on your system. So it is something that probably should be run as sudo or root, otherwise you may not get all the information that you're requesting. As you see if you forget and you hit enter it says warning this should be run as a super user while it's scanning the system. You will get an output and a lot of information but there are certain things you can't view without being either sudo or root. So we'll sudo that, type in password and it will scan and run and give me an output which of course I can pipe into a file or grep through and gives you hardware information on the hard drives, the motherboard, USB stuff, all your hardware, firewire, PCI, you know and there's other commands for these individual but if you wanted to grab all that information and once it is list LSHW so list hardware. Now of course as always you can man that and see all the different information and there's a lot of different information here on how to output, you know you can output the device tree as an HTML page. So if you're running this on a server you can have the output display in HTML for a server side script so you can view it in a web browser, XML, you know a bunch of different stuff here. So you can just go through that, definitely look through the man file, that's great to do with any program you're using, it's not very long for this one and you can also instead of piping it there is a dump feature to dump it to a file. So that's just a quick look at that but I also wanted to show you a application very similar. LSHW is installed on pretty much most systems if not all systems by default but this next one is not, it's called LSHW-GTK probably in your repositories but it's the same thing but it's a GUI interface for it and I like to show this stuff every once in a while because anything that can be done from the command line can be done in a GUI interface as well. It's just usually more convenient to do it from the command line but just wanted to show this as an example and this is well as well you can run it as a regular user but you won't get all the proper information so sudo or root it and let me pull this up here so you can see now that I've run it and it just ran a scan so here's my computer gives me a basic overview right there basically it's taking all the same information and displaying it out bit by bit here depending on what I click so I can click this or double click this there's my motherboard double click that I can click on my CPU I get information on my CPU here memory so it's all the same information just displayed in a GUI interface that you can click on so that's another look at that basically GUI front end for the command line back end so that's LSHW-GTK and the command line version is just LSHW same information just displayed in a GUI interface so I thank you for watching please visit filmsbychrist.com that's Chris of the K there'll be a link in the description of that as well as my Facebook fan page check that out to keep up to date not only on my videos but just Linux and open source news I post things there regularly also if you enjoy my videos please visit my website and think about clicking on the donate button and donating a few bucks if you can also you can follow me on Twitter there'll be a link to that on my website I also have an RSS feed that I've been working on but as far as my Twitter account my Facebook account tend to post the same stuff on both but if you want to follow me on both that's great as well so as always I thank you for watching and I hope that you have a great day