Now we have two speakers, Kukan and Peters, about algorithmic trading with Python. Very interesting, thank you. Hi, this talk is an algorithmic trading with Python. Just to clarify some terms, by trading, I mean, buying and selling financial instruments on financial exchanges. By algorithmic, I mean, there is a computer program running some kind of an algorithm that decides on what to buy and what to sell in these markets. In winter capital, we manage about $35 billion using a platform, primarily constructive Python or largely constructive Python. We also use other Python for researching and data analysis for those activities.  We'll do a quick company overview. We'll have a little bit of an overview into our research activities and the trading pipeline itself, and then yours is gonna go into quite a bit of detail about how and where we use Python. My name's Isto Kucan. I'm the head of Core Technology at Wynton, and yours is a head of a very exciting new project we have the Data Pipeline Project, and it particularly is a heavy use of Python there. If you've come across Wynton in the past, you may have seen us call a Quant Fund, a algorithmic trading outfit, a hedge fund, commodity trading advisory. All of those are valid, but I think a lot would much rather be described as an investment manager in the company that uses scientific method to conduct investment. What do we mean by scientific? Well, a empirical study of a lot of uses, largest of empiricism, hypothesis testing, experiment construction, and statistical inference in how we derive strategies that then we trade upon. We have around 100 researchers, which is about a quarter of the company, typically with a background in academia, academics, ex-academics, or post-docs. These are organized in teams. A lot of the activities be reviewed, so it's a fairly open activity in how we arrive at the signals. Another quarter of the company is in engineering, which again is a fairly empirical discipline itself. Geographically, we're primarily a UK company, so roughly 400 stuff in the UK, mainly in London and some in Oxford, but we're expanding globally four offices in Asia and two in the US. A lot of those offices are not just sales offices. A lot of those offices are actively growing. So for example, we have a new data labs outfitting San Francisco, looking at esoteric data. Okay, so this is a Python conference. So what about Python and Winton? Winton's been active for about 20 years, and for the initial few years, the systems were far simpler than what they are now, and effectively ran over an Excel spreadsheet. Now, of course, then gradually C++ extension started creeping into that Excel spreadsheet, and gradually those things were taken out of the Excel and formalized as a set of object called the simulation framework. And that was and remains the core modeling tool and also execution tool for our training systems. But we found that as the framework gained flexibility, we needed Python to start combining these objects in a more flexible way. So for example, if I want to do a Delta series and then a volatility series, I would be using the same two objects as I would if I wanted to do a volatility series and then a Delta on the volatility, but I wanted to combine them in a different manner. So Python was quite useful to do that. As soon as we engaged in that, as soon as we started using Python in that manner, it became very attractive for us to start writing strategies or parser strategies in Python themselves. And from then on, it never really stopped. So over the last 10 years, we've adopted Python for constructing the training platform but also increasingly in data analysis and in research. So I'm sort of starting to create these two terms, research, investment technology. We have quite a strict distinction of what is exploratory activity and what is training activity. So exploratory activity research is looking to things that may lead to something or often will not. And again, the research itself is conducted along three lines, I would say, core research, which is research into signals and let's call them market behaviors. Data research, which is research into data and properties of that data and then in an extended sense, deriving data analytics like volatility profiles, volume profiles, correlations from that data directly. And we now, as I said before, have a data labs section in San Francisco which looks at esoteric data sets, speculative data sets like satellite imagery or the deep dark corners of the internet. But once signals are derived, we transfer them into the investment technology section. Now there's a much more rigorous exercise where we have a quite static trading pipeline. And again, the key there is that you can do things in a very repeatable, very reliable, very secure manner with some sign off and the data pipeline itself is composed of roughly four stages. Let's call them data management, signal generation, auto management and post-trade monitoring. Now Python is used a lot in research but it's also used now extensively in data management and signal generation parts of the trading pipeline. With data management, typically the things we do in the trading pipeline is obtain large sets of data, clean them, transform them into the things we need. We use things like versioning to make sure that we can repeatedly see data as it changes. Python underlines all that architecture. For the signal generation part of the pipeline, we also use Python extensively. So Python still drives simulation which is a time series transformation engine and increasingly so. Python is also interfaced to a data storage engine called a time series store and Nearest is gonna go into that in a bit more detail. Right, so I'll give a bit more detail about how we actually use Python, some low-level detail where exactly it sits in our stack. So the main reason we use Python really is because it presents a quite a friendly face to research. Our low-level code is all in C++ typically, so execution, our simulation platform. It's not something you want the researcher to write so we expose all our codes, all the APIs are typically in Python. There are a few other options but Python is definitely the main choice. It's not just for research because it's such a nice and programmatic interface. We use it for monitoring, typically to serve as a web service as well and directly in signal generation. The reason we chose Python is extremely well known. It's very easy to learn. If you don't know Python, probably not too long before you do. And it has a lot of, just comes with a lot of support for data analysis visualization so it's quite nice to, as a researcher, just to get all that batteries included. So this is a fairly large-scale overview of our training pipeline. There's a few kind of core principles to it. The whole thing is event-driven. So something happens which causes something else to happen. In this case, for example, we get our data, for example, from Bloomberg. As soon as the data is there, automatically we construct our equities prices or futures prices. Once that's done automatically, all our strategies kick off and that kind of event-driven flows sit really at the core of the winter technology these days. And then we have, as Istok mentioned, the simulation sits at the right bottom there. So whilst our, winter is pretty much a graph, it's a real-time graph, it just sits there all-services, listening to stuff happening. But we have the simulation, which is kind of like an in-memory offline graph. So essentially it's really catered, it's really designed to do kind of time series analysis. So you kind of spin up a trading system. That will kick off one of these simulations. You run them, you can tear it down, you can serialize it. And that's kind of the other main technology that we have. So I'll give a bit more detail about both the real-time graph, which we call COMETS, and then the kind of simulation strategies to test simulations, back test simulations, which is written in C++. First simulation, it's written entirely in C++. It's been going on for about 10 years now, I think, right after we moved away from Excel, pretty much. If you just ignore the left-hand side, it's kind of similar in concept to what these days appears, things like TensorFlow. So essentially it's kind of like a graph. It's very well optimized. It's, in our case, strongly typed, so you can just not feed everything in everything. You have to, it's strongly typed data. There's an example there of a graph. It's quite a simple one. Two data sets, two data series feed into something like a formula. It can be just some of these two series, and then you calculate that thing. Now that's all running remotely on a calculation server. Typically these things can be thousands, or tens of thousands of assets. You don't want to run that on your local machine.  But we expose, on the left-hand side, the Python client. So any user, any researcher can just connect or launch or spawn off any of these simulations, connect to it, and has full control over the remote simulation. So there's actually an example there on the left. It's a real Python script. So the first thing it does, it starts a remote session, which is gonna cause one of these simulations to be constructed and launched on the server. Then it constructs these two time series, and it constructs a formula, and then it calculates it. That's the only thing you have to write, and you have full control over the simulation. This means researchers don't need to know any C++. Anything they need really is in the simulation. It comes with data, it comes with training systems, it comes with universes, all the kind of stuff we need. And essentially gives them fairly high level control over anything they need to do. A little bit about the technology. I'm not gonna go too deep here. The bindings, the Python bindings are extremely lightweight, so they don't know anything about simulation per se. As soon as they launch a simulation, they get everything they need from that simulation. They populate your Python clients with all the objects that the classes are dynamically generated. The objects are spawned into your namespace. If you create new objects, they are created both on the remote client and on the local client. Essentially, it gives you full control locally as if you were doing it remotely. It's very friendly in Python, so all the data is returned as pandas series, data frames, all that kind of stuff. One thing you can't do with the simulation bindings is once you can control the graph, so you're kind of limited to these things like formula or value-based series or universes or particular training systems that technology has implemented in C++. What you can't really do this way is if you have a complete, outlandish training system that you wanna try and you wanna kind of plug it into this graph. If it doesn't fit this kind of formula or data, they're kind of stuck. And for that, we designed embedded Python. So what you can actually do is in Python, write one of these objects that run directly in the simulation graph. So from that on, anybody can launch them remotely, run your training system, and you don't have to write any C++. You can just contribute your Python code and everybody can run it. It wouldn't normally go into training, but this is more intended for rapid prototyping. A researcher can pretty much build their training system in Python, test it in the simulation, which means they can back to this from 1970 to now. They don't have to write C++, because there you might often have to wait like a month or two months for technology to actually implement it. It's not a really good turnaround. So they can just build their thing, run it, test it, and if we're happy with it, then we can still implement it in C++ afterwards. That's kind of the idea. It's definitely around to rapid prototyping, although some of it is actually in training as well. The technology there, unsurprisingly, the C++ executable hosts a Python interpreter. We use Boost Python to do the marshalling. All the data is exposed in NumPy, so we use the NumPy C API for performance. Yeah, essentially you've got full control to these embedded Python for making your own Python trading system available in the C++ backend. It's extremely powerful. So that's kind of the simulation. As Isdok mentioned, we have this problem that we need to shift lots of time series back and forth. There's an enormous amount of time series to be saved. We have hundreds of thousands of assets. You need to be able to very quickly load and write these to a database. Things like SQL are way too slow because we do so much historical backtesting. We have to load all the data for 300,000 securities from 1970s now in memory or distributors and then write the results back. So what we designed, at the time when we started this, there wasn't really a good alternative. So we built our versions and the duplicate data storage, which I'm not gonna go into too much detail, but it's a columnar format, so it essentially is super effective for storing lots and lots of time series and lots of columns very effectively they're typed. It's backed by MongoDB. That's kind of an implementation detail. Anything that can store something from key to a binary blob would have worked. And really keys, that's immutable data. So one thing we don't want is once you've written your data frame, if you do it in Python, all you want is to get exactly that data frame back. We don't want the data to change. If you've written something, it can never change. You always get it back exactly like that. That's it really at the core of our strategies. Obviously, if you're testing a strategy, you don't want the data to change underlying what kind of reproducibility you want to know exactly what you did, you want to be doing that forever the same way. So this time series store is really revolutionary. It actually opened up a lot of possibilities. Technology there, kind of the same pattern again. So we tend to do something low level in a really optimized way, C or C++, and then we expose all kinds of high level libraries to make it more accessible to users. So the store here is backed by MongoDB. There's a C library that sits on top of it and that kind of deals with this columnar storage, so that we can essentially store it very effectively. And then we build very thin libraries on top of that, C++, C-sharp, and Python here. We're building a JVM one as well. And essentially these can be accessed by different kinds of technologies. C++ would typically be the simulation, but a researcher might use the Python library. And rather than having to deal with this kind of low level columnar storage, a researcher can just put data frames in there. It will get translated into C arrays and then get given back to you as C arrays as well, as data frames. Yeah, so small implementation details about how we've done this, we used to see the foreign function interface. And the nice thing is that it's such a friendly Python interface, you give it a data frame and you get a data frame back. You don't need to know about any kind of table formats or type conversions and all that kind of stuff. Combat transforms, like I told you in the beginning, Winton is essentially a graph and simulations, services are sitting there, they're waiting for stuff to happen, they react from inputs, they produce outputs. The next thing is gonna listen to those outputs and go on again. So this is what we call the combat transform system. It's microservice based, they sit waiting there on a topic from a bus, it's Kafka actually. There's an example there, it's super simple. We get the data from Bloomberg, we write it to the store, we announce that we've done that. Yeah, go ahead. The equities transformation is gonna pick that up, it's gonna write to the store and then as soon as that's done, our strategies are kicking in. We've got loads of strategies, so there might be five strategies waiting for the equities prices, they're all gonna kick off simultaneously, distributed. And as soon as they're done, we can start going into execution. This is the next to last slide. A little bit about the technology with combat transforms and kind of bringing it all a bit together. All the red things are where we use Python. Everything that's not red is kind of low level and exposes Python as its external API. So all our events are posted on Kafka, we use protobufs throughout for the communication. It's really nice for the strongly typed and the kind of, you can increment your schema. And then our service stack currently is in C-sharp, so it's a proprietary service stack that essentially deals with getting the protobufs, translating the protobufs. But the combat transforms themselves are hosted by the C-sharp, our Python interpreters, so anybody can write anything and become part of the graph that is Wynton just by writing some Python code. That Python interpreter might be a strategy that launches a simulation that will use the same bindings, as I explained in the beginning. That simulation can host your own trading system that you've written that would be an embedded Python. The simulation will read and write its data from the version store, which is our efficient way of storing time series. And that then, again, can use in the Python library, can be read by the Python store library so anybody can read that data that's been written by the simulation. Everybody has access to it through the Python store libraries. Whilst all this is quite complicated, there's a lot of technologies going around, the theme is always relatively the same. It's low-level code that is really optimized, tends to be written in C or C++. Implementation details are quite proprietary. It can be protobuf, it can be Kafka. But as a user, you're only exposed to kind of well-chosen APIs that we've defined. They're quite flexible, they're programmatic, because it is Python, you can do anything you want, but it is tailored and it is accessible. And by using, by providing that as an interface, it's still extremely performant. And we find that this is really good. So roughly as that said, it's all good. We think this works really well. We're quite happy with the system.  You'll just see Python, you don't even need to know that there's any C code under there. It's the primary interface, really, for data management and signal generation. Because it gives such fine, great control, you don't really need anything else. There's no need to go into C or C++, you can. And that's what technology does, if it needs to go really fast. But as a researcher, typically you don't need it. So you can define all your own data transformations. You can do with the data whatever you want. You can store data, you can retrieve data, you've guaranteed it will never change. So there's a time series store. As discussed, it's backed by a very low level C++ code that is implemented by technology and owned by technology. And the main reason we're doing this is because it's so great for analysis visualization, rapid prototyping, maintainability, I mean that because it's such a programmatic interface to all the underlying codes. You can write web services, you can write monitoring systems. Everybody can essentially start contributing to them in Python, which means we have an enormous view on what is actually going on in Winton's trading systems. Yeah, so it's all good. And that's also all I had. So thank you. Thank you. Yes, one minute left. Okay, oh many questions. Okay, so I just start from here. Hi, thanks for the talk. Thank you. So you're using C and C++ code because you're in the high-frequency trading or is this legacy code or? No, neither actually. So we are in low-frequency trading and it's definitely not legacy. So even though it is low-frequency, we do continuous historical backtesting. So it means even though we might just trade one new data point, we want to be able to very quickly test the simulation all the way from 1970 to now. So yeah, so that's the main reason, has to go fast because we test the whole of history. But we do trades over periods of months. So common tools like pandas and similar don't meet the data requirements? Like for huge backtesting? I think we found that it probably doesn't cover our needs. We kind of, we run, the trading systems that researchers contribute are in pandas and so a number of them go into trading. So things like the tracking error control, it does run on pandas and it does run into trading. It doesn't actually have a C backend. So it's not a C backend, it's not a C backend. It's not a C backend, it's not a C backend. We actually have a C backend, but we find that if things need to go really fast and we find we need that kind of speed, then the C implementation is still considerably faster to the extent that it's worth doing it. Hi, have you open sourced your time series store? And if you haven't, why not? Open source, which one? Your time series store for data. No, we haven't open sourced that. For no particular reason. So there is actually, initially we, it's only quite recently that we started looking into open source. I think this is actually on the list of potentially being open sourced. There's nothing particularly trading specific about it. It is very generally applicable. So yeah, that might come up. More questions? Your slide suggests Python 2, probably 2.7. It is exactly 2.7, yeah, 2.7. Why not 3 and what is the incremental cost of migrating to 3? So there's a lot of code in, there's an enormous legacy code base in Python 2. Upgrading it because we have all the C extensions right now is not trivial, but it is being actively pursued now. So all the new code that we're gonna start developing will be in Python 3. And then we should gradually migrate all the stuff. The problem is there's not really an enormous business case right now. It's a lot of efforts and we don't necessarily get a lot of it back at this very moment. But we definitely realize that especially as maintained as a support is gonna be dropped, we will have to have moved to Python 3. So that's gonna be our main reason. And obviously there's a lot of features that would be considerably better, especially with the multiprocessing and stuff. Yeah, that's for me personally at least. We need a good business case to move really. Hi there, thanks for the presentation. Which exchanges do you trade on and how many bytes of historical data do you have? Which exchanges, we trade all the exchange, but that's more is stuff. I asked you about 20 or 30 different exchanges. I wonder if it is, but American equities, European equities, Asian equities, futures, FX now, and fixed income as well. How much data we have? Depending how we describe data, typically we just probably about a billion numbers a day. We have a petabyte class total capacity. But that's a rough measure. How many we need is a different story, but that's how much we have. Thank you for your great talk. I have two questions. There is any authentication or authorization system? There is some researchers can see only few machines or something like that. Yes, sorry. And how does it work? The API does it? We have our own proprietary authorization system. It's basically token-based and then we have the SQL server. So you got the SQL code is backed by Microsoft. So we got the authentication there and then we got Mongo database, which is backed by certificates. So it's certificate-based authorization.  So all your data is going through HTTP and it does how it's so fast because millions of events can be sent. So a researcher would actually go directly to the store. So they make a direct connection to Mongo. They wouldn't necessarily have to be mediated. They can and we're actually considering to build like high performance services in the middle, like GRPC based or something. But right now the library that we exposed to researchers that sits on top of Mongo makes direct connection to Mongo. So that's why it's so fast. Yeah, so our authentication works. By the certificates. Yeah, certificates to MongoDB. Yes. Okay, thank you. More questions? Just a fairly semi-light question, but have cryptocurrencies or anything like that crossed your radars yet? Crossed the radar and then left it, I guess. It's not something we do yet. Thank you.  I have a question. If nobody... Okay, what's the real, why is there a Kafka thing and a protobuf? So what's the issue with this one? There are no other arrows than just this one. Kafka is in the message box there. So you see the message was at the top there. We use Kafka to back all our events. So we have pop-stop style events, which mean any consumer can kind of connect to any event that happens. So we needed a pop-stop message box essentially. We chose Kafka and we put protobufs on the wire because it's strongly typed and it's actually fairly compact. So if we need to send a lot of data over it, then Kafka plus protobuf is actually a really good combination. So essentially it's Kafka sits really at the core of Winton. It's all the events go through Kafka and everybody can chip in. Can you give an example of such an event? So is it a trade or what is it? It's at different scales. One thing that an event that can be announced is where Bloomberg says, I'm done. I've actually downloaded Bloomberg data, co-finered in the store, but at a lower level actually we do send every single piece of information across Kafka as well. So that's before this. All the data that we ingest that we download is streams over Kafka. And then depending on who's interested, it can be stored in Mongo, it can be stored somewhere else. It can be actually transformed. It can run tests on it. So all the data goes over the bus as an event itself as well. Thank you again. I just wanted to know why aren't you using any event-driven infrastructure such as Apache Storm or something like that? It looks like it's a perfect solution. It's possible, yeah. We are actually investigating things like Storm, Spark, Flink, all of them. Do you have something to say about that? We're a company that's 20 years old. So there's a lot of technology that comes on the radar that of course you would immediately like to have, but you can't because it takes time to migrate and you need the business case to migrate as well. So something being unisex is not a business case. Of course, having 35 billion under management also means there's a lot of risk. Making a small mistake on such an investment just so that you can get sex new technologies, again, not something that's very easy to justify. So we do like adoption of new technology, but we have to be cautious at the same time. Hi. So I'm interested in how are you testing technical systems like this? Because I could argue that there are thousand things that can go wrong. Yeah, it's a distributed system. It's a real-time distributed system. So what's your approach to testing? When I say that initially we gain so much by the immutability of data, that's definitely one thing. If you know that your data is not gonna change, then you don't have race conditions about it. This might need to be right before this reads. So immutability is definitely one of the core principles. And then everything is strictly event-driven. It's strictly a DAC. So it means that because everything is defined by the events, you can write extremely good tests. If the whole history can be reconstructed from the events of Wynton. More information. All the simulations are run every day for the entire history. So for example, for our simulation today, we'll compare the simulation up to the previous point, let's say yesterday, and we'll make sure that every single data point in the entire history of the simulation is the same. And the incremental daily step is also usually human-verified. There's still some human interaction, not because it's needed, but because it's a silent process. So there's a checkpointing process for this human. More questions? Yes? Just say if you are tired. No, I can go on forever. Thank you. There was a slide with C-sharp usage in C++ and part and all together. The main and the other one probably, whether it was service usage as well. Anyway, the question was... Yeah, exactly. How the communication is done between the components, between different languages and so on. Sorry, can we repeat that? So once again, the question is how the communication is done between the libraries in different languages. The communication between the libraries or in the company? Between the libraries in C++, in Python and C. I don't know the details about all of them. I do know in Python, we use the C-form function interface. Essentially what we always aim for is a fairly simple C-89 I think, interface to C-library which compasses all the logic and then the other libraries are built on top of that. They tend to be fairly high level and just through the mediation and marshalling of data. So all the logic tends to be in the lowest layer and then all the other layers are just representing data in something that is useful for the language itself. Does that answer it roughly? Okay. No more, there's still one question. We are also at the boot, by the way, if you have more questions afterwards. Yeah, maybe this is then the last question, but why not? Hello. I would like to ask you if you save some pre-computed data in the history. Save some signals, you take the source data and compute something from them from all the history.  Yeah, so that's what Istok alluded to. In order to make sure it fits in with your question, in order to make sure that nothing has actually gone wrong in the meantime, that no bug has been introduced, we run everything from history, from the beginning of history pretty much, to now, to yesterday. We check that everything is exactly the same and only then do we allow the newly generated points to go through, yes. So it gives us an enormous amount of certainty that, yeah, nothing has gone wrong. So you then have problems with the immutable data that they can change because you improve your algorithm and find some error in the algorithm. It can happen, and then we re-baseline essentially. So if we do introduce a change, it has to be in a controlled fashion. So the only thing we want to avoid is uncontrolled change. But of course, if there's an improvement, then we will re-baseline the system, yes. Thank you. Okay, so I think very nice talk, very interesting. Thank you.


Good evening everybody and welcome to the Center for Strategic and International Studies. I'm Andrew Schwartz here at CSIS It is such a pleasure to have you here Thanks for bearing with the weather or non-weather, whatever we call it here in DC This is a terrific panel we have tonight, but before we get to it I have some people I need to thank first I need to thank the Stavros-Nearchos Foundation who have made this series possible with their generous grant I'd also like to thank the Horned Frogs at TCU. We didn't make it to the national championship this year But we would have been Ohio State, so My wife's from Ohio and half the family went to Ohio State, so I probably shouldn't be saying that in public Without further ado, is there anybody better than Bob Schieffer? Did you guys watch last night on the State of the Union? Bob Schieffer Let me introduce a Gotta work on the hand drops a Terrific panel here tonight Jim Lewis, and I think most of you here know him director of senior fellow of the strategic Technologies program here at CSIS used to work at the Department of State the Department of Commerce Foreign Service officer authors has authored numerous CSIS publications on the relationship between national power and technology got his PhD at the University of Chicago where one of the Sanger children is about to Enroll and he I have to say is the go-to guy on cyber and when I Moderated the presidential debate Last time out. He was the first guy I talked to about cyber and it convinced me He knows more about it than anybody. I know so we're always glad to him have him Sean Henry Now the president of crowd-strike services and company It's a computer security company. He's a retired executive assistant director at the FBI credited with boosting the agency's computer crime and cyber security Investigation capabilities oversaw crime investigations across the globe Including bank and corporate breaches and the state-sponsored intrusions and our buddy David Sanger As I'm sure all of you know national security correspondent for the New York Times twice been a member of reporting teams that have won the Pulitzer correspondent bureau chief for the times in Tokyo for six years Covered Japan's rise in the world as the second largest economic power then his downfall into recession He has also shared his reporting and insight many Sundays on face the nation In the author of a recent book on all of this stuff and we're always glad to To have David with us. Well, let me just start out by asking the panel a question last night Of course was the state of the union message the president Addressed cyber security I just like to hear from each of you and we'll start with you Jim Tell us what the president said What did you make of it and should he have said something other than what he did say? So I'd like to thank David for opening the statements here, no, I'm just kidding It was a good try though. It didn't work I didn't part of it is it they'd already shown so much leg and gone around briefing people I didn't really think there was anything particularly new in what he said They're They didn't go into a long Exegesis on cyber security, you know the drill in the state of the union is you always try and get like your issue or your Paragraph in it and because then we can say that are on an interagency discussions while the president said in the state of the union But in this case, I think more is going on than he said they put a lot of emphasis on Information sharing because that is the current primary task for the guys doing cyber security at the White House They have both a legislative package not particularly new and also some kind of executive action that we'll see probably next month so One of the things I think you said this use You were David said a little surprised you didn't utter the words north and Korea in sequence It would have been nice. I guess they thought maybe it was a little I didn't hear about a sequence well If you look at the beginning and the end of the text he actually never mind You know that one has turned into more of a debate than I think people would have expected and it says something Maybe about why they they kept it as a relatively Neutral statement is that there is a lot of skepticism about what the US is doing a lot of questions You know, I know that more is going on behind the scenes so it didn't particularly bother me How about you Sean? I I know that there's more going on behind the scenes, too I think that was the perfect stage for the president to tell the general public what's happening You know the public is looking at Target and Home Depot and JPMC and a whole Sony and a whole host of other things And I think that there's a lot of concern in the general public I think that there are people who don't know what's happening They don't understand how the the threat to our critical infrastructure what the risks are and what the cascading Implications are of something like that when there's a destructive attack not just the theft of personally identifiable information But actual hardware that's being destroyed from 6,000 miles away I think it was the perfect opportunity to tell the general public two things First that we're addressing this. This is important We recognize it as an issue and secondly there are actions that we're gonna take You know we're focusing on terrorism certainly we're focusing on crime certainly But this issue which is a new threat and emerging risk to the American public It's something that we're attentive to we understand and we are gonna send a message to the rest of the world about how We are going to address this because the risk is too high so I think there was a missed opportunity there and I think that much of the statement was General and and needed to be a little more emphatic what would you say Sean that the threat? That cyberposis to this country is greater or less than people Recognize right now. I think I think that it's greater than the average American Recognizes I don't think the average American really understands and I'll use the Sony breach as as an example Why what I've read and what I saw and I've watched a lot of media on it and I've heard people talk about it I've read the blogs etc. What I saw and heard about the Sony breach was Angelina Jolie is a spoiled brat and it was in a in an email and that you know people stole these emails which happened and People wanted to disrupt the distribution of a movie which happened, but there was a physical attack There was a hardware attack. There was destruction of physical equipment that is a Changing and evolving risk and I don't think the average American Recognizes what that risk is what they see in here is theft of emails somebody lost twenty five thousand dollars Somebody lost their credit card number. I don't think they understand the physical implications of this and that to me as a concern You know, I that's an interesting point. Someone was telling me today and I was totally aware unaware of this Sony's IT system was pretty much wrecked I mean there were people in that company out there still don't have email because Of the damage that was done to their system and that's something that I have no idea about you've just made my point That's exactly right, but some would say it's it's a good thing. They don't have access to email after David Tell me what you know, I know as a journalist I don't expect you to pass judgment on what the past president said last night, but You just hear two gentlemen say that and I know you never do. Oh, of course, I I have no opinions as you know but You just heard these two experts on this What could the president have said? No, I agree with Sean This was a missed opportunity and the reason was a missed opportunity as the president said less Then he actually said in his December 19th news conference where he took the major step of identifying North Korea and its leadership as the group responsible in his view for the Sony hack Now why was the Sony hack different? Sean hit the first point on it. This was the first major state-sponsored destructive attack in the United States We've only seen a handful of major destructive attacks Stucks net Olympic games that was the US and Israel against Iran an Attack on South Korea that people are widely believed but hasn't quite been proven was North Korea in 2013 An attack on Saudi Aramco, which has been attributed to Iran There was an attack in Germany recently against a steel company But you can count on one hand the number of serious destructive attacks that we've seen and this was the first one in the US It's the fact that it was a destructive attack Combined with that sort of 9-11 ish sounding threat to the theaters that you heard the week before Christmas That raised this in the White House in a way that other attacks had not prompted a national response Think of the other attacks that were mentioned just before Home Depot target JP Morgan Chase could have added Banco America from a year before in each of those the president's view was The people who run these systems are are responsible for protecting them and the US government can't step in and Retaliate for every commercial attack that's out there in the Sony case He made a different decision Largely because it was a destructive attack and he wanted to begin to try to create some sense of deterrence And that's why he said we will deal with the North Koreans at the time and place of our choosing We've never heard that before in the case of a cyber attack. And so the missed opportunity Bob I think was just to repeat What he said in front of a much smaller audience during that December 19th Press conference so that he could begin to create some sense of deterrence A last point on this we did an interview back in June with Admiral Rogers the new then new head of National Security Agency and we were talking about some long-term goals and he said, you know David The one thing I've got to do while I'm here is get past the moment where people think that cyber attacks on the United States are cost-free There's got to be some sense of a price to pay and The Sony one was the first one where they began to create that Jim, I know you've Written a lot about this, but I'd love to hear from from both of you on it. Why did so many experts? so called experts Question the government's Conclusion that this was the work Well, the the internet is you know people talk about it being a platform for democratizing information and expertise and it is but democratizing doesn't necessarily mean good So it's kind of a giant sounding board where bad ideas can reverberate Endlessly people aren't trained in this kind of analysis. It's not something you just pick up by watching a couple of the born movies you know and there's a I For obvious reasons a not out of insight into the intelligence activities that lay behind the US decision That said I think the fundamental reason was distrust of the intelligence community a distrust of the American government Damage to the credibility of the government over the last 15 years has been What we saw reflected in this in this incident night, I'd ask some of the loudest critics, you know, I said you You believed in Snowden you said everything that Snowden said was true and You love snowden. In fact, you wear a snowden mask at home at night in your bed, but But Snowden said it's not so I have a snowden mask upstairs if anyone wants to borrow it You snowden said NSA spies on everyone all the time everywhere And yet you refuse to believe that they knew it was North Korea not talking about how they knew but how is that and You know that people get defensive and say well, you know, it's not the same and blah blah blah It's you don't want to believe that NSA could do this You don't want to believe that the US could do this and again this is a point We were talking about a little earlier We have not always been the most open government when it comes to the capabilities We have acquired in cyberspace either for offensive action or now for attribution and that's because of the close link to intelligence Sean you don't have any doubt that North Korea is the one You know attribution is is very very difficult I think that you've got to look at the totality of the circumstances of the totality of the pool of evidence to come up with a Conclusion a reasonable conclusion, you know, we we put people death row for proof beyond a reasonable doubt It doesn't mean that there's that's never wrong, but that being said, let me say this We in at CrowdStrike where I work we Look at intelligence indicators indicators of attack indicators of compromise we We work very very diligently to do attribution in a whole host of these types of investigations Because we believe that it's really important for some of the points that both David and Jim made It's important to do attribution so that you can send a message to the actors whether from a criminal perspective you're bringing somebody to justice or from a national security perspective you're able to To launch some type of sanctions. There's got to be a cost people have to understand So we do attribution and we have looked at the indicators that that the FBI put out and we have been tracking North Korea for Six years and we believe that it is also likely as the government came to a similar conclusion That North Korea is involved based on the tactics and techniques that were utilized in this in investigation or in this attack Based on prior attacks. We've done had conversations with folks in South Korea about attacks that David alluded to earlier where It's it's believed that North Koreans were involved and when you start to look at Numerous attacks in the aggregate and all the intelligence that's collected over a protracted period of time You're able to start to put the pieces of the puzzle together In listening to what the government has said and in listening to what some of the skeptics have said Those skeptics have a very narrow view of and they've seen a very narrow view of what actually has occurred and the totality of Circumstances the totality of the investigation, which is the intelligence community. There's forensics on the ground. There's interviews. There's Siggent through the intelligence community. There's humans through the intelligence community. There's Siggent through international partners There's human through international partners I know what goes into these investigations because I worked them for many years in the bureau and it is a wide-ranging Collection of intelligence that leads to a conclusion So we looked at the indicators as a private sector company and we said we believe that this is likely tied to North Korea The US government made a similar conclusion their Totality of circumstances. I'm certain is much broader than ours and David that was a whole Basis of this extensive reporting that you did you you reported how the United States was able to to make This determination, you know, I come into this Bob as a reporter like you not as a cyber expert like our other panelists but One thing I know from covering cyber issues for as long as I have is that attribution is slow and painstaking Takes a lot of time so This attack was on November 24th and By the second week of December, we had people in the US government saying there's definitive evidence that it was from North Korea and Then the president comes out and says this on December 19th So not even a month after the attack and as soon as those words came out of his mouth He had to think that he must have seen something pretty definitive because what we know about President Obama is he is very cautious He's particularly cautious and somewhat suspicious on intelligence issues and He was highly critical of his predecessor Leaping to conclusions on intelligence about weapons of mass destruction in Iraq. In fact, it's part of how he got elected president So the fact that he was willing to Basically get out there and make this accusation Not simply that the attack emerged from North Korea, which is something they've said before about Attacks from Iran or Russia or so forth, but that the leadership was responsible told us that he must have seen something and It was an illusion made before to the Snowden documents a lot of people talk about the Snowden documents And a lot of people talk about the privacy issues in them If you actually go and read the Snowden documents or at least those that have become public you find a lot about US use of implants in China and Some in North Korea and in fact just this weekend just before we published our piece Der Spiegel the German magazine Published a series of new Snowden documents and one of them referred specifically to one operation to get inside North Korea We link to it from from our story. So There's not a huge amount of mystery here if you're going to have Rapid attribution it's usually because you can see the attack massing and you can only see the attack massing if you actually Have your computer surveillance underway inside the the location where it's coming together Well, what should CEOs take away from this attack and and Sean Wilkins seed They should hire your company that would be there first thing they ought to do, but I mean what Jim what? What should they What did they learn from this or what should we learn? What we should learn and what they should learn are different and Sean gave you a clue as to what the CEO should learn Which is they probably underestimated risk? They need to recalculate the risk of cyber attack and most companies have risk management Committees, I hope they all have cyber as part of the risk management portfolio. They need thank you They need to think about Recovery plans because a company is not going to be able to stand up to a state opponent They need to think about what they put in their email Some of us learned the hard way that you should think about what your emails will look like if they're on the front page of the times my Example of course used the word frog in an unflattering way towards the NATO ally and when you see that you have to think When I'm writing this it's a postcard. What do I think about so you got to think about damage control and mitigation? You've got to think about Estimating risk and spending more and you've got to think about what you put an email I know that's tough for people because we're used to thinking of this as a private communication That we can write anything we want and it's just not true. So that's what I mean Those are the same things I tell my 13 year old granddaughter, but I mean it's all As adults we all need to remember that too Sean What do you tell me? Oh, I agree with everything Jim said I think that Organizations I do a lot of of speaking to the C-suite and to boards and I can tell you that this this conversation has Evolved in the last decade from system administrators who are the people that are you know kind of hands on the keyboard working on the networks all the way through To CIOs corporate general councils and now it really is sitting in the board of directors That we are we can't prevent these attacks What many people still today don't understand is that the US government we're talking about the government the US government Has a fundamental responsibility to protect its citizens That's its primary role in my opinion and we do that in the physical world Against missiles and against foreign armies and against foreign jet fighters But we do not do that in this space and that there are malicious ones and zeros that are landing on corporate networks every single day And the government can't stop it What I tell CEOs is you are responsible for this because the government can't not because they don't want to But there are policies and there are capabilities and there are capacity issues that won't allow it today under to in today's environment And I know there's legislation that's pending etc. But under today's environment. It's not going to happen Therefore mr. CEO you've got to move Into a detection phase because you cannot expect that the government will prevent this you can't expect that your existing Technology is going to prevent all of this You've got to move into detection and into remediation and if you detect these issues quickly enough You can mitigate the consequences in most of these investigations that I've been involved with both from the the FBI side and from the Crafts Rec side we have gone into environments where the Network has been breached for months and in some cases years Completely undetected and if you give the adversary months or years of time on the network They are going to do whatever they want on that network theft of intellectual property research and development and Destruction of data if you can detect it quickly you can mitigate it So it really is about them understanding the risk and then taking proactive actions on their network To identify these threats today and not relying on somebody else to prevent it You know Bob in the case of Sony it's sort of interesting because we think about this as an attack that happened The week before Thanksgiving, but it actually began in early September. Well Over the summer there were warnings to the State Department a letter the North Koreans sent to the UN Secretary General saying that they The release of this movie the interview Which was not necessarily the most sophisticated comedy I've ever watched with the kids But you know nonetheless Would be an act of war. Okay, so in September There were spear-fishing attacks on Sony and what we determined in the court that spear-fishing attacks of those emails You know you get that they all you know, they may look perfectly friendly but when you click on it it downloads some malware into your into your system and As we went back and did the reconstruction What we learned was that the North Koreans then quietly spent or whoever these hackers were but if you believe the government's account the North Koreans They spent the next two months Mapping the Sony system undetected Figuring out their file structure figuring out what was vital what wasn't so when the attack actually launched on November 24th that's why it did so much damage and shut down so many computers and You know one of the few mistakes That the hackers made was that they put this sort of skull and crossbones and so forth up on the screen Which warned people who actually had their computers out at the time and were sitting in front of them There was an attack underway and some of them reached back and actually unplugged their computers Which is why they were able to recover some of the data from their hard drives had the North Koreans not put that up on The screen and Sean would know this a whole lot better than I would I suspect they probably would have done even more damage well, you know I Not to get too basic here, but What should I do every day? Okay? I work in CBS. I have I have a computer. I'm you know I'm online all the time What should I be just just as as a person there? What are the things I should do? You need to go back to pen and paper, Bob? You know, I'm not against that Particularly, you know, but I mean okay the things pop out there It just don't open anything that you are not aware of I mean you don't know where it's coming from or what you have to assess How much risk you think you face as a person and because a lot of these things are things you're not going to want to do So change your password. How often do you change your password? It's a pain in the neck, especially with this complex password stuff I Changed my password after every trip, right? That's a hassle, right? Yeah, and I change it randomly and I do things that make me forget it and I can't log into stuff And it's a pain in the neck right change your password data back up Do you have the the stuff that you care about stored somewhere that's not online? It could be a thumb drive thumb drives are amazing now you have to store that stuff and then you have to think about what you're writing and I was going to say communications discipline, which is one of my favorite lectures, but I won't you have to think about what you're You have to think about what you're writing and if you you know, it's the New York Times what the New York Times test or sorry The Washington Post test depending on Are you going to want to see it in the times? So password the other stuff is basic hygiene that hopefully your network administrators take care of at home You know, are you doing the patching and updating if people want to get you they're going to get you but you can minimize the damage by storing offline and by restricting what you actually communicate Do you have any tips on passwords either of you? I mean, well tell me yours and I'll give you a tip But it should not be anything that is familiar to it is it better to use numbers is it better to you? So one time I was in a meeting with some of our foreign friends from a signals intelligence agency and this probably gives It away, but you know the London Times Crossword puzzle which I can get zero on this guy was sitting there It was actually a woman doing it in ink and it took her about six minutes Right the people who do this Are going to figure it out if you use a word right for crying out loud and especially use a word to link to you And you've put it on your Facebook page. Oh, thank you Thank you social media because I I can I can harvest your password. So don't use your dog's name Don't use stuff. Don't be cute, you know eight hackers. I'll put a z at the end instead of an s I'll never figure that out, you know, it's like so you really have to But that makes it hard, you know, that makes it hard So I won't even tell you which accounts I'm locked out of right now because I can't remember my complex password But it's a choice Do you if you don't do the password you're going to be you you and we've we've said this before You two guys and probably you too. You're great targets. I mean access to your files. Who wouldn't want it, right? Access I don't know what you keep on your computers, but it'd be a lot of fun Right, so you you are a higher risk At higher risk than most most of us But when you talk about this spearfishing, I mean, I guess the point I'm trying to make this could happen to any of us I mean these and is it these people that have these Of extortion schemes from Nigeria that you know, what happens if you open that? I mean when I open it by mistake and then I delete it immediately But that's not good enough is that you really shouldn't open it depends So if there's a if it if the email has a payload in it that will Infect your computer and give the remote operator some kind of control Yeah, you're in trouble if it just I get them all the time and I'm so upset man Where are my friends? I just told me I won the French lottery and If you get one of those that just say hey, you won the lottery, you're probably safe You won't get the money, but you're Why did North Korea do this is it for what would seem to be the obvious reason? Or is there something beyond that?  And so people go to china You know, I They see how people live in china, which is infinitely better. They've given them smartphones You're you're a vulnerable god in a way that perhaps your grandfather wasn't vulnerable And somebody does this movie That tastelessly shows you Being blown up at the end and this is damaging to you in a couple of ways. Yes, boy Yes, right But the comparison you want to make uh tooth out here. I'll do it. I'll do it. They can't stop me Hans bricks. Oh, no Team america Kills kim jong-il, right And the north greens couldn't do anything about it. Here the the the god figure had been Killed and there was nothing they can do. Thanks to the internet. There is now something they can do It was political Oh, can I for yeah, I I agree with everything that jim said there, but um as somebody's covered north greea. I'm off since I lived back in tokyo Cyber is a weapon that is perfect for failed states That feel as if They've got very little ability to strike back at their adversaries And it's perfect for several reasons. First of all, it's cheap Don't underestimate the importance of cheap when you're dealing with a country like north greea Secondly, it's a lot more usable than a nuclear weapon a nuclear weapon is on and on off switch The north greeans like to have the nuclear arsenal because it can help preserve the regime But they know that if they ever used it, they know what happened 60 minutes later 32 32 minutes Thirdly, there are a lot more accurate than north korean missiles Think about north korean missiles And They're on instead of being on and on off switch They're they're like your thermostat. They're on a rheostat, right? So you can turn a Cyber weapon up or down And try to modulate it in a way that you don't think that there will be retaliation especially if you think somebody can't figure out that it was yours And uh, this is why I think adamel rogers was talking about why there's got to be a price Because because the attribution is so difficult it makes a lot of people feel safe in using them And because they can modulate it they think well, I can stay just below the threshold where someone's going to come back And get at me And in this case the question is did the north koreans Get that setting right I'm not sure the sanctions that the Obama administration announced Against north korea on january 2nd are going to make a huge difference to their lives since they've been sanctioned by every president since Harry truman and it's kind of hard to figure out new sanctions that would make a difference So in the end, I think the question about this is going to be did they really pay a price? do uh One thing that we were talking about what the president said last night and I think david reminded Has the president ever talked about our? Offensive capability if he has I've missed it and I watch it pretty carefully It's interesting because obviously the united states after Uh as the nuclear age started because we had dropped a bomb on two bombs on japan It was no secret we had offensive Nuclear weapons and we spent 25 years debating how to use them and ended up in that debate in a very different place And we started off in the case of drones the united states wouldn't discuss them for years I was white house correspondent for most of the bush administration and you know the white house press secretary couldn't even use the word drone Remember how many times people came on your show and they wouldn't use the word drone even though you did in the question, right? But in the case of cyber the united states government is still in that phase where they think that They don't want to acknowledge having a significant arsenal of offensive cyber weapons And even after the disclosures about olympic games the attack on iran And the fact that the code itself got out in that particular case because of nobody's fault It's just an accident even then the united states has never acknowledged it At some point the us is going to have to do it because at some point We're going to try to get into negotiations as a country to set some norms about how one uses offensive weapons It's very hard to have that conversation if you don't acknowledge owning some What about that? Well, we are having that conversation and of course what governments say in private to each other Might be a little different than what they say in public and that might be unfair DoD did put out some recent doctrine that if you know what lies behind it it points towards offensive capabilities We are having discussions with other countries about the norms for the use of cyber techniques For force for attack and for coercion It hasn't been an issue in part because everyone assumes the nice thing about So a friend of mine was telling me when I started negotiating with the chinese long time ago He said you have to remember that they think we're like the borg You know our motto is prepared to be assimilated. We're in advanced technological power that brooks no opposition And that's what people think when you talk to the russians or the chinese So we don't have to tell them we have offensive capabilities in their wildest imaginings They think we have death rays and flying saucers and you know What have the americans thought up now and they're freaked out about it so I The conversation on norms is difficult and this might be one reason why you don't see As much public discussion and there's an issue that you don't usually think about but it's the third party issue To carry out one of these attacks you have to go over somebody else's networks It's very unclear how international law how the laws of armed conflict apply to that and so countries currently take in the position of saying What are you talking about? I don't third party. I don't know any It's it's too hard to deal with and there are discussions. There are norms. There was agreement in 2013 on Some very general norms by the general by the un general assembly. So all nations Those discussions are ongoing, but they're very difficult legal issues that make you think maybe I shouldn't be so open about this Nobody nobody else comes out and says I have it too And nobody else has come it's it's interesting though because you you raise a a big question then for me Because if the chinese and the russians both believe that we've got certain capabilities That has not dissuaded them That in and of itself has not dissuaded them from stealing the intellectual property and research and development from major corporations around this country And quite frankly around the globe We've done attribution back to russia in us energy companies. We've done attribution back to china in you name the sector And that's been communicated and it's very clear. There's really no doubt there yet They have not They have not They have not acknowledged any red line They continue to cross over because they have not seen the results or the the impact of any of that breach No, I think they have acknowledged a red line and so under international law Espionage does not justify the use of force in response You can commit espionage and nobody's going to go to war over it. We should be thankful for that, right? And there is a there is an implicit threshold now becoming more explicit in international discussions The threshold is use of force or armed attack What qualifies as use of force or armed attack that would justify a military response espionage is not use of force or armed attack I'm not suggesting use of force for that. I'm talking about Going back to the the norms that you discussed There's not been anything Dictated certainly not that I've seen that clearly lays out what the ramifications are going to be for crossing that red line And they are over that red line every single day right now today Well, the the the only legal instrument the only international may agree to instrument that the We possess would be in the trade realm in trips And in wto i'll accept that so china nobody forced china to sign on to wto They did it willingly to get market access But they also committed to protect intellectual property and treat foreign companies the same way they treat their own companies They're not living up to that commitment. Now. You're asking why haven't we as a nation pushed harder on that? That's a different question ask that question why? Well, some of it is this is still it is uh it is um Closely linked to intelligence people don't like to talk about it The trade lawyers are relatively conservative when I started talking to trade lawyers about this He's three or four years ago the the first one I talked to said there's nothing we can do I was like don't open the talks by surrendering, you know There's a reluctance to take this on and you as with north korea People put this in the larger spectrum of our bilateral relations. And so with china for example Um previously there was a desire Not to rock the boat when we were in our economic recovery And so the decision was we're not going to push them on this Because it might hurt us economically. I know I'm being an articulate on this That's usually that and when I close my eyes means I'm thinking a class if I thought Um, you're trying to I thought you're trying to remember your password. No No, I've given up And things have changed now. So the indictments were a good first step now. We have to see what falls on Bob, you know just listening to the president's language on this Tells you How difficult it's been for the government to deal with this subject So in some interviews he did just before he went on vacation in hawai you may remember He was asked whether this was cyber terrorism Happened right after he identified north korea as a culprit and he stopped and he said no, this was cyber vandalism And then we had a couple of senators Step in and say president doesn't know the difference between Cyber vandalism and cyber terrorism. This is cyber terrorism. And then we asked some people on the hill So what's your definition of cyber terrorism? And they said well if you conduct a Cyber attack with the intention of destroying Thanks So we asked the question so if it was the united states that was behind the attacks on the Natanz nuclear enrichment plan was that an act of terrorism? Well, no not exactly If that's what happened. So we were having real definitional problems here What uh When david cameron came here, uh, we know he came here And he was going to urge the president In some stories they said to criticize u.s companies for Resisting efforts to give the government greater power to read encrypted messages. What does that mean? So just to a quick follow-up to david before tackling that one Remember that espionage is a two-way street and we've told other countries.  We spy on you. This is what big powers do There's a difference though. We are not involved in commercial espionage U.s government is not stealing from foreign companies and giving it to u.s companies and that is happening U.s companies are being raped and pillaged every single day. Uh, and so In discussions with the people's liberation army and the ministry of state security I've heard u.s officials say Exactly that and the response has been You know, uh in china For us building our economy and growing our technological base Those are national securities issues. We think this is legitimate And that's a fair point to have a discussion on and maybe we need a new norm relating to espionage But you know, they want to play the ball game that's going to do the best for them We want them to say no play by our rules where you automatically lose Gosh, why won't they go along? So I we can't we can't send attorneys over there Then who say we can't do anything as their opening position on his when his stuff on olympic games came out I was talking to some chinese friends and I said, um, so were you guys surprised by that? And they said no, we always knew it was you and I said, yeah, oh, yeah How do you know it was us and he said and the individual said who was a chinese official? He said no other country has that many lawyers involved in in their programs. So So but then I said you guys don't have lawyers and he said no, but they're increasing so Give him a couple years. You can send you can send your you can send your lawyers. I'm not saying lawyers. You went over there with a lawyer But on this thing is this I guess the another question I was going to ask related to camera Are we going to see a different attitude in this country toward the national security agency and so forth after seeing this This hacking incident here. I don't think frankly There was a a very vocal Group probably I'd say 20 25 of the population that was very hostile to nsa very Hostile to its activities and I'd say the majority of americans understood what it was doing I don't know if you agree with that. It looks like you don't well I was going to say I think that there's something that has changed in this encryption debate I think you touched on it, Bob If you listened to Cameron last week He said we need to have a way of getting into iChat and we need a way of getting into What's app and other encrypted forms of communication and what's really changed the argument I think Came about last fall When the new operating system For the iphone came out along with the iphone 6 Because this operating system automatically encrypted data But not only did it automatically encrypted it did it in a way where apple didn't hold on to the key So only you had the key really to go do it and that changes things because in the old days The fiza cord or an ordinary cord could issue a warrant They could go to apple with a With a physical telephone and say download all the messages that are in here Now with this new operating system as it takes off If you go to apple with that and you ask them to download all the data They will hand you a large pile of complete gibberish And say good luck send us a postcard when you've decrypted this in about five years And that is what's driving the intelligence agencies and many of the law enforcement agencies a little bit crazy Because at the moment that apple and google and some New products they're going to be rolling out and microsoft as well Throws away the key then they feel like they can't get at it and that's what cameron is talking about banning Now president obama Had never really spoken on this before but if you look carefully at what he said At that news conference on friday with cameron He seemed to come out closer to where cameron was than i've ever heard him before But not endorse it entirely did not endorse it entirely Let's have some questions from the audience anybody have a thought this lady right here Former senior you unofficial former former Former head of the united nations institute for disarmament research in geneva And I I have two questions. Um, this was a fascinating panel. It was really interesting and different perspectives Um, but I have I have two questions and they they tend to relate to the issue of seeing Um, the cyber issue always through the national security lens and how Perhaps that's not quite The whole picture the holistic picture. So the first question Is we were talking about this attack on sony as being the first attack on attack on infrastructure That did something destructive in the united states except that the first attack on infrastructure that did something bad was Pretty much we think from the u.s. And israel on iran in stuxna And there are a lot of lawyers international lawyers who said at the time That the iranians might have been able to make a case under un Under under law international law that this was an armed attack that they could respond to They didn't I would like to ask the panelists about how they think about that and what they think might have been the motivations for not and Why we didn't and all of that The second question I have is that if you're really looking at national security We have a big problem here because in order to defend stuff Both nationally and internationally the government who knows about Ways of attacking and and bugs and etc. Etc. Has to actually warn the private sector But if you want to keep offensive tools In your arsenal, then you don't tell anybody about those zero day attacks that you've ordered from hackers On the on the gray net or the black net and you're keeping in your arsenal to be able to attack other people But every time you do that you leave everybody else, especially the private sector open to attack So how do you balance that in a national security realm? because that's that's a really Hard question, right? So I'd like to hear the panel some response of that. All right. Well, let's see Uh, Iran was killing American soldiers in Iraq The u.s. Chose not to go to a war with uh, Iran It's a political decision and so You know from pueblo on The fact that it qualifies as the use of force does not mean therefore That it it is an act of war. And so that's a political decision Iranians made the right choice. They wouldn't have liked anything. They started Um, I forgot the second question because I got annoyed with that one. So go ahead I I agree. It's a political decision. It's the reason we have to have these discussions I think nation state to nation state we have to define what the norms are and there has to be clear lines drawn about what's acceptable and what's not and I see this issue similar to weapons of mass destruction and a nuclear issue and at that level because the cascading impact of of these types Of attacks is going to impact the globe Critical infrastructure It's going to have a global impact on human life. Uh, I believe that so I I think we've got to have those discussions at that level Another question Right over here. Okay The guy who's looking around that he's Here's one Hi, uh, jason tom from brookings just pull on the thread on the, uh deterrence theory piece and the cold war analogy So We don't want to propagate our capabilities seemingly Because we have superior offensive capabilities or at least we think we do But at some point deterrence theory doesn't there have to be a credible threat of response You know, for example in the cold war it was the nuclear testing regime What what's is there an analogy for that in the cyber realm and will there be a day where we have to Propagate our capabilities. Thank you David, do you want to talk? Well, sure, you know, there are a lot of analogies between cyber And nuclear and we do them because nuclear is familiar to us during the cold war We went through a period of time where we were ahead with the weapon and then the Soviets got it the chinese got it the indians the pakistanis the israelis and others and And so all the questions about deterrence that came up in the nuclear era Are the same questions in some form or another that you would ask in the cyber era The problem is all the answers are different And they're different because in the nuclear era States were completely in control of the weapons by and large except in the past couple of years after aq con began to To start peddling weapons around So In the cyber arena States can have cyber weapons patriotic hackers think about those who Attacked other states on behalf of the russians criminal groups think of the target case or the home depot case Teenagers And teenagers and criminal groups They don't tend to sign treaties much So uh, so that's why this is so much more complicated in many ways Than nuclear is now on the other hand it may not be as devastating Which is you know Truly good news and we used to talk in the nuclear age about how you could get bombed back to you know, the stone age In in a big cyber attack, we could get knocked back to like the 1970s or something like that We'd all be dressed really badly But uh, so there's a little bit of it. There's a little bit of a difference I let me add if I can david because I I agree But there's one disagreement and that is with those that asymmetrical threats So if it's not specifically in the nation state and you can't have a treaty with a teenager organized crime group What you can do though is hold nations accountable for its citizens in terms of cooperating I've personally worked and I've traveled internationally quite a bit talking to senior level officials in law enforcement and intelligence agencies about this About them helping because it is a collaborative effort between the fbi and other agencies with our foreign counterparts And they've got to hold their citizens accountable if those agencies came to To the us because there was an organized crime group targeting their financial services sector I know what the response would be here, but I've not seen reciprocity Sometimes not all the time I should point out quickly that the csis and the nuclear threat initiative have a big project on trying to figure out how deterrence Works in cyberspace and one of the issues is Credible threat to whom in the cold war we had one opponent Now we have multiple opponents and different threats might be what works with one may not work with the other One more question right here Stove pipes in the pentagon and through our government. So maybe for shawn and the panel How would you assess, you know, the united states government? All kind of corporations as far as like pulling the information together. So we have a unified front I think that there's been a lot of progress in that area In in the u.s Domestically here the bureau started the national cyber investigative joint task force Which was a collaborative effort with the entire usik nsa cia dhs all flavors and dod to look at All the intelligence that each of those agencies bring against common adversaries China russia and nation states as well as some of these organized crime groups, etc Each agency has different authorities different capabilities different capacity And for each of them to be able to reach in and utilize their specific skill set and bring that to the table Makes the nation much much stronger in intelligence collection and then also in what the response is providing senior policy leaders or officials With key information so that they can make policy decisions. So that has happened here I think that that can always be better But I think it's on the right track No, I think shan nailed it. I mean the Cyber coordinators office the the response to sony is a good example Look how quickly they did it. So and some of this comes from what was the name of that guy they caught in bangkok You remember the experience of terrorism has helped shape Cyber security because it's the ability to combine intelligence disciplines get law enforcement fbi nsa DoD working together that gives us an edge And in that we've learned the hard way why it's good to cooperate Well on behalf of tcu and csis. Thank you all for coming


I was gonna say yeah strand we got the right BG, but then I realized that we kid can strand specific Yeah, I think I'm just gonna get out and try and kill everyone quickly We have any last we have a shaman right there. Press last brother. I'll let someone else do the car Do these die quickly? Pretty quick Like where is our team man?  It's unreal man These guys are not real gamers in their boss.  I'm not going in there.   No, dude. How? He has to do one more shot. He was just afk. I cannot believe it. What have I just witnessed?   I'm proud of my team. Usually we just lose the first round We actually infiltrated the gate. So it was good shit, right? We got this we got this boys Taking the tanks out. Just press the wrong button To waste my trinket now fill with this, but maybe I should have need that tanks not lasting a while mate I'll tell you that much for free I was gonna let him get a bit of distance, but I doubted him too much fuck. They've got the graveyard on the other side I guess it's not that valuable right now Can they make tanks here if this thing is dead and not dead, sorry Maybe that's like a nutty strat if you can do that.  I feel like that's a good strat Do I get T2 web tomorrow? Yep Been doing my daily BGs to make sure I got enough points for it. So Actually not that much. I thought it was more. Are there any bombs that aren't here that you can pick up?    We've gone getting good damage on these guys already This guy's got a bomb. We've got focus and then Very good Stealthers grabbing bombs from them. So yeah, see that's what I'm worried about. That's what I was saying We need to get a flare in the door They can still step to the cannons though We need a couple of people at the back really right now. They're trying to do it with demos So this could be bad actually our randies have been a bit lackluster here. I think okay. We've got the tank stand Stop the bombs boys. Oh No, this might be all right. It's might be all right So I'm the bombs boys.  It's so loaded 150 So I'm it 130 one minute This guy's going for it man look at them. They're desperate now, man. They're not they're not in sink anymore But literally okay, we've got lads at the back. That's good.  We've got the reswave. Oh no the tanks No, no, no, no, no, no. No, no, no, no, no, no, no, no, no, no, no, no, dude It's gonna be rogues man. It's gonna be rogues The rogue knows someone worry on the roof.  Kill the rogue boys! I'm just actually in the whole game, bro.  Like, everyone came together, though.


What's up everybody Jason here from Jasper's casebricks.com 2019 panini national treasures baseball two box half-case break picker teams number three just sold out guys and Here we go Traditional two box half casers, so it'll be two boxes to the left two boxes to the right Roll the die whatever side we use here would be for this break the other half will be used for Break number four, which has already done the two spots left So sure depending on what's hit here or what isn't Impact how quickly that next break sells Here we go here we go so again remember 2019 Bernando Tate's junior rookies Eloy Hemenez Vladimir girl junior Pretty stacked glass. I have it have a nice life liking One two three on the left one two three on the right go ahead. Here we go Six one two three four five six, so these two right here will be for number four Four and I'm sure we'll break that Little bit later tonight the famous wood and tee box We're gonna hear the famous Tape right there. Sorry, we can upgrade to the Mets of the Jays get some pita Lonzo Or some bladdy Garrett Hampton in 99 Colorado Tuki Tucson Atlanta you got no a synagogue to 49 for New York a little Tate's Seven-way relic 31 out of 99 Last bumble Joe John D. Got the podgers a little Carl for Rillo for Brooklyn Dodgers And a Hyrule about 25 out of 25 nice beautiful three-color patch. I actually call it four colors I think it's a little maybe not just three colors Looks like a little black on the edge of that Juan Soto Washington Nationals gonna Gregory beautiful 25 out of 25 You got Christian Stewart 16 out of 99 rookie signatures We have sweet and loud brown and loud 78 out of 99 and I think about to put these in like two nineties or something doesn't fit Or slider box either one. Whoa What is this a little one-on-one Colossal patch and autograph Kyle Schwarber For the Cubs going to Jay the majestic tag last one is a cut signature materials Joe. So well Or see well Sue well One out of five that was nice shortstop for Cleveland man That's Cleveland John Wenzel to an Indians. There you go one out of five Alrighty, that is a great first box to correct Correct the Mundo next box. All right guys So let's go to the next one guys next box. It's here the little famous little tape No need man. How about this right here like check this out. It's okay We don't need Joe no more TJ all aboard the big hit express He has like that Most of you guys have heard it already. All right, Josh Bell To do 49 two out of 49 Pittsburgh with that one Eric We got kukuchi 99 You got Austin Riley to 99 Don Manley Donnie baseball five out of 25 six-way relic Kyle writes to 99 And we got a rookie patch autograph Brian bull Ricky or bull Ricky To 49 I Do not think this is gonna fit on top. I gotta get a 260 point Put that right over here, and we got a Kevin Kramer 10 out of 99 same thing Pittsburgh with that one we have a Nathaniel low To 99 We have a colossal Trevor story a little Jersey auto 51 out of 99 Very nice hit right there Rockies Tom Jones Last hit of the break is a legendary silhouette dual duals Kirby Puckett and Harman Killebrew to 49 Both for Minnesota Minnesota Twins Brandon There you go. Very strong. There you go. Very nice They go a quick two box break in the books guys remember number four is in the store We two teams left Mets, and I believe blue jays Nothing crazy here for both of those teams so potentially could be some nice hits hiding in the last two boxes of the case Appreciate you guys Jaspies case breaks calm


The water's always in motion. Why is that important? If you don't have the water moving, you'll cold adapt within a couple months depending on the person. Then you're not challenging yourself. You're basically cheating. Even if you use a lot of ice, you're self-insulating. I don't know if you cried still water recently, but I've been doing it night and day difference. Is it really? The first 30 seconds, it's like, okay, this is cold. Yeah. And then you get used to it and then it's like your body doesn't have that fight or fight going off like you do in the blue cube.


The Seneca Dragonfly Mark II multi-pump pneumatic pellet rifle operates utilizing the butterfly high-efficiency pump system. Unlike other multi-pump pneumatic pellet rifles, the Mark II has a unique triple joint assembly comprised of a push bar, a pull bar, and a linkage joint. Utilizing this revolutionary patented pump assembly results in an incredible 50% reduction of the effort required by the user when pumping compared to the original Dragonfly. Not only does this system reduce the pumping effort, but it also maintains the same effort for each pump stroke, unlike traditional multi-pump systems where the pumping effort increases with each stroke. This makes the Dragonfly Mark II with the butterfly high-efficiency pump system the easiest multi-pump air gun to operate on the market today and possibly that has ever existed. Each time the butterfly high-efficiency pump system is employed, it engages the piston assembly, which compresses a small amount of air into an air reservoir. The number of pumps executed by the user determines the amount of air stored in the air reservoir. The amount of air stored then determines the power output of the air rifle. Operating the Mark II is done in the following steps, beginning with lifting the bolt and pulling it back. Doing so opens the breech and engages the trigger seal. These actions complete the cocking sequence. Before beginning to operate the pump mechanism, the user should engage the manual safety if it is not already engaged. Extending the pump handle forward to its maximum angle and then pulling it back to its original position creates the first pump. The Mark II is able to fire with just three pumps, but maximum power is achieved by pumping the rifle 15 times. The user can control the output power of the gun by altering the number of pump strokes for a given shot. Once pumping is completed and with the breech open, either a single projectile or a magazine that allows for multiple projectiles can be inserted. After insertion is completed, the bolt handle is then pushed forward and down, thus loading a pellet into the barrel and making the rifle ready to fire. To complete the firing process, the user disengages the safety, takes aim, and pulls the trigger. The Seneca Dragonfly Mark II employs a single-stage non-adjustable trigger. Pulling the trigger disengages the sear, allowing the compressed air to escape from the firing valve, propelling the projectile from the barrel. The Seneca Dragonfly Mark II pellet rifle employs a manual crossbolt safety. It is mechanically designed to serve as a button. Lateral force exerted on the button to either the right or left blocks or unblocks the trigger mechanism. When the button is slid to where no red coloring is showing, the trigger mechanism is blocked. When the button is slid exposing the red coloring, the trigger is unblocked.


Appalachian Wireless has some of the best data rates in wireless on our plans and not to mention an amazing deal on one of our hottest smartphones but that's all the time I have to explain. Seastore for details, better service, bigger savings. That's today's Appalachian Wireless. Thousands of egg hunters were hunting thousands of eggs over the weekend as Tabernacle of Praise held an event in Pikeville and Elevate Church held their event in Prestonburg. I came down here Easter egg hunting spending time with my family and I wanted a bicycle. I came down here to have an Easter egg and spend time with my family and I want a bicycle. I came down here to hunt Easter eggs, spend time with my family and want a bicycle. Although everyone had a great time and many walked away with wonderful prizes the real meaning of the holiday was not forgotten. We have a really big heart for our community and we want to see as many people as we can know who Jesus is and be able to experience his joy for themselves in their life and that's what our mission at our church is and so we put this on just to show the community just the happiness and joy that comes with being you know a child of God. Well really it's a positive thing there's so much negative and this is something that the kids and the adults alike can do that is a positive in a negative world and so and just unity really a lot of people come together can talk and fellowship and just have a good time. Event organizers of both events would like to thank everyone who helped make the egg hunts such a success. Reporting from Prestonburg and Pikeville I'm Sean Allen for EKB News.


Inflammation of central nervous system, CNS, is usually associated with trauma and infection. Neuro inflammation occurs in close relation to trauma, infection, and neurodegenerative diseases. The level neuro inflammation is considered to have beneficial effects whereas chronic neuro inflammation can be harmful. In an immune system consisting of pattern recognition receptors, macrophages, and complement system plays a key role in CNS homeostasis following injury and infection. Here, we discuss how innate immune components can also contribute to neuro inflammation and neurodegeneration. This article was offered by Apishek Shostri, Domenico Marco Bonifati and Uday Kishore.


I came this country in 1997, December, third of December. I left Turkey when I was 18, and I started for washing dishes. It was empty, this cafe, in three, four pictures, nothing else. But I did paint it and everything. We are not a toy, we are human. Immigrants is great people. Great Britain, great immigrants.


Soul Zero has missions for you to find certain enemies and destroy them. One of those enemies is a lady by the name of Handsome Jackie. She also has an exclusive weapon too. Today guys, I bring you her exact location and their preview of her exclusive legendary weapon. How's it going guys? My name's DP Gen. I'd like to thank you for stopping by and checking out the video. If you enjoyed leaving a like really helps out and subscribe if you want more or lands free videos. Okay so Handsome Jackie is an enemy hiding on Prometheus and her exact location and how you get there I will showcase to you at the end as it's quite a trick. You will have to be on or have done the Space Laser Tag mission to get to her otherwise a lot of the path is blocked off. Okay so her exclusive is called the Iron World Nimble Jack.  With greatly improved accuracy while airborne as well as critical hits ricochet one bullet at the nearest enemy. So it's a pretty decent legendary and definitely one to add to that collection. So this is a place you can farm as per usual as there is a save point just near here. So if you don't get this weapon first try just quickly game load it back up and get to farming people. So the location in which you start your trek to Handsome Jackie starts on Prometheus within the Skywell 27 area. Go to this fast travel spot then follow the route I take on screen. If you are doing this as part of the Space Laser Tag mission you will have to progress through via defeating the enemies. If you've already done this mission you can just run past all the enemies straight to this point. But yeah guys this is how you get the Iron World Jack shotgun. Enjoy people. And on that note we have come to the end of the video. If you guys enjoyed it leaving a like rule helps out and stay tuned for many more Borderlands 3 legendary weapon locations. But guys thanks as always for stopping by. Hopefully you enjoyed the video and hopefully I will see you on that next one.


Okay, so this one's a molarity problem, the first molarity problem that you guys saw on some sort of quiz or exam this semester, but it says what volume of a 12.0 molar HCl solution is needed to provide 0.60 moles of HCl? So it gives us the concentration of HCl solution, molar HCl, and it also says that you want to have 0.60 moles of HCl. So it's wondering what is the volume of this HCl solution that you're going to need to get 0.6 moles in? Does everybody understand the question? So when we do that, all we got to remember is what we're going to be doing is using these kind of conversion factors. Okay, so I'm going to start with that moles of HCl that the problem gives us. Okay, but here we got to remember that big M is moles per liter, right? So this means 12.0 moles of HCl per one liter of HCl solution. Okay, so when we look at that, right, so we're looking for a volume, so liters, or I think in this case we're going to get it in milliliters. We've got the conversion factor that we need to do that already. So what we're going to do is put the moles, of course, on the bottom to cancel that. One liter of the HCl solution on top, and then again, like I said, it gives you a few options, but one of the option that is the right answer is in milliliters. So one liter of the HCl solution is 1000 mls of the HCl solution. Just go ahead, in terms of 1000. I guess we're going to take a look at that. To two sig figs, right? So you're going to put the decimal point there. Mls of the 12.0 mole or HCl solution. So again, you're just using that polarity as another conversion factor from moles per liter of your particular solution. Any questions going on?