that when we take down hate speech that kind of pushes it towards darker corners of the internet that we can't regulate towards platforms or even creating one's own platform on which to kind of create that echo chamber and push those ideas in the community that they're accepted I think that maybe what's even more dangerous than the extremist hate speech are people who are kind of on the margin or susceptible to turning towards it and when hate speech is in the public you have the chance to delegitimize it and speak to it on a public platform in a way that those people can see faults in it or see flaws in it or even see it as a crazy person with a megaphone if that is the case but when we just take it down do we not legitimize it and give it power and if that is the case then how can we make sure that in taking down hate speech we are delegitimizing it instead of pushing it to corners where it can grow and give it power so great point and much of the speech that we find offensive must be dealt with by counterspeech absolutely and that's a core principle of the First Amendment for the last hundred years and I agree with that and Nadine Strossen has written a book called Hate which came out last year which she makes that argument very successfully I think at the same time I think there's a difference between a private individual who's your drunk uncle, proverbial drunk uncle who's contankerous and then state aligned or state actors in Burma it was the generals calling for the genocide against a Muslim population as part of a long standing set of tensions between the Buddhist and Muslim population in that area that I think is different and maybe handled differently than the drunk uncle who's simply sounding off but you're absolutely right that 4chan and 8chan are radicalizing environments the shooter in Christchurch the shooter in the Powe synagogue are citing other actors on 4chan and the posting of the actual video of the shooting happens there the manifesto it gets circulated there so there is clearly a problem with suppression and mainstream environments sending it into the dark corners of the web where it then thrives just a few words about the internet as it's grown and evolved is the dimensionalities have taken on a scope that people say oh we can't regulate it or if you have for example going back I'm familiar going back as an attorney to the days when you were innocent when you had broadcast programs and there's actually a convention called the Brussels satellite convention where was the information mapped into signals communicated via direct broadcast satellite I was working at the UN in Paris at the time I was part of my job it came up again recently in the trade agreements the US, Canada, Mexico so in that context when we went to the diplomatic conference they didn't realize it was a broadcast it wasn't war mongering or signal jamming and they wanted to turn it into something to control the actual information itself I don't call it content it's the information represented in some form and at the time they went ahead with the Brussels satellite convention and they referred to the outer space committee of the UN to deal with the more political issues about war mongering and jamming it's much the dimensionality is larger because a lot of people can communicate more than they ever could before now when you look at it today and you see what they call platforms they're information providers and as an information provider they could be good or they could be bad they could be state actors so I would be very reluctant to actually have as a legal framework run out of private groups that may be actually state actors so there has to be the rule of law and that's why we're here today to talk about you could pin it off of say the declaration of human rights that's grand but how do you enforce that how do you take measures to prevent those information sources bringing information in how do you stop it at the border oops there goes some bits what are they but there are ways to do the management of the information itself right now that may improve this but I just wanted to say let's not say it's an impossible to do this now there's an example recently where they actually did it I'm a copyright lawyer so they had the notice and takedown in the copyright structure based on the white boat treaties to indicate your information is not really openly public so if you did you can actually ask the platform to have a notice and takedown but then there's a legal process if something is not carried out properly if you didn't if you said to take it down you weren't authorized or if you were the platform and you somehow wouldn't obey it so what I'm trying to say is that there are examples and to think it through before simply saying we're going to turn it over to somebody that's not under a legal framework of some sort that was more an observation than it was a question sorry that's right thank you I think we have time for a couple more questions can we get a microphone over here I think I might have grabbed it first that's okay my name is Adeline Hyde I work for the Washington office on Latin America and this report actually on the case of Guatemala it's very timely and I'm just wondering I have a quick question while we wait on some of these models to be adapted or accepted on a larger scale how in a worsening political context like that of Guatemala what can we tell individual human rights vendors or organizations that are experiencing this and might in the next few months have a heightened experience of this in this moment as someone who doesn't work on these issues as frequently thank you so did you hear the question she works in Latin America what can we tell human rights defenders in Latin America or elsewhere about how to respond to these issues that was a question we asked social media companies and Twitter said that they were engaging in a number of training exercises it is clear that training can help in greater literacy on social media to know how to flag to know how to respond to a program of harassment but it's also very difficult to say to someone if you're experiencing a troll storm and you've got 10,000 bots hitting your Twitter account is there anything you can do about that probably hide under a rock for a while but it's clear that that then triggers a responsibility on the part of the state to if it is neglectful of a pattern of abuse of intimidation and harassment then that does trigger international and national obligations legal obligations on the part of a state to protect its citizenry and we don't allow no country allows incitement of murder you can't do that and if they don't enforce it or they enforce it in a way that is only for some people and not for others like human rights defenders then that's a problem and so then one can argue that it could be taken to a higher order I'd love to see the Inter-American Commission on Human Rights take this case I'd love to see the Commission or court probably the Center for Human Rights thinking about this I thought about this months earlier so I'm just behind the curve here but yeah that would be really interesting to get legal guidance on those kinds of questions and one last question a sort of a practical question earlier you mentioned that how in the US or in certain countries Facebook and other social media platforms are essentially clubs and are private they can make their own terms while in other places they're more public and I wonder if in other areas of our legal system sort of classifying social media platforms as public accommodations for example like for purposes of the American Disabilities Act classifying Facebook as public so they have to comply with web accessibility if there is a potential impact in doing so in that realm and how that could have an effect on whether or not Facebook is allowed to basically have these terms that might be seen to violate the first amendment it's a great question so the question is to what degree would designating companies as public companies or public utilities allow greater enforcement for instance of the American Disabilities Act it's a great question and I could see that what you're describing could be a useful process to explore but here's my my reticence and that is if for instance these social media companies were designated public institutions or public utilities in some way then they would be like public universities and the ACLU has a 100% record of suing public universities for setting up hate speech guidelines Michigan being perhaps the most prominent example when University of Michigan created hate speech guidelines in the 90's they essentially just ended up writing a check to the ACLU for their legal costs they lost that one and so there the very capacious framework that social media companies have created for regulating hate speech would go away and it would become much more constricted it would go right down to the first amendment which allows certain kinds of suppression of hate speech for instance Virginia versus Black 2003 the cross burning statute of Virginia was upheld at the Supreme Court because it only referred to intent to intimidate whereas the Michigan statute intent to intimidate on the basis of race or ethnicity was struck down because that was content discrimination so then we would have a much more restrictive framework if that were the case and do we want that open discussion thank you all so much for coming to this plenary session thank you to our panelists as well everyone I have a housekeeping announcement you've noticed that we have far more to say than we have time to say it I'd like to ask your cooperation in helping us recover some time for our afternoon panels by trying to commit to a 30 minute lunch the afternoon will be just as interesting I promise thank you so much good afternoon so I guess we'll get started my thanks to Linda Beshai and Mary Greer and the entire ABA ROLI team for hosting this great event and inviting me to participate in it I'm Steve Kelly I'm FBI's chief of cyber policy and I'm pleased to moderate this panel on cyber crime enforcement before I introduce our esteemed panelists I'd like to set the scene so a cursory glance at the news on any given day will inevitably yield a major story on the cyber threat indeed many of us have become desensitized to the major data breach announcements stories on election hacks foreign disinformation campaigns compromise of national security secrets and the penetration of critical infrastructure control systems the single factor that makes cyber threats more challenging than all others is the fact that the perpetrator is rarely in the same country as the victim an internet infrastructure and traffic traverses national boundaries in very inconvenient ways in response nations around the world are building up their capacity in network defense law enforcement their intelligence service and their militaries to protect and deter against these threats and increasingly working together this morning we heard about norms of responsible state behavior in cyberspace fighting crime and addressing malicious cyber activity emanating from one's own territory is a fundamental responsibility of states and while solving cyber crimes remains challenging the U.S. and its allies have had success in gathering evidence and making cases with the assistance and support of victims retained cybersecurity firms and outside counsel this panel explore the current state of play in cyber crime the importance of cooperation by victims experiences in international law enforcement assistance the U.S.'s recent experience in charging state actors with computer crimes and the effect these efforts are having to reinforce norms and deter destabilizing cyber activity so let me introduce our panel we have four guests here covering the spectrum of cyber crime prosecutions all the way through private counsel to victims so we have Mick Stawa is here next to me deputy chief for cyber crime with the justice department's computer crime and intellectual property section next to him is Will Lyne he's a cyber crime expert and a liaison officer with the U.K.'s national crime agency he splits his time embedded with the FBI and at the British Embassy and Luke Dimbalski partner at Dibba Voice Plimpton and then next to Luke is Sean Newell he's deputy chief for cyber with the justice department's counter intelligence and export control section so welcome panelists so I'm going to turn it over to each of them we'll just go in this order to give a couple of minutes of who they are a little more detail than what I provided and kind of how they come at this topic so just a little bit of framing and then we'll get into the substance of the discussion Mick right so am I on? so I am the deputy chief for computer crime so really the limiting factors on there are computer and crime so I am dealing with all of your attacks and intrusions that are conducted for criminal purposes not necessarily every attack or intrusion that constitutes a crime because there's a division in the context between those of us who are prosecuting criminals for criminal purposes and those who are prosecuting nation states or terrorists and that would be I'm sure you hear more about at the end of the panel so all of the computer fraud and abuse act prosecutions in the country come to my office for consultation and we run an international capacity building program to help improve worldwide capability to fight cyber crime the iceberg part of my title as I like to call it is electronic evidence because it's not actually in my title but it's actually more of what I do I help federal law enforcement actually comply with the laws that regulate our collection of electronic evidence including from networks and then we try and take those two things what we know about how people are attacking computers and the laws that restrict the ability to monitor computer networks and we try and turn it around and be helpful to the private sector in what we call our cyber security unit I was the first head of the cyber security unit and I've now passed that responsibility to one of my attorneys so those are three of the big areas that I cover. Thanks Mick. Will. Thanks very much Steve yet my name is Will Ly and I'm from the UK's national crime agency and we lead on all aspects of serious organized crime law enforcement in the UK. I head our cyber liaison team over here in the US so we have a team of three going up to four people over here embedded in various different locations. Cyber crime is by its very nature international and the US is an incredibly important partner for the NCA in tackling serious organized cyber crime and yeah that's what me and my team over here in the US do. Hi and I'm Luke Denboski and my career in cyber started much like the work that these guys are doing I worked for 14 years as a prosecutor at the justice department a part of it with Mick in the computer crime section a part of it with Sean in the national security division two and a half years as our representative in Moscow to the Russian government on cyber and other issues and when I left in 2013 I like to say that I had absolutely all of that under control and I'm not really sure what has happened since these guys will have to explain that. For the past three years I've been heading the cyber and data privacy practice at a law firm I'm based here in D.C. the firm is global devil boys in Plimpton and so my teams are working on at any given time between maybe six and ten cyber incidents it could be an insider attack it could be a nation state attack could be something that came through a vendor or supplier so we're dealing with a wide mix of incidents all at the same time around the world and that's my job these days I'm Sean Newell I'm with DOJ's national security division Luke I think the issue for Russia is we're going to send you back if that's okay with you it's been nice knowing all of you so I'm in the national security division I oversee our portfolio of cyber investigations that deal with nation states and their proxies for those of you who don't know the national security division we are the department's newest litigating component we were stood up in 2006 following the September 11th attacks the idea of staying up in national security division had to do with trying to tear down that wall between criminal and intelligence tools and attacking the national security threat at that time the primary impetus was the terrorism threat but we've definitely applied those lessons and methods to a variety of national security threats including a national security cyber threat where we use both criminal and intelligence tools to eliminate the threat and figure out how best to disrupt them and kind of manage the criminal unclassified side of the house and the national security like sensitive techniques use the US intelligence community and things along those lines so we can all work together to counter this problem great well let's get started so generally speaking we're going to start with the cyber crime side so the pure criminal side and then we'll work our way through the topics to state activity and norms so Will's going to help us get started so Will can you give us a lay down on the state of play in cyber crime yeah well I'll do my very best so I thought I'd just run through and touch on a few points probably some just some emerging trends that we see in cyber crime at the moment which I'm sure hopefully some of these will be familiar to everybody in the room here so I think probably the overarching theme is we're seeing increasing levels of sophistication across the board cyber crime is getting more technically sophisticated by the day secondly and probably most importantly there's been a real low lowering of the barrier of entry into conducting cyber crime which in effect has meant that there's a proliferation of high-end cyber criminal tools and techniques around the world 10 years ago to conduct a cyber criminal scheme you needed to be pretty technically sophisticated you probably needed to be a hands-on coder nowadays you just need to go and know where to look in the right place and buy those things bring them all together and you can conduct a very sophisticated very impactful cyber criminal scheme so a really good example of that is criminal cash out a few years ago you had to have had a mulling network of people in the countries where you had to make fake cards and all that type of thing nowadays you can just pay some people to drop the clean Bitcoin into your account thirdly I think we're seeing increasing levels of collaboration something that I think I've definitely noticed recently is that actually lots of cyber criminal networks are all interconnected in a way that we probably didn't realize before and the reason that they're better than us cyber criminals collaborate better than us and there's reasons for that but yeah there's certainly connectivity in a way that probably I never used to appreciate and lots of these cyber criminal groups and networks are all collaborating with each other to an extent that I never really properly appreciated previously fourthly I wanted to touch on it hybrid actors I mean we're probably quite a good example of this sitting on the stage as we were split between national security and criminal to a certain extent and they're generally stove pipes in government and actually they're probably stove pipes in the private sector and elsewhere and there's these structural gaps in our system whereby we deal with something as a criminal threat or a national security threat and actually we're seeing lots of what I would call hybrid type actors that sit in the middle and that makes delivering a response to those particular individuals really different so what do I mean by hybrid threat criminal actors for example who have the support or the backing to some extent of a nation state and yeah that makes delivering a response to those particular cyber criminal networks really really difficult and that's something that we're seeing more and more in the criminal space and I'm sure the national security guards will say they're seeing it more coming from the opposite direction and then lastly on the criminal side I think Steve just asked me to touch on geographical areas I think we still see predominantly high end cyber criminal threats coming from Russia and former Soviet Union countries but as I mentioned earlier in one of my previous points the definition of those tools mean that cyber enabled crime crime that you used to commit West African fraud type crimes you used to commit on the phone or through mass mail is becoming increasingly easier and increasingly profitable to do online so you touched on some of the specialization that exists within the I've heard it referred to as the underground economy or the cybercrime underground network can you spend a little bit more on that you did mention as an example the cash out business service but there's a whole bunch of others can you touch on a few of those to illuminate yeah I mean you can go on to the cyber criminal underground and by basically anything I was describing it's a bit like that bar out of Star Wars I forgot what it's called now but you can get anything you want in there I don't need to have technical capability in anything I can go in and I can purchase a malware package of somebody I can get a service that will spam that malware to millions and millions of people around the world I can get somebody to check that my malware doesn't hit against antivirus vendors there's no point spamming malware to millions of people and they're getting blocked by McAfee and Kusplersky and all the rest of those common AV tools and then if I want to monetize the access that I get on victims well I'm just going to pay somebody to do that as well so a really good example in how this has proliferated is ransomware I describe it as probably one of the almost like it's getting on towards a perfect crime you know a lot of the areas that we used to be able to cash out was obviously a big vulnerability for some groups because if you wanted to cash out victims in the UK you needed a money milling network in the UK and it involved buying high goods and posting them to various places around the world and it's really expensive for criminals to do that nowadays I'll just drop some ransomware on your machine and just wait for you to pay me in Bitcoin and that's how this proliferation is getting is scary in a way and I would agree with all of that and I guess what I would say is I think it's been even more than just the maturation of the cyber crime marketplace like any marketplace that starts out build your own and then you find other people that know how to do something you don't know how to do and there's some collaboration and then there became carting forums where there were sort of closed communities that you had to know about and be vouched into in order to kind of enter the criminal underground the explosion of mass market crime as a service I think is a direct result of the emergence of torrent and services so now not only do you not have to be a technical person to know how to commit a cyber crime you don't have to be a technical person to hide yourself online and find the people who will sell you these services so we see places like Alpha Bay and before it's Silk Road having these mass bizarres and it's true a lot of what was on those market places are the sorts of things that are hard to buy legally like illegal narcotics but there was also a surprising amount of cyber crime as a service on those market places and it's really made it available to a mass market with a very low skill you lower that barrier to entry to becoming a cyber criminal to almost zero because it's very very easy to access these very public market places yeah and I think you narrow the gap of impact so a good example of that is DDoS attacks I think probably ten years ago there's probably only a few intelligence agency nation state type capabilities in the world that could knock a country offline and a few years ago some kids sitting in their living room in the UK did it and they're like 17 was that MRI you're referring to? yeah there was when Liberia, the entire country of Liberia got knocked off line and some other kids attacked a DNS provider and knocked a fair bit of the eastern sea border the US offline for a period of time as well from their living room in the UK and so you're seeing in narrowing that gap of impact that these individuals can achieve the proliferation of these high end tools and techniques wow so Mick so given the global nature and also the increasingly high impact of some of these schemes and capabilities are nations successfully working together to do what this issue? that's a qualified answer Steve I mean we are working together that's for sure we've got a representative from the UK sitting right here who's embedded with the FBI that's something in and of itself like I said my section has been pursuing international capacity building for well over 15 years where people including some of the people in the audience here who used to be part of my section would go out and train internationally to build the capacity in other countries the Council of Europe has been a leader in that as well but honestly even though we've had operational successes things like going back to when Luke was with CSEPS and did Game Over Zeus which there was a lot of international cooperation those operational successes really relied on individual relationships and our willingness to partner and I think the movement that I'm seeing is that countries are starting to realize we can't always just rely on the goodwill of another country we need the ability to effectively investigate international evidence using our own domestic tools right now if you want evidence from the United States there's somebody who has evidence at Google and they want that evidence in the UK they've got to come to the United States and ask for us to help and we do our best to help but we have to teach them concepts like probable cause that I'm guessing there are probably a fair number of lawyers in the audience today that find that a difficult concept themselves try teaching it to someone who doesn't participate in our legal system at all so we are reaching for ways to improve international investigations where countries have to realize I'm willing to give up some of my own sovereignty my own interests some of the control I could exert in order to make other countries more effective in this space so you've seen the United States do that we're actively negotiating at the Council of Europe for a protocol to the Budapest Convention the Budapest Convention is the leading multinational instrument that helps advance cyber crime and electronic evidence are there ways that we can sort of let another country use its domestic process to reach evidence in the other parties to the convention and there are now over 60 parties to the convention it's been growing rapidly in recent years and including in Latin America in Africa in Southeast Asia there are now parties from all of those regions of the world it really is becoming a global convention I guess the other thing I would say is some of you may have heard that not too long ago just over a year ago we passed the cloud act which is an incredibly significant new authority for the United States to enter into bilateral agreements to lift all restrictions at the Council of Europe we're talking about things like maybe they should be able to get subscriber information just to find out who's behind an IP address in the cloud act we would actually enable a partner like the UK to use their own wire tapping, pen trap authority to lift all restrictions because they have a system that has a strong human rights record that has adequate protections checks and balances, high standards for privacy and so why shouldn't they be able to use their own system to investigate domestic crime without the US law standing in the way so I think that's really the change that I'm seeing that has to do with the rule of law is sort of countries recognizing others authority that's fascinating so let's pivot a little bit into the role of victims in investigating and dealing with these issues we're talking about these investigations kind of in the abstract forgetting the fact that perhaps that there are major victims here and elsewhere in the world Luke can you help us to understand the role of the victim organization in responding to a cyber incident and what must they consider in cooperating with law enforcement to get things moving so that we can begin to figure out who's behind it and impose some consequences Sure Steve so it's complicated and not every victim has the same level of experience working with law enforcement if you're in the US financial sector you're pretty battle hardened at this point especially the major players they've been around the block they've been part of information sharing and analysis centers and organizations so they have an idea about how to cooperate with each other and how to cooperate with the government but even they struggle with these issues because to continue with that example they are heavily regulated and not all parts of even the US government have the same lens or approach when they come at them some may think of them less as a victim and more as a party that was derelict in protecting people's private information and so it's complicated for them there's a lot of fear and panic I guess for my role what's particularly gratifying is in major cyber attacks you are basically meeting them in their worst moment you don't hear the word existential too much for too many things I suppose terrorist attacks can rise to some level like that but cyber it's out there quite a bit as a word with our clients where if this really happened and it hit our entire network and took everything down and we were offline for days that might be the end of us and when they're thinking as individuals it certainly could take a lot less than that for it to be the end of my job and my career so they are afraid and it's a chance to help them think in advance ideally through these issues what does the decision tree look like people mentioned you know Mick and Will mentioned relationships and I really think that over the course of my 18 years around these cases the technology has always been interesting but it's always changing and it's in the background the bad guys have been interesting and they are changing to some extent and that's always there what has been a constant formula for success are relationships relationships of trust when I was in the CSIP's role or in the NSD role I would go to a place like the European Cyber Crime Center in the Hague on an operation like Silk Road or Game Over Zeus and we would sit down with 16 or 18 countries and it was all about how can we find common ground how do we trust each other with certain things and certain levels the company, the private sector victims are wanting to do the same yesterday I brought a major client of ours over to see Sean and the team they wanted to understand how would a national security threat play out so we brought the general council and their top reports in and they spent time with us explaining how they would work what kinds of things are people going to come in and raid jackets and take over our building are you going to issue a press release that blindsides us and makes us look like a fool in front of our shareholders, regulators and everyone if we share something with you how will you use it who else will you share it with if we are hit and we want to go out and do our own dark web activity or searches or gather threat intelligence where are the lines there what can we do and where do we cross it so these conversations are best had in a time of peace and not in the fog of war so that's a little preview of the victim issues how common is it that you're working with a client of yours prior to an event where you're advising them on incident response planning, developing these relationships is that the outlier that you just described or is that becoming more common for the more sophisticated clients it's becoming more normal and people are realizing they're in this digital Darwinism era where if you try to hunker down and stick your head in the sand it's not going to work out it's not a long time game so the first thing the GC said yesterday is we're a when and not if company you know it's going to happen to us if you got anything to sell any valuable data valuable intellectual property or anything you're going to be on someone's target list if you're not being targeted I think you need to find a more profitable line of business or something because you're on the menu you're on someone's target menu and it's a question of how high up the food chain you are as a target and so they came in with a right attitude if I can just add to that I mean when people ask how do I prepare for a cyber attack the first thing I say is hire former CSEPS attorney that's in the private practice now but the second thing is go to cybercrime.gov and read the best practices for victim response and it really does talk about building these relationships in a time of peace be ahead of it, do those things ahead of time and think about what information you'll be able to share with law enforcement before you're faced with that very vulnerable feeling of I've already lost control of this information I'm not ready to let anybody else have it even the FBI it's very hard to think clearly in a business continuity disaster type of scenario continue to pull this thread a little bit slightly out of order on my roadmap here so in terms of an effective organization managing their incident response process so back to Luke how do you recommend that they interact with some of these third party providers perhaps they don't have a huge incident response team internally maybe they're working with a company that does this for hire or maybe a threat intelligence firm and then also the role of firms like yours outside council how does that whole system work so I think the best prepared of our clients they have those vendors the Mandians, Crowdstrikes or whomever of the world they have them retained through us as outside council or their preferred outside council for really for two reasons one is attorney client privilege is most obvious because now when you announce your data breach at 4 p.m. you will be sued by 4 p.m. you will be sued in a class action in Florida 7th circuit or the 9th circuit 11th 7th or 9th you'll be sued by 4 p.m. sorry so if you aren't ready with a privilege mentality and an evidence preservation mentality that an attorney brings you're hosed to use a technical legal term the other thing is these are high speed excuse me internal investigations the prior generation of attorneys that came to this space were predominantly privacy and compliance lawyers they are very important and still needed we have them on my team but they were more ready to receive the facts from the forensic team and say okay you have to make notification in Iowa Tennessee and Belgium we'll send out the letters I don't want to demean them but their major was not how do we craft the forensics to prove the negative if we know that APT 10 doesn't really target PII in the 75 prior FBI investigations because we called these guys and they told us that because we knew who to call and what to ask we know that we can have a good position with auditors and regulators that they weren't looking for those droids on these servers they were after our IP that's their plan so understanding the threat space being able to speak about forensics and design an internal investigation to possibly create better legal options on the back end is what my breed of cyber lawyer does and then the privacy gurus can take that and say alright now with this more nuanced view on the facts we can see here and here where we have notification considerations and here are the risks okay so your mention of perhaps APT 10 and understanding what they're after is a great lead into this question which is attribution figuring out who is behind a particular act how's that are we doing better at that is that becoming easier or harder getting harder it got a lot harder on the criminal side when TOR became standard for a lot of crimes but even when we were dealing with proxies and there were proxies far before TOR the electronic trail could be difficult but we still had the money trail and I think a lot of cybercrimes in the past 10 years or so were solved through following the money trail and not the electronic trail and that is going away now going away now too with virtual currencies so when you have these mass market ways to anonymously transfer value it is enabled new crimes like ransomware and it has made investigations more difficult but I leave you with the impression that it's hopeless there's no need to report a cyber crime you can just hire Luke and feel like you can sleep well at night at that point because you've covered yourself the truth is that we are still solving significant cyber crimes just take a look at the last two weeks we announced one of the largest data breaches in history was conducted by a charged individual in China the Anthem Breach we took down one of the largest referral services that were referring people to the dark net and were taking money according to the allegations in order to drive traffic to illegal narcotics sites and all the way across the Atlantic in the Hague at the European Cybercrime Center it was announced that there was a group of six different international bodies or countries that collaborated to take down the Gosnym criminal network so you now have cyber criminals who are being prosecuted not only in the United States and we still are prosecuting a significant number of them in the United States you're seeing cyber crime prosecutions successful ones and ones with significant deterrent value in countries like Moldova like Ukraine like Bulgaria so these are places that really weren't prosecuting cyber crime to the same extent 10 years ago and yet today because of capacity building they're able to accomplish these things on their own that's great Will, do you have anything to add to that? Yeah, I would agree that absolutely I think attribution is getting more difficult in general I'd say there's a number of ways that we come by that collaboration is clearly one of them data is incredibly important really, really high in cyber criminal might not have been such a high in cyber criminal be careful 3, 5, 10 years previously and then overlaying humans with what we know human intelligence what can people tell us what can the lower level individuals tell us in this and so as the scale of this gets bigger attribution in general gets more difficult but actually we're maintaining an operational tempo it is still possible in this space and delivering operational results is still very achievable Great Well, Sean's been quite quiet to fill in the panel up to this point but it's time to pivot to him so far we've discussed cyber crime but recently the Justice Department's been making news by seeking criminal charges against individuals acting at the best of a foreign government so Sean what's the thinking behind this? Thanks Steve I would actually say it hasn't been that recent now it's been 2014 when we brought charges that's five years ago it's not too recent in my mind so we've been doing this for a while I think back then I'd say pre-2014 we brought our first case against nation state actors we charged five members of the People's Liberation Army with among other crimes intellectual property theft through hacking prior to that time I think the bureau and the department had gotten really good at figuring out what these actors were up to and watching and figuring out their modus operandi the issue was that is very valuable and I think that's a traditional CI approach to a national security threat a counterintelligence approach to a national security threat at some point in time I think leadership rightfully so said okay the traditional CI approach if you watch the adversary learn what they're doing figure out what their network is wait for them to go visit a dead drop in the middle of the night and then maybe figure out who their sources are was not working in the cyber realm because it sounds cliche but people are things were happening at the speed of cyber and so you would watch these actors and figure out who they are by your intelligence collection methods and while you're watching them they're sucking out millions of dollars some even go higher than that of intellectual property and other valuable information from our private companies clear defense contractors and things along those lines so pre-2014 leadership aside there had to be a switch we'd still take that CI approach and it was still necessary but we would be better at figuring out when to switch to a disruption and deterrent effect operations and so one of the methods because the department of justice in the FBI or law enforcement agencies was thinking we should be bringing the rigor and intensity of our criminal investigations to the CI threat and using that as one of the many tools to disrupt and deter these actors so that's what we decided to do number one we'll try to arrest these hackers granted when they're either intelligence officers or working for an intelligence service from effectively a state haven in their home countries we do have a very low chance of arresting a lot of these actors and we recognize that but we felt that there was still a disruption to turn effect from bringing criminal cases or just at the very least taking an investigative approach a criminal investigative approach to these actors and there's other ways to disrupt them and the idea behind that is number one you give you can test the space you don't make it a zero cost approach to these hackers which it kind of was spearfishing emails and entities or trying to watering whole attacks and things along those lines with very little risk of being caught identified or disrupted and so now we're just raising the cost contesting the space it gives the network defenders some breathing room it we help pitch information from our investigations to those network defenders so they can better defend themselves but with regard to the adversaries and their decision making calculus you don't have to insert in their minds am I truly as anonymous as I think I am will my future be as carefree as I hope it would be can I travel these places when I want to maybe if I want to give up being a hacker someday and go work for a reputable company can I do that these are all things that we're now putting in their minds and hopefully as part of other efforts we'll discuss later on starting to change that cost benefit calculus and having a deterrent effect with a lot of these adversaries so what is the relationship between the types of cases you're charging and the norms that we heard about earlier either from the 2013 or 2015 group of governmental experts report and then also kind of related to that are we concerned that other nations will want to criminally charge US government employees that may be doing things that violate their domestic laws so what's the relationship there between what we charge and expected state behavior right so if you actually take a look at a lot of our cases that have been public that we've done out of the National Security Division working with US Attorney's offices around the country you actually notice that a lot of those cases probably track many of the norms that have been out there through a group of governmental experts and what not or through other international kind of consensus building entities so for example we have had a lot of cases targeting the theft of intellectual property for the benefit of foreign foreign companies by state sponsored actors that's something that I think we in the United States have been talking about for a long time and a lot of our partners and allies have for many years now been saying the same thing which is you know granted espionage it's traditional tool statecraft it's everyone does spy I think you know President Obama said at the time but what we there we think there's certain red lines as an international community that should not be crossed and that was one of them you should not steal intellectual property for the benefit of your own your own companies or state-owned enterprises things along those lines so you see a lot of our cases that are coming out publicly do tend to track those norms other cases you know vows should not attack critical infrastructure critical infrastructure in peacetime we have several cases where we have actors I can think of the Iranian DDoS attacks against the financial sector that were done on behalf of the Iranian sorry the Islamic Revolutionary Guard Corps same thing these are types of cases where we're saying that is the type of activity that we believe countries should not engage in in peacetime and you'll see a lot of our national security type cases track that to help reinforce those red lines both with our partners and in our adversary size and in which case we're making a policy statement that the U.S. does not itself do those things yes absolutely so that brings me to your second question which is you know do we have a fear that our own operators will be on the receiving end of criminal charges in other countries so I go back to the point of what I said I think that there is all countries spy that's something that is normal and we hope that our countries are gathering intelligence to better protect ourselves down the road but what our intelligence agencies do not do are those types of things that we are saying other countries should not do which is attack critical infrastructure in peacetime steal intellectual property from a let's say a Chinese aerospace giant and hand it over to Boeing and they do not do those things and so I think we tend to stay on the right side of those lines or what we believe are the lines but at the end of the day personally I have a hard time buying the fact that foreign governments or adversaries maybe are not charging our operators out of some sort of trying to recognize norms I have a feeling if they can identify the adversaries they would engage in their own cost benefit analysis and charge them regardless of what the Department of Justice and the FBI are doing so we are on the little bit of the home stretch for my guided discussion that will pivot into question and answer but a couple more things so back to Luke for an organization that has been the victim of what might appear to be a state sponsored attack are there special considerations that they would have in mind or need to be able to navigate in the incident response phase that would be different from if it was a pure financial motivated criminal attack and especially if you're working with the Justice Department and there's discussions around whether your client would be willing to be named in a public charging document as being the victim of a state sponsored attack a lot of considerations one is do you have cleared personnel people with clearances or access to them to be able to go into classified space and find out a little bit more whatever the government might share about the nature of what's hit you you may be served with national security process normally as a victim the approach is a voluntary one as far as that can go but it's conceivable that your client could receive what's called a FISA warrant or issued pursuant to the foreign intelligence surveillance court or an NSL national security letter which is the rough equivalent of a criminal subpoena and there are specific rules about who you can share these with and how you respond and you can't just run off to your board and tell everyone in your HR department about this you could be crossing legal lines there there's some sophistication required in preparation for the response to this in terms of the media aspect and what you're asking about Steve it is a big consideration to have your client named in an indictment that points a finger at a major foreign country because you may have business operations there or dealings there and the retaliation risk could be high and so a lot of our clients are very worried about those issues they want to get justice so to speak or at least make it better for their stakeholders they want to do the right thing as a participant in critical infrastructure and broadly from that standpoint but they're also worried about the collateral damage and retaliation risk just on the going back to the question about attribution and you talk about whether you as a company have been hit by a particular nation state I think when you suffer a major incident as a company that could be a national security or criminal risk out there I think it's on that attribution perspective it's taking a longer period of time to figure that out and actually with the emergence of these hybrid actors could you have actually been the victim of both at once and how do you deal with it as a business are you going to be subject to national security process or criminal process or you know actually or mixture of both I think that's getting a little bit more complicated I think those lines in the road are no longer as clear cut as they used to be right okay well I'd like to wrap this up maybe back to you Will with the final question that Sean made a pretty compelling case for why the U.S. has taken the attack that it has with trying to deter malicious cyber activities by states with criminal charges what are some other ways that we can achieve deterrence in cyberspace I think there's an enormous toolbox that we have at our disposal here to kind of like counter that maligned behavior and that can really be anything that associates a cost to that particular piece of activity so for me it's not just about delivering specific tools like indictments or U.S. treasury sanction designations it's about two things firstly it's about using a blend of those tools all at once so making your response multifaceted and secondly it's about doing that in a really collaborative way we are finding that we are conducting all of this activity as coalitions of people that all believe in these types of norms in cyberspace so I don't want to go into particular types of ways that we can counter this but I'd say that actually as a community moving forward we're using a blend of those different tools all at once and we're doing that in a much more collaborative international way if you don't do that I think we've found particularly in the criminal side that you risk potentially driving an evolution in the sophistication of the adversaries tactics if you hit them with the same thing every time they will evolve to notify that tactic that they can use against you Sean quickly and then we'll move on to questions I think that's a fabulous point I just draw some public examples of this recently we've had two instances recently in the national security context where we've had more of a coalition of international an international coalition of like minded countries have come out to speak about some malicious state of monster behavior in cyberspace the first one was in October 2018 there was a Russian operation to go after anti-doping agencies and to put information about athletes out on the internet somewhat was kind of tweaked in a way to make it look worse than it was and that was through a cooperation of I think the United Kingdom Canada the RCMP were a very important partner in there and also the Dutch military intelligence you know go back two years ago you would never have seen that type of cooperation in the national security threat especially with regard to a foreign military intelligence agency cooperating with an American law enforcement agency to counter one of these threats and I think it's a really good, useful example and then more recently the APT-10 in Diamond Sur probably about 12 or 15 countries that came out together with statements not just the Five Eyes, the US, UK, New Zealand and Australia but like other countries in Europe and Asia that came out said this activity emanating from China is unacceptable and we think that it should cease and so I think that's a very positive trend Great, great point. Okay, so we have probably about 10 minutes for questions and we have microphone holders and so because we are being live-streamed and also so the folks can hear please do use the mics so I see a hand here and if you'd make sure that you do ask a question and not give a speech and then we'll have one of our speakers address it and then we'll move on to the next question so we won't all five try to answer your question I guess I address this primarily to Michael in terms of law enforcement response to sophisticated cyber crime, what are some of the investigative tools that are most important in solving these crimes? So if I can change your question just a little bit it's not so much the investigative tools it's what do we have to do to make a difference and make an impact on people not doing this crime I think what we need to do is solve a lot more than what we are now I'm not a criminologist but I'm starting to be educated on some of the literature that says that the certainty of punishment is a lot more effective than the length of punishment I've put people in their 20s before the court to be sentenced and they've received 10, 12, 14 years I'm increasingly convinced that that may not have been the greatest deterrent message we could have given rather than have 10 or 15 of these individuals before the court so that they were all being punished for somewhat less amount of time the problem is the easy cost effective thing for congress to do is increase penalties and that shows they're serious about cyber crime I think to be serious about cyber crime we've got to have a lot more cyber investigators we've got to retain the experience once we have maybe even with increased pay and we've got to and we've just got to have a lot more of them great I see let's see I think you had your hand up earlier right so we'll come back here this is a quick question what about if a victim got an email and the sender says you know I have your password and your password is the following and if you don't give me $100 I'm going to use it to steal your identity what should be the answer or the reaction or what should the victim do in such a case can I ask our moderator to step into the world because I know the FBI has guidance on when you're being extorted over the Internet we typically don't recommend paying extortions that's the official line if they say they have your password I would immediately seek to change all your account passwords on that account and any other account that may resemble it but and then also report that report that incident to the Internet Crime Complaint Center because we are aggregating a large number of small individual incidents and losses so that we can put together larger cases but I guess it depends whether the person is offering proof that they're in your account or whether they're just blowing smoke but definitely have good practices on your two-factor authentication if your email service or brokerage account offers it, use it it's quite a great counter measure for preventing that sort of thing but we don't recommend paying ransom because you're fueling the very activity that we're trying to prevent and let's go to Mary and then we're going to come here next Hi, I'm Mary Greer I work at the ABA Rule of Law Initiative and I wanted to follow up, Mick, on your observations on the evolution of technical assistance at least over the 15 years that CSIP's has been providing such great services do you have any reflections for us and I want to include all the panelists on what the constellation is what's been more effective than not typical training models capacity building the multi-agency approach and Will had made the good point that it's beyond the enforcement rubric and I'm wondering again within the confines of rule of law technical assistance other things that we should be doing or thinking of doing or stop doing I think the first thing that I would say is that I do think that capacity building is incredibly important in fact the State Department has been wonderful in using some of the money that they're allocated to actually fund Department of Justice attorneys to go out and do that we've started with one or two here or there mostly focused on intellectual property but we have just created the first global law enforcement network of international computer hacking and intellectual property liaison prosecutors that's a lot of words but we just call them the Glenn of iChips and the Glenn of iChips is going to be a dozen federal prosecutors fanned out across the world in Panama, in Brazil in Croatia including one that will be stationed at the European Cyber Crime Center specifically to address Eastern European transnational organized cyber crime working collaboratively with the collection of the European police and prosecutors who gather there in the Hague so as far as what we should be focused on I think one thing is we tend to do more comprehensive training across all of the necessary levels investigator, prosecutor and judges I think judges are the ones getting the least amount of training right now I think investigators actually probably are getting the best training getting some training and we hope to improve it significantly with the Glenn of iChips but judges are getting less training and ultimately all of these cases if we're successful are headed to a court so I think judicial training is critical on the capacity building front I mean lots of different countries lots of different agencies around the world have a range of different really really excellent capacity building programs one thing that I think we're probably not very good at however are the NCA in a particular country providing a forensics course that the FBI or the DOJ provided to a similar group of people two years earlier so that's what I say on the capacity building front and secondly it's not all just about enforcement like cyber crime is the one crime of all of them that you're not going to arrest your way out of this problem and actually 98% are going to arrest themselves by just undertaking some really basic cyber security hygiene, cyber security measures and actually that remaining one or two percent are going to be victims if you have good cyber hygiene or not so we should be utilizing operational success is better to put that protect message out there so yeah the two halves of the of the same coin there Thanks Will, okay so we're coming here Hi one question that I had for both on the criminal side and the nation state side is this question of commercialization so clearly it's reducing barriers to entry for conducting cyber crime as well as nation state attacks but do you see this in your work as introducing any sort of opportunities for you to improve your investigations or to better hold people to account so at Citizen Lab when we're tracking nation states targeting dissidents in civil society commercialization is sort of very helpful in that once you've seen one instance of an attack or one actor using a commercial tool it's much easier to map out the entire extent of that activity and uncover other attacks Sean do you have any views on that? Yeah I think you raised the advantage of commercialization for being able to figure out what's going on and who's doing what but again I think it kind of goes back to the point made about cyber criminal underground and these people who established their niche these companies are now enabling I think foreign intelligence services or law enforcement services all around the world to get into the game of cyber investigations and you know some of them just don't have the same rule of law protections that we like to have that we have here in a lot of other countries around the world I think that's where that friction is posing problems significant problems. Yeah I think it's really interesting I mean it allows the various acts as whether you be a criminal or nation state to go in and do what you want to achieve at a low risk and a low cost but at the same time it presents some sort of opportunity to us as law enforcement or investigators across the pieces and actually if you have visibility on a criminal service that they all utilize then obviously that's a real investigative advantage. Okay I think we have one more question in the back I saw a hand earlier yes you and then this will be it. This is a follow-up question on ransomware there was an NPR piece this morning on Baltimore and the whole city being shut down by ransomware and Atlanta being hit earlier Baltimore does not have the resources to deal with it how do you they're even looking at possibly going back to paper for house transactions what recommendations do you have for cities with limited resources? Well ransomware is a huge huge problem and we are seeing that which would be presumed to be a cyber crime offense maybe that would be aimed at the normal individual starting to hit critical infrastructure owners and operators, towns that sort of thing. DHS and FBI have put out ransomware advice out there on U.S. and other websites I know with Baltimore our folks are working on that with the city it's a major problem and organizations need to be fostered to be able to deal with that issue so it's hard for me to diagnose that from afar anybody else want to weigh in on the problem of ransomware? I would just say there are many many resources out there some of them are more expensive than others but certainly one that is heavily involved when there's something like that happening is the Department of Homeland Security it's less of the law enforcement concern where we are concerned about trying to find out who did it but it is DHS that if you want to give the analogy to the policeman and the firefighter they're the ones who are trying to assist in the recovery of an event like that and so really it's our Department of Homeland Security I think that would be most involved just one word on that from the private sector experience is you know preventing it segmenting your network all those things but the real coin of the realm for our clients no pun intended has been ability to restore from backups you know and if it's just a mirrored rolling backup that's going to be encrypted as well so it's expensive to store a snapshot and to keep a periodic snapshot but when our clients have the snapshot they can restore back to that snapshot and they can come back and they'll lose maybe some days or hours and that's expensive for a government to think about but the alternatives are far more expensive to me so I think maybe it needs to be an investment in that along with all of the hygiene and segmentation steps cleaning up RDP connections dealing with phishing and trying to raise the game on all of that from a prevention standpoint but having the backups is a fallback Paul thank you very much panel and thank you Mary and Linda for 20 years and suddenly everyone's listening we care a lot about internet freedom we care a lot about openness we care a lot about privacy we care a lot about security and that creates interesting tensions and we are here today to talk about some of those approaches kind of the considerations that people are taking into account and just how we should be thinking about this as we go forward in order to create the internet that we all want so we will and then we will have a conversation and think of questions we will open up for Q&A maybe 10-15 minutes before we wrap so first off we have Dr. Andrea Limago who is the I want to get this right Chief Social Scientist of Virtue and thinks about the social side of these technologies how do we insert the people we've been on a panel before that ended up being absolutely fun because we were there to talk about cyber security and we all ended up talking about people which is actually I think the right thing to do so go ahead thanks very much it's great to be here and so like Heather said we're going to a software company that focuses on data privacy and encryption and the technology is very important obviously but the interaction with humans as well part of my work with a background in global relations in conflict studies is how some of those technologies fit into the broader global trends that are going on and one of the big areas that I've been focusing on recently for the last few years is the rise of digital authoritarianism and so over the last 8 years internet freedoms have been on the decline I feel like freedom of the net from freedom house based on their metrics and so what that entails a variety of means and I'll talk about some of those how governments are using these tools and then how some of those tools are spreading to non-state actors as well but basically it was tightening the control over users data I recently saw in our article a few days ago the hegemony on communication and I kind of liked that notion because that is what they're trying to do complete information control really by whatever means possible and a lot of it is some very creative means that just weren't foreseen even several years ago and so the way that this is manifesting is everything from elections which we hear a lot about in the US but to be clear the election interference via digital means is only one area that's getting the interference and that's what's going on in over a dozen countries in 2016 so it continues to expand in that area other areas in addition to we think about the cyber tax that kind of at least seeped into public consciousness as far as what's going on to infer and jump privacy but there are other areas that are going on in addition to some of the more cybersecurity issue areas that go into the legal realm and the realms of data localization and a lot of those laws are increased which basically requires local data storage in those countries and then with some of the different regulations as well also entails government access to that data when they want to you know they can say under only certain conditions of data they can demand it and so that's led to issues such as Apple storing some data in encryption keys in China it also has led to in Facebook the recent manifesto saying that we're not going to be adhering to some of those localization laws of authoritarian regimes and so there's a big discussion going on both amongst governments and amongst tech communities and businesses and corporations for how they're going to respond to those localization laws and it's not just China and Russia and this is something that we need to be really clear about across all these mechanisms from cyber tax and cyber crime that we just heard about the proliferation to non-state actors of these tools really is just it's expanding significantly and so the same thing happens when it comes to data localization Taiwan and Thailand and Vietnam are two of the more recent ones that have been passing various laws as far as either government access to that data or government storage of the data and both of them especially the Vietnam law really mimics a lot the Chinese laws that you see and so you do start seeing the rise of the rest going on across the globe when it comes to digital authoritarianism and so what that also is leading to is something called the splinter net or balkanization of the internet so instead of the global a free and open and global internet based on internet freedoms we're really seeing it breaking apart into national cyber sovereignty kind of internets that are going on across the globe and to take it to the extreme we've got the Great Wall the Great Firewall from China but they also have Russia recently passed a law to try and ostensibly cut off access to the rest of the globe Venezuela is considering similar legislation right now and so these are the things that what happens in China doesn't stay in China what happens in Russia doesn't stay in Russia these are expanding and so far there's been a bit of a vacuum on the side of democracies to push back and whether it's I know Norm was discussed about this morning the cyber norms has fallen apart a bit at the governmental level there's still some efforts there but at the UNGGE progress was being made and then fell apart and so some of the tech giants are starting to try and fill in and so absent democracies doing more some of the corporations are actually trying to fill in that gap to create the appropriate norms and the appropriate behavior of what to do in cyberspace and how to preserve security and privacy and so that's I think an interesting trend that's going on as well and just as far as on the one hand a lot of what my work does seems to focus on some of the doom and gloom that's going on from the surveillance society and the spyware and everything along those lines and we can't be naive to those threats and I used to work in the DOD I'm very familiar with the threat landscape that's going on but at the same time we also need to be aspirational and try and still build the internet that we want it to be and try and push toward that vision of the free and open internet and for me and I can get into some of these more later I think it's a combination of the technology as far as usable security and making it much more usable and accessible for people and just raising greater awareness and we can also talk about some of the bad information that also comes out of the tech community as far as what's secure and what's not but just following some basic security best practices can really help out 90, 95% of the people at a minimum but in conjunction with some of the norms that could be pushed forth and then some of the privacy laws we can talk about GDPR aspects and what's going on in the United States right now is we've really left a vacuum globally as far as leadership and establishing what the guidelines should be for data privacy across the globe and so we've got California coming out with their law that's at the farthest that we've got in the United States different states again are coming on doing something either copycatting or doing their own version it's very much a piecemeal patchwork is usually the analogy given and so we really do need some U.S. leadership then where we can push forth for data privacy law at the federal level to create some efficiencies and innovation and really protect data privacy also can serve as a model for other countries around the globe great, thank you next up Bill Marzak is a research fellow at Citizen Lab the co-founder of Bahrain Watch and a postdoctoral researcher at UC Berkeley thank you so unfortunately I didn't have the foresight to print out my remarks I'll be reading them off of my phone it's very modern I'm not checking Facebook or anything like that so at Citizen Lab I do work looking at commercial spyware so these are companies which are selling products to law enforcement to intelligence agencies to essentially hack into phones and computers of targets and the way that law enforcement sort of talks about digital investigations these days is they say there's a going dark problem and this sort of spyware industry the surveillance industry is offering the solution they say well, yes, there's the going dark problem everything's encrypted so what do you do? You need our products to hack into computers and phones of your targets and see what's really going on bypassing the encryption so the companies spend a ton of money on research and development finding ways to get into the latest iPhones the latest computers running the latest software and typically the way that this works is that they'll send some email or some message containing a malicious link or attachment the target opens up they're infected and then everything on the computer or phone is subject to surveillance including encrypted calls, encrypted messages as well as being able to turn on the webcam and microphone of the device in order to spy on activity in the vicinity and you probably also heard from the tech news a couple weeks ago this crazy story of the WhatsApp hack where your phone could be hacked by receiving this missed call from a malicious a malicious caller and what was happening there is even if you were asleep let's say they could send you a missed call the call would ring for three times and if your phone was on silent while you were asleep you wouldn't hear it and the call would not show up in the call log afterwards so it was sort of like the perfect the perfect hack and one of the components of this story is that there was a human rights lawyer in the UK who was targeted by this and the reason he was able to notice this is because he was sleeping with his ringer on so he would awaken to these weird missed calls from unknown numbers and his phone would ring and then it would disappear from his call logs and he questioned who am I dreaming what's going on here but it turned out that at times on this a leaker basically a spyware salesman from inside NSO Group collaborating with NSO Group felt he had been stiffed on an earlier deal and didn't get his commission so he decided to leak some details about how this surveillance works and the company the spyware company involved was allegedly this company NSO Group based in Israel that sells this sort of spyware around the world and the tools themselves they just sell it to governments who operate the tools but interestingly enough this human rights lawyer is sort of famous for filing a case against NSO Group so he was suing NSO Group at the same time he was being apparently spied on by the company and of course NSO isn't the only player in this space you've probably heard of others like Fin Fisher, Hacking Team, Cyberbit and this is way more powerful than surveillance getting wiretaps because a wiretap you hear what someone is saying and you get access to the phone whereas with this sort of hacking you get access to the phone and what do you have on your phone it's your whole life it's your calendar your transportation with Uber and stuff like that your emails it can even turn on the webcam and microphone to create new data and exfiltrate that and the common refrain among these companies of course is well yes we know this is very powerful technology we sell it only to governments and only to fight terrorism to identify those sorts of crimes but not surprisingly a lot of repressive governments are interested in this stuff too so we've seen governments like Saudi Arabia UAE by this sort of technology countries that define terrorism like Saudi Arabia to include criticizing Muhammad bin Salman that's considered terrorism according to the 2017 terrorism act there so this is obviously an area that's very ripe for abuse in terms of these surveillance tools and the companies of course like to tout the lawful uses of their tools but one interesting case is they said oh well our tools were used to capture El Chapo in Mexico but what we found actually is that so El Chapo one of the things that came out of his trial is that he paid a hundred million dollar bribe allegedly to the former president of Mexico and interestingly when El Chapo's ordered a hit on a Mexican journalist and he was killed in the weeks afterwards his wife and his colleagues began getting targeted with the spyware that the government was using particularly interesting in the context of the bribe you know there's a bunch of other cases we've documented at CitizenLab including one of Jamal Khashoggi's close contacts who was being spied on by apparently the government of Saudi Arabia so here's a question some of you might be wondering is this guy Bill how does he know about all this stuff from this crazy world of government surveillance and intelligence agencies it's supposed to be private these are supposed to be actors that are operating in the shadows well as I was saying earlier in many cases the way that this spyware works the way that you get hacked is you get sent something that you have to interact with in order for the hacking to happen like a text message or a link or something like that the WhatsApp thing is sort of very rare so the whole enterprise here relies on the people who are targeted not realizing that a suspicious message is actually malicious because if they do then they can forward it to someone like CitizenLab or someone like Amnesty International or EFF the electronic frontier foundation who are all sort of operating in this space and then what we do is we can actually analyze what we've been sent in fact there was a famous case in 2016 activist in the UAE Ahmed Mansour got some suspicious text messages prompting him to click on a link about new secrets about torturing UAE prisons I think was the title which is kind of odd to get that from an unknown number so he forwarded it to us at CitizenLab we clicked on it and were actually able to get a copy of the spyware and a copy of the bugs that the company was exploiting to break into the iPhone so we just closed that to Apple Apple released a new update back in 2016 iOS 9.3.5 and of course when the bugs are patched the spyware as written no longer works so this is a way to sort of pressure these companies because once this is disclosed all the customers for the company start saying wait a minute the spyware doesn't work anymore we paid all this money for our system what are you doing so there's a bunch of challenges in doing this sort of work getting in touch with the right person at the right time but this sort of field that I was mentioning when I was talking about the cyber security researchers as I'd call ourselves are looking at this field so it's clear that this is sort of a wild west type atmosphere you know you have governments doing deals with shady companies doing deals with shady governments who are then targeting activists and dissidents one of the subtle points here that I want to bring up though is that there's sort of two markets for these companies legitimate players so we know that some of the spyware is purchased by legitimate law enforcement agencies like for instance the Belgian federal police or the Swiss purchased spyware from NSO group in addition to other governments like Saudi Arabia and Mexico so the surveillance industry tries to cater to both of these types of clientele but it's a bit awkward because you have cases where the abusive customers the reckless customers like let's say Saudi Arabia UAE, Mexico are sort of sending this stuff out to civil society who forward it to us we disclose the spyware things get patched and that hurts both the legitimate customers as well as the illegitimate customers and so one of the things I want to talk about is like alright so that you can think of that as sort of a form of regulation as a last resort right like we're trying to send the message well if you know you target civil society your things will get disclosed and then you know bad things will happen for all of your customers so maybe you want to ensure that doesn't happen but what's a better way to think about this sort of regulation well so the sorts of spyware that we research are called intrusion tools or are classified as intrusion tools under the Vasanar arrangement for export controls meaning that governments who are party to the Vasanar arrangement should hopefully seek to regulate these and the regulations basically will if you want to export it we've got to sign off on it first and the problem sort of here so far has been you know we've had cases like for instance the government of Italy issued a license to the famous spyware company hacking team that said this is a global export license you can export this anywhere in the world and we don't care and you've also got cases like in Israel where NSO groups second customer their spyware company their second customer was the UAE and this was a big scandal apparently according to reporting in Israeli media that Israel issued this export license for the UAE a country which doesn't have official relations with Israel and of course there was recent drama in the press regarding NSO group in Saudi Arabia where it was alleged that some of the outreach from NSO group to Saudi Arabia was done without approvals from the export licensing agency in Israel so there's also been a you know for those of you in the Vostner debate there's a number of areas of pushback particularly among cybersecurity defenders who say well you know you're regulating these intrusion tools and bugs which is great but the problem is that cybersecurity defenders in other words the good guys who are researching this stuff often need to send samples of the spyware or things like this across borders and we don't want to apply for export licenses for that and then you've got problems with industries who say well yes we love regulation but you know we don't like this we want a level approach we want a level playing field and the same set of rules for doing business everywhere in the world so in another frontier in the regulation here has been trying to hold governments to account so like the government of Saudi Arabia maybe the government of Bahrain the government of Ethiopia which are using these tools there was a case right here in the DC circuit Qadani versus Ethiopia just a bit more than five years ago the evidence was pretty much a slam dunk in the Ethiopian dissident in the DC area who was monitored with spyware sold by a company Finfisher apparently to the government of Ethiopia and the spyware of course is supposed to be super stealthy and leave no trace but there was a hiccup in that this guy's Skype calls were being recorded and temporary files were written to his hard drive showing evidence that his Skype calls were being wire tapped so he sued the Ethiopian government under the wire tap act because there's a private cause of action there and of course the Ethiopian government was like oh well yes if there was a tort here it was committed entirely within Ethiopia if there was a tort here it was not committed entirely in the United States and there's the foreign sovereign immunity act and the entire tort rule that says that if the entire tort doesn't occur in the US you can't hold the foreign sovereign liable there's ongoing cases in other countries like the UK where a UK law firm is suing Bahraini activists who are spied on by this company FinFisher and in that case the UK precedent seems to be a bit more favorable because they don't have this entire tort rule but these cases are rare it's kind of hard to get the right kind of evidence at the right time and I think there's been much less success in holding the firms themselves to account so I'm looking forward to talking about more ideas of that once we get to the questions. Awesome, last but not least Robin Greene who is now a Privacy Policy Manager at Facebook to talk a little bit about how Facebook thinks about this and maybe generalize a little bit about the industry that seems fair to me and protecting users and their relationship with law enforcement so thanks thanks and sorry I'm not used to these mics are intense so thank you for inviting me to speak to be able to talk with you all today so when I think about public safety issues and I think when Facebook thinks about public safety and law enforcement access issues in particular we spend a lot of time thinking about exactly how we started this panel thinking about people thinking about the individual Facebook started 15 years ago as a pretty small community of college students who were just connecting to get to know each other to stay in touch with their friends and everything like that and in a little over a decade we have grown into a community of platforms that serves over 2.7 billion people around the world and so when you think about the size of responsibility that we have in order to make sure that we can fulfill our mission to help people build communities to help bring people together and give them ways to share and to organize and to start businesses when you think about the mission to allow people to communicate with one another and to bring the world a little bit closer together the only way that we can do that is by fulfilling our immense responsibility to keep our users safe and that means fulfilling our responsibility to protect the security of their data to protect their privacy and from arbitrary interference with government and to make sure that we are protecting public safety as well and so that sometimes puts us in the uncomfortable position where we are having to work with law enforcement while at times having to push back against law enforcement sometimes the very same law enforcement when they're requesting users data but that's also exactly the thing that animates our decision to move into interoperability and to end encryption across our messaging services to make a public statement as we did a few months ago that we will not locally store data pursuant to any government's mandate in a country that has a bad track record on human rights where we're not confident that we can keep those data safe what we do to make sure that we're protecting privacy and enhance public safety at the same time is apply a really rigorous process for responding to law enforcement demands and so what this means is we basically respond to law enforcement in several ways but in two primary ways the first is in response to lawful requests and because we are incorporated in the United States and in Ireland the way that we wind up responding to requests is if you are from the EU you're from an EU government you're asking for users data you have to go through Irish law that are incorporated and if you're from the US or from outside of the US and you're looking for the contents of users data you have to go through the US judicial process and if you're not from the US government you have to go through the Mutual Legal Assistance Treaty process which essentially enables you to obtain a warrant based on probable cause by working with the US Department of Justice as long as you can to obtain that warrant and then we receive that warrant in our compel to hand over the information but what this means is we receive requests for information and we scrutinize every request so whether it's for contents or non-contents and no matter the country we look to make sure that the request is lawful that it's appropriate in scope and that it's appropriate in purpose and that it complies with international human rights norms but if ever those elements aren't met we will push back on their request the first thing we'll do is go back to the government and work with them to see if we can clarify what they're looking for and see if we can identify a way to make sure that if it's a legitimate investigation and they really need the information can we figure out a way to scope their request in a way that is lawful and properly scoped and in compliance with international human rights norms we will push back and we'll refuse to comply or we will take it to court where possible and challenge the request judicially the second way that we primarily wind up giving data to law enforcement contents data is where there is an imminent threat of death or serious bodily injury we think that it's incredibly important that we contribute to public safety so if we see information on our platform that's suggestive of that kind of imminent threat we want to make sure that law enforcement or the appropriate authorities are able to act on it and so that's the other main instance in which we hand over users information to law enforcement but I think when it comes to increasing requests and the changes of technology and how most information is digital now we wind up seeing certain pressures and so Facebook in particular and tech companies across the board face a lot of pressures around issues like cross-border data requests and technical assistance requests surely with our move to end to end encryption across all of our messaging services we'll see increased pressure on both of those fronts but this doesn't sort of like hinder our resolve we are fully committed to deploying encryption across our messaging services and to making sure that the people who are dependent on these technologies for their privacy, for their cyber security sometimes for their freedom and their personal safety are able to enjoy their benefits but I think there's a lot to talk about when it comes to challenges with cross-border data requests and technical assistance requests and certainly we're facing those challenges all over the world so I'd be happy to dive more in it during the discussion Excellent so I think that actually that's a really interesting set of things to launch off our discussion with and what was really striking to me Robin and especially is thinking about how you balance these competing interests because I don't think anyone up here would say that we are against law enforcement and I don't think anyone up here would say we're pro-surveillance either and what's Bill, what do you think the appropriate balancing test is because I think that you will have having seen these things used improperly how do you decide what is proper and what is improper with those balancing tests and to what extent do you think about it globally versus kind of regionally or nationally well that's a great question I think that clearly we have a lot of experience in enumerating what is sort of inappropriate so targeting activists or journalists simply because of what they say rather than because there's any sort of real crime going on there other than freedom of expression I think though that there's sort of when we're thinking about regulating or thinking about targeting the companies themselves I think though the question is well picking customers and vetting customers before they're doing the sales so while it might be appropriate to sell to like a western governments there's maybe some more responsible use of surveillance than Saudi Arabia and I think the key distinguishing factor there is there's a couple of distinguishing factors one of them is oversight so to what extent are the people who are operating these spy tools accountable to someone and then to what extent do people who are wrongly spied on have the ability to achieve some remedy for that wrongful surveillance like Saudi Arabia where there's really no rule of law there's really no oversight if MBS decides he wants to spy on someone then he calls up his intelligence guy and gets him to spy on them and nobody can say anything about that so I think that in terms of picking customers it would be great if these companies were forced to achieve some or abide by some sort of principles in terms of vetting customers I mean they like to say well we have external vetting of all of these deals there's some external committee composed of experts that decides whether we want to sell or not but I think there really needs to be some requirement or some way to say well look there was no reason this sale should have happened you should have known that the surveillance was going to be abused if you sold to government X or government Y and what do you think about that particular balancing act where do you draw your line on that question there's so much in security as a balance and that's where I would love to start seeing more global norms even if they're not formal which means that they they're not going to be fully there's no repercussions for not adhering to it but can at least establish the baseline we'll know countries that adhere to certain aspects of those norms are not going to be using spyware for instance against their own populations, against human rights those kind of areas whereas we also know we more than willing to not want to adhere to those and there it is going to be a balancing act I can also guarantee that there will be some democracies that are not going to want to have to adhere to those kind of norms because they're going to want to use it for some kinds of national security events that they want to be looking into so it is really hard I think in the case of spyware however and I think you nailed it as far as doing what seems to be the best case scenario and having greater trust in what individual actors whether they're government or organizations how they're going to be using it and and implementing guidelines on those areas and then ideally pushing forth and getting more so to a place where more individuals also can protect themselves against those kind of spyware and so that also the other side of the coin is empowering a lot of individuals also to be able to defend themselves against that when it does happen and having the tools and not making it so complex that they have to be a computer science technology to actually be able to protect themselves and so that's where I'd like to see additional information being shared in that area for strengthening defenses against those kind of spyware because they're going to be misused at the end of the day we should take it as a given that whatever those tools are going to be some actor out there will misuse them You anticipated my next question I think that we've been talking a lot about government access and we should in government use of tools and governments using them badly or appropriately How do you translate those thoughts into I'm not going to say more benign but smaller scale uses of these tools and what I'm thinking of is kind of stalker spyware that can either be sold to figure out whether you're middle school or made at home or figure out where you're a strange spouse is Do we have to think about that differently? Can we think about it kind of in the same vein? I don't know who to point this at but I think Bill Okay so yeah I mean stalkerware as it's called or spouseware as it's also called this is sort of a very kind of creepy thing to think about I mean I think a lot of companies have been doing some interesting stuff here so want to buy occasional collaborators like Eva Galperin at EFF has been doing a lot in terms of trying to get companies at a bare minimum level to try to get antivirus companies to detect and warn you if this is installed on your phone or on your computer because the idea is these companies might say oh well there's legitimate uses here like let's say I have two phones and I want to use one phone to monitor the other phone or something so okay maybe we shouldn't have antivirus software automatically delete this but at least warning or prompting the user to make sure that this is not being done without their consent which I think is the key thing behind the use of this stuff like if the person who you're using it on is not consenting then there needs to be maybe you're a government and you've got a wiretap subject to appropriate judicial approval and oversight okay but other than those circumstances it's sort of hard to see how spying on someone else without their consent is okay and with monitoring children I don't know that sort of gets into kind of a tricky area I could definitely see wanting to find out for instance the location of your child if they're a minor where are they getting into trouble or something and I don't know that's sort of a trickier question which I I don't know if we had good answers no one would want to listen to us this is all tricky and this is all like very hard very hard questions you talked a little bit about actually I think all of you collaboration and the role that kind of outside thoughts plays and I'm thinking in particular you mentioned Eva at EFF another organization that I know at least I have worked a lot with is National Network to End Domestic Violence which looks a lot at this from that stalkerware perspective hence the question and I know that Facebook has a program where they bring in folks like NNEDV and others to kind of give an outside perspective do you want to talk just a little bit about the importance of that yeah so unfortunately I'm probably not the best person at Facebook to talk about this because I focus very squarely on law enforcement access and data protection issues we do take these kinds of issues really seriously and the safety of all of our users is one of our top probably our actual top priority and so we do have you know global safety summits where we bring in different stakeholders to really make sure that we're hearing everyone's perspectives as we develop our policies around public safety but the best person to sort of like talk about how those summits work and how that you know policy making process works probably are the people who actually do all right wasn't sure how involved they were in the kind of law enforcement side of things so Andrea do you have thoughts on collaboration and building these global norms that you were talking about yeah and I think one of the comments that I took away from the last panel was the criminals are doing a better job collaborating than we are as defenders and I think that's great hearing that coming from law enforcement because it's what I always think that and it's nice hearing someone else validate your hunches so across so many different areas we could collaborate better there's so much innovation that can still be had on the defensive side that I think we're just really starting to do it because we've had the pendulum swung so far towards locking down the data and that we're now not able to reap all those benefits of actually unlocking that data through collaboration whether it's through you know threat intelligence sharing to get better pictures of the threat landscape for you know for banks for fraud or what not or just for the IC sharing data in that area but doing it securely enabling that kind of collaboration to happen there's so much room to improve in that area where at that intersection of technology and people where that could actually happen and so even if we had the technology which does it you know existence increasingly there to help secure it and enable some of that collaboration we now have to get over some of those social and cultural hurdles of well you know it's my data and I'm not going to share any of it with you and so if we can do more of that data sharing we can really address a lot more of these threats a lot which then gets into some of the norms area where you know just sharing some of the lessons learned doing a bit more of the collaboration you know across the globe and you just recently example for myself you know I've been working a bit with a woman in Australia and just the different perspectives that I think we each bring to some of our side conversations is phenomenal and get like the encryption wall that passed in Australia hearing it from someone in Australia about how it happened and in that whole situation you know we can we can read about it here and read a fair amount about it but it doesn't replace actually hearing like what you know some some of the more finding details about society and what went through the legal process for how it actually happened and by having those discussions we can ideally help prevent somewhere laws from happening within the United States because again at some point we all get there's going to be another you know encryption battle going on and there's some side ones already but there'll be another FBI versus Apple or something along those lines that are prepared to help ensure that encryption stays strengthened so they're just there's just I think so many different areas where collaboration can help us both on the security and privacy side but this is another area as far as just innovation at large from doing joint research and all those kind of areas that we really have just gone so far back to just walking down all the data which is understandable and it's human nature we don't want to have all of our data thrown out there for all its legal repercussions for reputational for financial risk all those reasons and we need to get back to a bit more of a balance through some of the various forms of collaboration securely though I think that that's actually getting back to the this tension and all of this being a balancing act do we emphasize collaboration do we emphasize security are they mutually exclusive I hope not but I do think that kind of how do we create information sharing norms that are appropriate because when a lot of people say information sharing they they mean tell law enforcement what they want to know but that's not actually going to and this is the moderators injecting yourself you know that doesn't necessarily fix anything so I notice that and that's why we are seeing it within industries themselves much more activity within the information sharing groups and so the organizations are starting to take it on themselves again just for within their own industry to help figure out how they can collaborate and share more because they're getting attacked by the same kind of groups of people the various actors and so there are a variety of solutions the tensions are there but people are starting to push back now organizations are and I think governments are against what has been the status quo because that status quo is not sustainable Gotcha so in that realm you talked a little bit about interoperability and of Facebook's various messaging platforms do you see kind of opening that up as a place where you're going to be able to kind of share lessons across your various platforms at Facebook because I know that they have not always been interoperable working at a tech company I understand why it's hard to make these things work together sometimes do you think that opens new avenues for hackers law enforcement or actually makes it easier to defend so I mean I think whether or not it's easy to defend is probably more of a question for a technologist I think that the thing that we think about when we are making these decisions and really what animated our decision in order to move toward interoperability and tend to encryption across our messaging services was the fact that we do see cybersecurity threats increasing across the board we see threats to users data from authoritarian regimes increasing and we see just general threats to privacy increasing and a desire amongst internet users for more private means of communication and so we are trying to respond to that and make sure that we can provide users with those secure means of communication but also make sure that they're not sort of in a closed garden where you can only use one service and not another and if your friends or your family are using a different service then you're not able to communicate with them securely and so that is why we are creating interoperability between our different messaging platforms and I think you know one of the things that we find challenging is we do see increasing demands from foreign governments for technical assistance and so this is in different forms but the most concerning form is where governments are asking for ways around encryption or encryption back doors as they're colloquially referred to and this is something that is just deeply concerning because it really it's it goes against the principle of providing secure communications but in addition to that we as a company just don't feel that governments should be able to tell us we should be building our systems where we should be storing our data and we think that it's important to provide excellent security to our users' communications but that's also one of the reasons why you know we are really looking for ways to change how we respond to cross-border data requests because I think as encryption use increases as data is sort of like de-territorialized in terms of where it's stored questions about jurisdiction become really pressing and what we find is we have a mutual legal assistance process in the U.S. that for Facebook applies to the whole world except for the United States which works through its own judicial process which is very similar to the MLAP process or the EU which works pursuant Irish jurisdiction but the mutual legal assistance process requires that foreign governments that are seeking users' contents get warrants based on probable cause which is a pretty tricky thing if you're a foreign government and you don't have concepts of probable cause your legal regimes are different your criminal laws are different and so as a result of the complexity and the labor intensiveness of this process it takes varying amounts of time in order to respond to MLAP requests but on average when it all shakes out it can take around a year to respond to these requests so this means that a foreign government that is responding or investigating a murderer or another serious crime that was committed in their country by one of its own citizens could have to wait around a year in order to see information that could be relevant to that investigation we think that the amount of time that it takes and the difficulty in responding to these requests that the government has to be clear that requests as soon as we get the warrant pursuant to the MLAP request we respond immediately but the process, the intergovernmental process takes a very long time and that length of time winds up undermining public safety because it can impede legitimate investigations into serious crimes so why we want to address this one by supporting increased resources and expertise at DOJ and throughout governments around the world to help speed up this process but the reality is like all of the resources in the world and expertise in the world probably aren't going to address the problem wholesale and so we also support alternative mechanisms for accessing data such as the U.S. Cloud Act which passed last last spring and that would basically enable foreign governments that meet human rights standards to enter into executive agreements with the U.S. government and upon entering into that agreement they could directly ask providers for users' contents and providers would be able to voluntarily respond so we would still be able to apply our test to make sure that we consider the request to be lawful to be within the jurisdiction of the requesting government pursuant to their laws appropriate appropriately scoped and purposed and then also in compliance with international human rights norms if those elements weren't met we would be able to push back and basically redirect the request back through the MLOT process but under the Cloud Act this at least creates a supplemental process for human rights respecting governments to be able to get information more quickly so we'd like to see some you know more investments along those lines we haven't actually seen a government enter into an executive agreement yet so the negotiations for that are still ongoing and lots are slow and so are governments so the panel description kind of promises that we'll talk about proactive strategies to move us in the right direction specifically to protect user rights and to promote safe and smart use of these tools whether you're talking about the internet your kid monitoring software your you know any of these tools that have the ability to be abused so this is for each of you what strategies would you like to see focused on that would really kind of impact people rather than necessarily governments I will start I'll do a sort of multi-pronged one focusing on the tech just to look at what some of the basic steps are while there's no you I think best practice is the term that people in the InfoSec community hate but at the end of the day try and encryption as best you can and do be wary of any of those kind of track whomever tools for the potential for misuse those are some of the big steps and obviously phishing is still a big one but those are some of the steps as far as just securing yourself and just taking back greater control and we're seeing public opinion pull after another in the United States for people wanting to take back better control of their data and so get better informed in those areas and try to do that at the same time get involved with the discussions that are going on at the federal and in your state level as far as data privacy laws that are coming up every state more or less is now debating some of them and so I tend to focus on the tech community and try and get more tech people involved in shaping it but obviously the legal field absolutely especially people have legal and tech background are super important helping shape these discussions so they don't turn the wrong way and so they don't infringe on human rights and privacy or they don't go the other way and basically do nothing by doing having a really low bar for that so I'd get involved in those and there's a lot of different efforts that are out there that you can get involved and to help get more informed on what's going on within the United States at that level Yeah so at least from my perspective the way to what you want to do to help people who might be potentially victims of the abuse of these tools is to reduce the probability that these tools will be abused so you in some sense I gave an overview of what's been tried so far but I think where we've had less success is in holding the companies directly to account so let's say Jamal Khashoggi's friend Omar Abdulaziz in Canada was spied on by apparently the Saudi Government using NSO Group spyware wouldn't it be great if there were a way to hold the company to account such that this could dissuade the company from making those sorts of sales in the future or from continuing to support or maintain Saudi Arabia as a client because you know this sort of relationship between the companies and the governments is not simply transactional it's not like oh you give me money I give you spyware there's an ongoing relationship where there's support, updates, training those sorts of things so the tricky part though about holding these companies to account is that it's really really hard to gather evidence in this space like in order for you to get conclusive evidence that someone was targeted by some spyware well okay maybe you have to inspect their phone you've got to get in touch with the right person and then the question is okay well maybe they were targeted you can show that they received the dodgy email or the dodgy text containing the link that you can show is to the spyware but was their device infected was anything successfully privacy breached in some sense so that's sort of like a higher level to show and it's much harder because like well okay these devices we're carrying around nowadays are pretty much black boxes to investigators so Apple, Google they really really lock down the device you can't even get like if you want to do a forensic analysis on it it's pretty much not possible unless you yourself hack the device in order to get enhanced access do the analysis so you know thinking about ways to make it easier for people to access their own data and you know help with these investigations you know in that same vein there's also well okay you know if someone was infected with spyware right then their computer their phone whatever is infected is sending information back to some website some server and we know in a lot of countries there are data retention requirements where the ISP has to give your web history for like a year or something so that law enforcement if there is a criminal investigation can then go and get that web history and investigate but it would be great to have if we're going to have those data retention laws it would be great to have you know more control so that potential targets or victims of these sorts of attacks could say oh well I want to find out if I was targeted hey ISP can you send me my history my web history then I'll go to you know some research and take it to them and see what they can tell me or you know hey Facebook, hey Google, hey Twitter you know you sent me this really interesting state sponsored alert can you tell me perhaps what what triggered it can you point me in the right direction so that we can perhaps document evidence of this so I think there's two parts there's getting better evidence which involves putting people more in control of their own data and there's also the you know somehow perhaps creating some new liability for these companies that are found to be assisting in these human rights violations so yeah I think when we're thinking about how to best protect users and what are some of the new things that we can do from Facebook's perspective you know we are really looking at it in terms of what are the laws that are governing user privacy because when we're thinking about how to hand over user data we're thinking about it in terms of what are lawful requests when are we required to hand over data or when is there an imminent threat to life or serious bodily injury that would necessitate us handing over information and so what we're really thinking about is how can we make sure that the laws are more protective so that whatever government it is that's asking for user data it's doing so right's protecting way and so as a result of that we've joined the reform government surveillance coalition to work to amend US surveillance laws we're a part of a Digi coalition which is a coalition in Australia that works to address the assistance and access act which raises some real concerns around compelled decryption demands through technical capability notices which we were talking about a little bit earlier and so we really try to think about like who can we collaborate with in industry and civil society to make sure that laws are protective of people's rights so that governments can't overreach and so that there are appropriate oversight mechanisms in place so that if they do there's a means of redress through an independent oversight or judicial body great so my plan is to let you guys go a little bit early so I think we'll get started on Q&A I know that we're the furthest room from your closing keynote and I want you all to get the best seats so we would like to take a few questions I want to be clear they have to be questions I will cut you off and we will we will kind of if you can keep them short that's perfect like tweet length and because we want to get as many as we can in so I Bill I you talked about the spyware and I know that there are rules and so on but can this is a spyware good enough to be able to get into Facebook and I mean Apple I remember all this talk about Apple not giving out anything can some of that spyware get into my iPhone so the question just for the stream is about whether some of this spyware is good enough to get into an iPhone even though it's relatively hard to get into an iPhone if you're law enforcement can they get into Facebook which pieces of our digital life can this spyware get into well there are companies out there that are selling capabilities to get into pretty much the latest versions of iPhones of androids of any device that is you know commonly owned by consumers you know and once in the device they can access you know your Facebook data that way so I think that companies like Facebook like Google have a very sharp security teams that are focused on not only protecting employees of the company against these sorts of attacks but also users of the company's products so I think that the vector for getting into the phone would probably be a vulnerability that exists on the phone so like in WhatsApp WhatsApp was shipping code they unknowingly had a vulnerability in their code which you know when you got a call it could exploit some bug in the call signalling processing and take over the phone so pretty much they'll be going after the devices themselves and vulnerabilities that exist in the devices and it's a really hard problem because the devices we're carrying around in our pockets are running tens of millions of lines of code and how do you ensure that there are no bugs in it it's hard you know we've built up this multiple layers of software really really quickly without you know in many ways without serious thought about security of each layer so there's lots of bugs in our phones and companies are looking for ways to get in and selling that for millions of dollars this conference is trying to develop approaches rule of law approaches to these questions and going back to your suggestion about norms what would each of you identify as the top one or two or three sources of developing norms that those of us interested in these issues or building legal responses should look to I'd start with reviewing what the UNGG has already gone through and find out where some of the hurdles were because that will end up make basically there was some collective action problems so if you want to help avoid some of those I'd review some of those but I also would then I'd make sure to look at what some of the corporations are doing as well and even the Paris call oh yeah sure yeah absolutely sorry yeah so the UNGG basically did about I think about five different iterations trying to establish just common basically the low hanging fruit for what cyber norms could be and so it could be anything from not interfering in a cyber emergency incident response team so allowing governments to respond to those kind of incidents to not attacking critical infrastructure in peacetime some various areas along those the U.S. Sino agreement from 2015 was ostensibly to not allow commercial corporate theft by a government to help their own commercial enterprises that has since there's been numerous reports on that the trade rep U.S. trade rep did a really good report a little while ago outlining how that norm was broken but interestingly we can show how some of these things you spread when the U.S. China did that then the U.S. and Canada did something similar and China and Australia did something similar so while it was broken we started establishing some baselines for those kind of areas and that was all governments doing that the Paris call is an interesting one because that is an integration of both tech companies generally tech companies, corporations and governments and the numbers I actually don't remember it was over over 100 I think countries and then some like over 60 major corporations all signed on to that trying to rejuvenate some of these efforts towards norms again with some of those low hanging fruits and Macron introduced it in November the U.S. did not sign on to that for me I wish we would I wish we'd actually help shape it as well and that that's one of those things but by not being part of some of these discussions we can't shape them then we can't help to shape a lot of those discussions there's the Charter of Trust from Siemens is another one for corporate level that has gone out and then there's the Tech Accord that Microsoft put forth and Brad Smith from Microsoft I think it was RSA Talk like three years ago that kind of outlined some of those and if you hear the notion of a cyber Geneva convention it's linked back to that so those might be some good places to start but do you either Robin or Bill have thoughts on that one or should we move on to the next question okay we're good yeah so one of the questions I had was sort of you guys have talked about how collaboration between governments and between the actual companies themselves are creating the software is so important and then you also mentioned activists so like so there's lateral and then there's horizontal right so how can activists who maybe they don't know the citizen lab and they might be watching this streaming service right now how can they sort of get involved and let people know like hey like I think something might be up I need help I don't know who can help me because you know we deal with a lot of activists on the ground who really are dealing with serious threats and for that reason we oftentimes are like the connecting point to say okay well here's your problem here's your solution but what would you want sort of activists to know about like who can they go to who can they report this to who can they work within collaboration because a lot of the information you get is coming from these people and you but for those people you wouldn't even get the information so what would you want them to know maybe right yeah that's a good question I mean in terms of citizen lab ourselves we're a research organization so we don't really have a whole lot of capacity to be providing support to to activists around the world you know there are really great efforts like the creation of digital help lines like the access now digital help line which is sort of a first point where activists or civil society who are experiencing you know something suspicious going on can contact so they assist in things like oh my my account was hacked by somebody can you help me get access back to that you know or you know oh my my phone's acting strange you know what do I do because you know it's it's it's tricky because you know there are specific signs that that people can look for that should be red flags but but often it's not the ones that are sort of intuitive like we'll often get a lot of people saying oh well you know my phone's running really slow or my battery's draining really fast and you know it's sort of hard to know well okay maybe the phone's old so the battery doesn't last as long or maybe you just installed some app which is very very power hungry or something like this and slowing down your phone but you know being aware of the sort of vectors that you can detect like for instance suspicious text messages it is you know almost assuredly some sort of targeted attack if a political activist gets a message from an unknown number promising you know juicy activist details with a link or attachment in there right you know without any sort of context about who the person is from a legitimate institution or anything like that so so that is you know what ideally people should be looking out for but you know for as a first point of contact I think digital help lines like the access now digital help line are good for that sort of stuff yeah all the way in the back is great I haven't heard much about repression of First Amendment rights where speech rights outside of the United States and I know that Facebook has recently decided to cut the access of certain people most of whom the people in this room would find to be objectionable speech to begin with but some of us at least have some objection to repression of any speech at all and can we get some indication of how determinations are made as to who's cut out and who is not thanks so generally we will well not generally we will always apply our terms of service so when making determinations about whether someone is going to be suspended from the service it's a question of whether our terms of service have been violated and we have teams that work on issues around content moderation and the development of the terms of service who work to make sure that those terms of service and the standards that they set forth are applied to everybody and so that's that's it okay cool and I saw I mean I think that all those terms of service are on the website that's right and I think that's just common how websites and services try to communicate that so I know that our add-on store has a similar set of rules that we recently enforced which was great fun yes thank you sometimes when I send an email I proof it and then after I send it it's been altered where it's been changed to make me look stupid like I don't know how to spell and I have malware bytes and I have a look out on the android 3c well somebody's it's probably NSA I was in the piece born in Lawn do you have a question? what can I do to stop this? well so that's a great question I mean I think there's definitely resources you can take advantage of now you know digital help line and mention sort of what's going on yeah I see two more and then we're going to cut off questions I think so thanks really appreciate the panel and I really appreciate that you guys had the conversation about Spouseware and Stockerware I work for one of the AV companies is trying to figure out how to be more helpful in this space so I'd be very curious I guess maybe this is probably a question to Bill what else should companies be doing to kind of help users better empower themselves to protect themselves we just recently announced a privacy alert largely due to the leadership of Ava and others but trying to figure out what else should the industry be doing at large to help in this space that's a great question well I mean you know so the AV industry is sort of one part of it I think there's also like Google sort of adopted their policy about you know not running ads for you know how to spy on your spouse and things like that so I think you know that I don't want to sort of step out of my bounds here I think you should talk to Ava a bit more for details on that but I'd say that you know in terms of AV companies I think it's great that you guys are are sort of taking the position about you know notifying people if there might be something operating without their consent like that that's a great question all the way in the back thank you actually just to follow on that I've noticed when you mentioned the other countries using surveillance technology is this an ITAR issue that you feel software developers in the United States needed to recognize I apologize if this was covered before but need to recognize early on and then also data destruction at the border oftentimes when activists come to America countries have their phone sees or information sees Facebook pages looked at what types of software developments do you develop to prevent that and also communicate that to other countries that might be using them yes so I'll answer your second question first about about border crossings this is sort of a very tricky thing to navigate because you know in many cases if you refuse to submit an inspection you can't cross the border right I mean the exception is of course if you're like a US citizen coming back to the US they have to let you in but if you're someone who is I don't know like a refugee or applying for asylum or a political dissident who needs to travel for whatever reason you're pretty much at the mercy of you know letting them inspect your devices the best defense against this that I can think of is to not have if you're worried about data being being inspected to not have it on the device not have it accessible from the device so you know there's things you can think about which again are sort of just things you can do maybe tech geniuses can do but need to be you know incorporated into perhaps user friendly apps and things like this like you know one thing you could do is say okay well I'm going to you know maybe I'll think about my computer I've got some sensitive data I want to encrypt this data in a way that I can't decrypt it but when