Our speaker will tell you about the status of coordinated vulnerability disclosure worldwide with a special focus on the Netherlands. And he has several years of experience working in this process, which we will talk to you, tell you all about it. Jeroen van der Ham is a researcher at the National Cybersecurity Centre in the Netherlands. He is organizing Science4SHA for this festival. So maybe we can give him a short round of applause just for organizing Science4SHA. His interests include ethics and coordinated vulnerability disclosure. And if you meet him for a beer, please ask him about the performance of the Dutch postal services leading up to this event. He has some great stories about that. But for now, please give it up for Jeroen, who is going to talk about Coordinated Vulnerability Disclosure. Thank you, thank you. So yes, my name is Jeroen van der Ham. I work for the government and I'm actually here to help to tell you how the government can help you. Yes, I'm really serious. So first off, who here has done disclosures of security vulnerabilities? There's a few hands there. And who's from the Netherlands? There's a difference in the set and that affects you. And I'll be talking about how we've been doing vulnerability disclosure policies in the Netherlands. And also, I'll have some good news about international developments and can tell you more about that. So first off, we all know that everybody has vulnerabilities. And the important thing, the important message that I'm trying to convey is that what's most important is how you're able to receive them and act on them. Because that is what affects most of the security in the process. So we have vulnerability disclosure. The most common term for the process these days is Coordinated Vulnerability Disclosure. The idea is that you work with vendors to fix vulnerabilities before they are disclosed. There are a couple of phases. You report, there's triage, there's development of the fix, and then there's the deployment of the fix. And it's important for organizations to realize that there are these four phases, but it's also important for the disclosers to realize that there's these phases. And the Coordinated Vulnerability Disclosure happens before this whole process. There has been a disagreement, there has been debate, there has been some agreement on disclosure of bugs in the last 30 years. We've been doing this for this long. First there was full disclosure, there was discussion, and now we're at responsible or coordinated. And we're seeing some more trends towards Coordinated because it puts less of a value on the behavior of the disclosure. And of course there's a lot of discussion, what kind of information to whom and when. There's also a lot of discussion about how long to wait for the vendor to actually fix stuff. These are all still open questions, unfortunately, and I think the debate just has to evolve so that we can actually find answers to these questions. The governments didn't pay that much attention to Vulnerability Disclosure in the last 30 years until everything became a connected computer. And now in the last few years we're seeing even more developments, now that transportation, healthcare, manufacturing. We're seeing self-driving cars, we're seeing pacemakers that are vulnerable, we're seeing lots of ICS stuff that is completely vulnerable. And you're also seeing some kind of liability for software systems, or at least there is some discussion on it. And well, yeah, the examples, there's even toilets that have been disclosures on toilets, I'm serious. And there have been national and international efforts to actually discuss Vulnerability Disclosure and how this affects the disclosures. I'm happy to report that there have been many, many international involvement initiatives and guidelines. The first one on the left is actually in 2013, the NCC in the Netherlands, they published a guideline for how to create and publish a Vulnerability Disclosure program, still called the Responsible Disclosure Program, and it included a way for hackers to act, to do their research, to figure out what the vulnerability was, how to report it to the vendor or the affected organization in such a way that they would not be prosecuted, either by the organization or by the government, even. There's also been a standard, it's published by the International Standards Organization, ISO 29147, and it's actually available for free. So just at the beginning of last year, it was made available for free. It describes the process of reporting a vulnerability without putting any, it doesn't say much about a prosecution, but it at least describes the process of what you should do when you receive a vulnerability disclosure. The other standard 30111 is a description of what the internal process should look like when you're fixing a vulnerability disclosure, or when you're fixing a security vulnerability. That one is unfortunately not free, but it's mostly aimed at organizations and not at disclosures. There has been initiative called at the Global Conference on Cyberspace, the GCCS, that was in 2015. We helped prepare a document there describing all the issues around vulnerability disclosure. The European Network and Information Security Agency, the ANISA, also looked into all of these issues. They identified this as being a very important issue, and they also published a statement describing all of the efforts going on and saying that this is actually a good thing. So that's mostly European. Over on the other side of the pond, the NTIA is a department that's also the department that was involved in regulating or governing the Internet, the ICANN function. But another part of the NTIA was actually talking to people. They did a multi-stakeholder process. They invited security researchers, they invited organizations, bug bounty companies, government, everybody, and they talked about how to do vulnerability disclosure and what the challenges were in vulnerability disclosure. And they published actually four different documents saying how to do vulnerability disclosure. They did a research on what kind of people and what kind of attitudes the security researchers have. This was actually a very awesome study that I have a graph at the end of the presentation to show you what the results are. They looked at companies that did vulnerability disclosure and there was a fourth one. Oh yes, they also described a more complex process in what happens when you have to do a coordination between several different organizations when they are affected by the same security vulnerability and they have to coordinate to all fix at the same time. And just in time for my presentation here, you wouldn't expect it, but the US Department of Justice has just published a framework document last week on how to do vulnerability disclosure and they even included some statements saying how you can frame this in the same way that the Netherlands did is they created some sort of legal loophole that companies can actually make sure that the disclosures are not prosecuted. The Department of Justice did the same thing saying this is the stuff that you can put in your policy to make sure that the security researchers are safe from prosecution. So I think that that is really an excellent development. Since the introduction in 2013 in the Netherlands, it has been a huge success. This is just a small selection of companies in the Netherlands that have a responsible disclosure guideline published and it ranges from security companies like Fox IT, ISPs like Access Roll, but even DIY stores, banks, municipalities, water companies, news blogs, any kind of organization is represented and actually has a responsible disclosure program. And these companies, they actually are very positive about this development and they even went so far as to come together themselves and they wanted to express their enthusiasm and they wanted to publish their enthusiasm about the coordinated vulnerability disclosure and they wrote a manifesto saying these companies, we support the idea of vulnerability disclosure. We want to promote this idea and they got on stage during the EU presidency of the Netherlands last year and put out to the world that they were supporting this idea. Meanwhile, we also have a big development. The vulnerability disclosure programs have become so successful and so popular that there are actually companies springing up right now and filling the void in helping other organizations do vulnerability disclosure. There's Hacker One, there's Bug Crowd, there's Synac and there's Zerocopter who are represented here today. And these companies help other organizations set up bug bounty programs or just simple vulnerability disclosure programs. And I think that that is a very important development. Let's see. Oh, I wanted to point out, if you have questions at any point, I can take questions during the talk or if you have questions about vulnerability disclosure policies, I'm happy to take them. So I've been talking about a lot of positive developments. There have been unfortunately also some negative developments and they also came from the Netherlands. In the town called, they started actually in the town called Wassenaar. They don't actually reside in the town of Wassenaar anymore, but they're in Vienna these days. And this is actually the Wassenaar arrangement for export controls of conventional arms and dual use goods and technology. So this is a whole mouthful. What does that mean? This is the way that 41 governments around the world, they talk to each other and they agreed on a list of goods that should be controlled because these goods can be used for both good and for military purposes. And you can think about stuff that makes very nano-chips or even aluminium tubes that could be used both for construction but also for making rockets. Chemical goods that can be used to make explosives or any kind of stuff. And we should actually know about this because this document is what started the crypto wars in the 90s. And actually it's still there. What they introduced back then is they, so the process is they agreed we have to sync on goods that are dangerous or for dual use goods. And the idea is that if you make this kind of stuff and you want to export it to a different country, even countries that are in the Wassenaar arrangement and talking to each other, then you have to apply for a license.  So for a certain client, you get one license. Or if you do your paperwork right, then you can get a big license saying you can export to these kinds of organizations in this list of countries. So you get more of like a blanket agreement. But in the 90s they introduced this. And this says that any symmetric algorithm with a key length in excess of 56 bits. Or an asymmetric algorithm where the security of the algorithm is based on one of the following factorization of integers in excess of 512 bits. 512 bits. If you do more than 512 bits, you have to apply for an export license and the government has to judge whether they want to allow you to export this or not. What happened in the crypto wars is that this is actually still there. This is still current law. But they put in an exception saying that if this is open source, if this is freely available, or this is an academic research and published, then you can actually export it because anybody can get it and it doesn't make any sense to export it. So we have crypto because everything is open source. If you have closed source crypto, you have to apply for an export license. This is finally being reconsidered. Yes. Okay, I'll repeat the question. The question is, in which countries does this apply? You can look at the website wasanar.org because it's 41 countries and I don't know all of them. It's most western countries. Actually, Russia, Ukraine are also in there. I don't think China is in there. Israel has a special status. They're part of the agreement, but they do follow it. They do implement it. So it's most countries that you normally deal with. So why is this relevant? Well, this is relevant for vulnerability disclosure because some time ago they introduced this clause that says systems, equipment, components, especially designed or modified for the generation, operation or delivery of communication with intrusion software. So anything that has to do with intrusion software and then intrusion software and also technology for the development of intrusion software. Then what is intrusion software? That is software specifically designed to avoid detection or to defeat protective countermeasures. So if you break security, then it's probably intrusion software of a computer or a network capable device. And to do extraction of data or to modify the standard execution part of a program. I have no idea what the standard execution part of a program is. And if we wouldn't be able to establish what the standard execution part is of a regular program, then we'd be out of a job because then the security problem is solved. So in essence, this was put in by people that made fancy statements that sounded good and seemed like they were talking about security when actually they were not that well informed. And they said hypervisors, debuggers, etc. Yeah, they might follow this kind of definition. So they're excluded and asset tracking and for some reason smart meters. They're excluded as well. I have no idea why. How does this work? Oh, let's see, how does this work? This was actually introduced in 2013 and the reason leading up to it was the whole situation around the Arab Spring where all of the governments were using spyware to monitor on the dissidents that they were causing the Arab Spring. So all of the countries said they're using all kinds of software that we don't want them to use and the only way that we can actually stop them from using that is by using export control. So they went to the drawing table, they didn't bother to consult with all of the security people and they just made legislation and said, yeah, this sounds good, so we'll do this. So in 2013 they introduced the definition of intrusion software. The deal is that the EU implements this as regulation, one year after an agreement in the Wassenau arrangement because all of the European countries are a part of the Wassenau arrangement. They don't want to do another discussion, so usually this is just hammered into law. So on the 31st of December in 2014 there was an update of the EU regulation and it's still there. So this means that it may actually be that vulnerability disclosure is covered under this list of intrusion software because the whole idea of technology, I didn't explain that, but technology for the development of intrusion software, that means anything that you need for the development of intrusion software including knowledge. So the transfer of knowledge, like I'm talking here to you, because you are of different nationalities, this is export actually. And if I were to describe a security vulnerability, and so the previous talk was a description of how to own CPEs, so modems, and he included a POC. And if you read the definition in a certain way, if you could interpret it in a certain way, he would have actually needed an export license in order to present it here. So that is something that wasn't well received in the US actually because the US was a little bit late in implementing this. And in the summer of 2015 for reasons that there was some disagreements within the government of the US and the Chamber of Commerce, the Department of Commerce actually made the most wildly strange interpretation of the Wasunar rules. So they included the terms zero-day in actual legislation, hacking, all kinds of different terms. So they included the whole worst case interpretation of the Wasunar arrangement. And then all of the technology companies that did vulnerability disclosure like Facebook, Google, all of these companies in the US, they went apeshit. So there were thousands of responses on the consultation of the implementation of Wasunar. And what resulted was actually that the Congress in the US told the government to go back to Wasunar and say, you have to re-discuss this. We're not going to do this. It took a while. Unfortunately the whole process in Wasunar is very, very, very long. It took a while and it looks like this may be resolved in Wasunar at the end of this year and then it would be solved in the EU at the end of next year. Meanwhile, the Dutch government holds the interpretation that vulnerability disclosure should not be covered by the idea of intrusion software. So in the Netherlands you're safe. I think in most European countries you're probably safe, but still they could go back and bite you with it. I've been talking about vulnerability disclosure a lot. The whole idea of the vulnerability disclosure in the Netherlands is that there's a couple of building blocks and how you do this. And the idea is on one hand you have the promises of the organization, the CVD policy. And on the other hand you have some responsibilities or guidelines for the behavior of the researcher and hacker. And they have to work together and if they do then you come up with an optimal process that the hacker can securely notify an organization of a security vulnerability. The organization can fix it and then in the end the researcher, once it's fixed, he can publish and he can go out the world or presenting at conferences, etc. The Dutch Public Prosecution Service actually thought that this was a very good idea. And since the implementation in 2013 there have been two cases, two court cases, where the idea of responsible disclosure or coordinated vulnerability disclosure has been named. In both cases the organization affected did not have a vulnerability disclosure program in place, but the judge still used the guideline as a measurement stick for the behavior of the hacker of the disclosure. And I'll go through the cases, so before I go through the cases I have to stress that there has been no case whatsoever of a company that had a vulnerability disclosure program in place against a security researcher. We don't know of any. And the Public Prosecution Service is actually very happy with this idea because it saves them a lot of work. So the cases. There are two. The first one is the Groene Hart Hospital in the Netherlands. They had a vulnerable FTP server with an easily brute forceable administrator password, in green 2000. The hacker found the vulnerability, the vulnerable system, and he informed the journalist because he was afraid of reporting it directly to the organization. But we have some protection of the press, so a journalist is considered a reasonable source as a way of disclosing this to a company. The journalist informs the hospital at 10 a.m. on, I believe, a Saturday morning. He actually published the story at 3 p.m. in the afternoon. And this was because the company in question just had a crisis management training just a week before. And so they got this telephone call from the journalist. They went into panic. They started their crisis management stuff because they had just had the training and they knew what to do. And one of the first things that they wanted to do is put out the press statement. And of course the journalist wasn't happy because that would ruin his scoop. So when he heard that the company was going to issue a press statement, he just published his story at 3 p.m. So that's why he published, that's why there was a very short disclosure timeline. After this happened, the hospital reports the case of the police. The hacker in this case, he used the port scan tool and he had access for at least two weeks. He was obfuscating that access by using a VPN. He probably retrieved the password hashes and brute-forced all of the other passwords or at least tried to do that. He shared some of the credentials that he found with other people. So he bragged to them on chat and IM and et cetera and he shared the credentials. To go even further, he installed malware on one of the servers to get persistent access, but he was using his own IP for that, not a VPN. So he made a mistake. He downloaded multiple medical files from the organization, the hospital, including those he was looking directly for some of the famous Dutch persons and downloading those files and looking at them and maybe even sharing them. And he sent screenshots of those files to the journalist. Finally, in his defense, he states that his actions were done in the public interest. When the judge looked at this case and said yes, the public prosecution service has a discretion to prosecute this because the way this works is the public prosecution's office has issued a statement saying we won't prosecute in this kind of cases when people follow the responsible disclosure guideline. So one of the first statements that the judge made is yes, these actions are so... are blurring the line of what is acceptable that you can actually prosecute and try to figure out whether he crossed the line or not. The judge will have to judge whether the actions of the reporter, the ethical hacker, were proportionate and subsidiary. He emphasized in his verdict that revealing security vulnerabilities can be in the public interest, especially when we're talking about medical files. He also found that there was no other way for the hacker to discover their vulnerability, which also meant installing the malware was actually acceptable to show that the hospital had very weak security. The hacker accessed the data and downloaded the data multiple times and that is unacceptable and not necessary to show that you actually had access. You don't have to do it multiple times and you don't have to download data of famous people and you shouldn't share that with others. That's the whole idea of coordinated vulnerability disclosure. In the end, he was sentenced to 120 hours of community service. So this was a reasonably light statement for what he actually did. The judge said this was not following the ideas of the responsible disclosure policy. The other case involved this famous hacker person, Henk Krol. He's not actually a hacker, although he has talked at hacker conferences at some point.  But in this case it was also about a medical organization, Diagnostik for you, which did dealt with psychological research about people. And in this case, there's a reference there also if you want to look that up on the internet. The disclosure is a politician and a journalist. Like I mentioned, these are Henk Krol. An acquaintance of him discovered a vulnerability. No, he discovered the login and password of a medical system. He just shoulder-served it and logged in and looked at records and all kinds of other things. So he did some reconnaissance. He prints a few files. In the first instance, when he printed those files, he anonymized them. And he calls the help desk of the company saying, yes, you have very poor security and you should do something about it. The help desk has no idea what's going on. They are completely caught by surprise. They have no idea what responsible disclosure is. They have no idea what security stuff should be. They have no direct line to the security team. So they actually did something sensible and they said, please write a report to us and send it to us and then we can figure out what's going on and what we can do. The disclosure is actually not happy with this. And he thought that they should have acted more and more decisively. So he calls a local TV station and tells them about his findings. He actually goes on camera with a few journalists demonstrating that he can log in and on camera showing medical files that he has access to. So the verdict in this case, the judge stressed again that the disclosing vulnerabilities of sensitive medical system can be in the public interest, in the general interest. It was reasonable to test these initial findings in practice. So you are allowed to log in and do some reconnaissance.  And the judge used a three-part test in light of the Article 10 of the European Convention on Human Rights. The freedom of expression and information. The disclosure should act in the general interest. The disclosure should act proportionately and he should act in the least invasive way. And portionality, it wasn't necessary to review the files with a journalist on camera. He just crossed the line there. Invasiveness, there was also no reason to go to the media directly. There was no big vulnerability that everybody could exploit because their shoulders served the credentials. And there was no indication that other people had access to those credentials even though they were not that complex but still. And the judge also thought that the suspect should have taken more effort to contact the responsible partner. Because he's a member of, he was, is again a member of parliament. And he's a journalist. It's reasonable to expect him to do, to take more effort into actually contacting the organization and getting the right people and to actually talk to them and get this fixed. The result was just a $1,500 fine. But it still was a statement saying you crossed the line and this is not acceptable. And it was very helpful to us to say that the judge actually very clearly identified where he crossed the line. There are multiple, multiple cases where companies are actually doing this in a positive way. And I've highlighted one because it shows that the company in question did a complete turnaround. It's the Dutch telco, the KPN. They've actually been owned very badly at some point and that caused them to completely think about their security, rethink their security and they actually published a vulnerability disclosure program as part of their complete rehaul of the security. And one of the things that happened is that they actually got a report that was very successful because two hackers found vulnerabilities in CPEs, the modems. These were widely used. It may have been the vulnerability that the previous speaker was talking about, I don't know. But the vulnerability could be used for DDoS attacks and to gain access to the CPEs and to actually intercept data of all of their customers. They give remote access. The hackers in this case acted very cautiously and they only tested this against their own equipment. So they didn't test it in any of the third-party equipment. And they reported the vulnerability to KPN. KPN in this case takes up the report. After doing some initial research, after acknowledging the initial report, they actually invited the hackers to the office because the actual vulnerability seemed to be very complex and they couldn't really understand from the report what was actually going on. So they invited the hackers to actually give a presentation to the security team on how they did their research and what they actually found so that they could actually start fixing it. They managed to fix this within a very short amount of time. The hackers were awarded with KPN goodies and they were also allowed and even sponsored, I think, to go to a conference. And I think this was hack-lew that they actually presented this. And KPN, furthermore, used this as an outward statement highlighting that they took security seriously and they invited others to do the same. They released a press statement and made a video with all of the hackers to actually promote this idea. And KPN is also one of the partners that was in the manifesto that I mentioned earlier. So that is basically what I wanted to talk to you about. One of the last things that I want to highlight is I mentioned earlier that there was this research by the NTIA about research and motivation. They had about 400 people who actually responded to the survey. And one of the questions was what motivates you to actually do the research? This was a multiple-option answer, so the answers add up to way more than 100%. 14% of the researchers wished to remain anonymous. Only 15% expected compensation for their work, even if the company doesn't have a statement saying that they actually pay for this. 53% expected least an acknowledgement and 57% they just want to help. They want to help fix stuff, even if they're not paid. And the most important one is 70% of disclosers they expect to be regularly communicated because they can't see what is going on inside of an organization when they're fixing stuff. So the organization should really reach out to the disclosers to keep them updated about the process and stuff that they should do. The key takeaways that I hope you got from this is there are governments that are taking vulnerable disclosers seriously. There are more and more governments in Europe that are actually doing this. Belgium has issued a statement that they want to do this. France has some kind of legislation. The Italians are actually thinking about implementing this. There has been talk about this in Hungary. And like I mentioned, the US also has now an official statement on how to actually do this. The court cases in the Netherlands have shown that if a company publishes a vulnerability disclosure program, or even if a government states that they support the idea of vulnerability disclosure, it doesn't give you security researchers a carte blanche in doing anything and completely owning the organization. There are limits to what you can do in coordinated vulnerability disclosure. And in the Netherlands we see that more and more organizations actually really appreciate this idea and they cherish the involvement of you, the security researcher community. So with that, I thank you very much for your attention and I'm happy to take your questions. We have about 20 minutes for Q&A, so please line up at the mic and take it away. One thing that I want to add, sorry, so the guideline that we made in the Netherlands has been published in 2013. We're actually reviewing that document because it's a little bit dated and we're reviewing it, rewriting it, and we hope to publish that somewhere at the end of this year. So if you have any comments on how to improve this, I'd be happy to hear them just after this talk. My credentials are, the ways to contact me are down there. Thanks for your talk. I mainly want to go into the Dutch issue. Actually, if a company doesn't have a responsible disclosure policy published, I as a security researcher am not covered by the law, I can still be sued. Shouldn't we fix that? I mean, it didn't happen yet, but it could happen, right? Well, when I wanted to show with the two court cases that I mentioned, the two organizations didn't have a vulnerability disclosure program in place and the judge still used the ideas of the vulnerability disclosure program as a measuring stick for the behavior of the security researcher. So if you act within the guidelines of the vulnerability disclosure program, then the Public Prosecution Office doesn't see any reason to prosecute you. Yeah, although in the current responsible disclosure policy, it's like that I cannot download any information from a server. And actually in this court cases, they went a bit broader than that. So shouldn't that part be fixed? I recently found a semi-open FTP server and I thought, yeah, what's on it? Let's download it. But then I'm already in violation according to the responsible disclosure and actually continue trouble with the company. Yeah, well, the idea is that you do the minimum possible to show that there's a vulnerability. So for an FTP server, then if you just do a directory listing and show what kind of files there are, and if you can already guess by the file names what's in those files, then you can probably use that as a statement saying, here, I have access to your customer database and that's not good. For, I mean, for databases, if you get like a MySQL access or something similar, if you just get the row headers, then you show that you have access without downloading any sensitive data. Yeah, okay, okay. But yeah, and shouldn't we shield the disclosure, the security researchers. So maybe have an organization set up with national security. So we actually tried to do this with the policy. If you still feel that you're not protected enough or that the company, if you think that the company will take aggressive action against the security researcher, then you can contact us, the NCSC, to do the disclosure. You only do this for vital sectors, right? No, I mean, for vital sectors and central government, we are the first point of contact or for most of the companies that we are actually the first point of contact and if something goes wrong, we're responsible for dealing with the issue. If you have found the security vulnerability in any other kind of organization and you reported it to them and you at least tried and it doesn't work out, then we can try to help. Okay, because yeah, my finding is and that of friends that we find a lot of shit and every time we report it, it goes bad and we just stop reporting it. Yeah, I can understand. I mean, we have limited capacity. We can't fix the security of every company in the Netherlands. But if it's very serious, if it affects a lot of people, then we will at least try. Thanks. Thanks. It's good to know that things are relatively civilized in the Netherlands. Can we give an overview of what is happening in other countries? All the rest of Europe, the United States, China, Japan, more economically developed. And the related question is what is a reasonably safe conduct in other countries because the laws and practices vary a lot. So how do you stay on a safe side if you are doing this in other countries and also third question, what happens when the hiker is in one country or a security researcher is in one country and the company is in another country? Yeah. So first of all, I'm not a lawyer and I'm not an international lawyer. There has been a publication so far it's only been in Dutch and we're trying to get it translated of what happens if you're a Dutch researcher and you try to report the vulnerability in Germany. And it could be that if the company in Germany is really pissed off, then you could be in danger of jail time in Germany. But that's very special circumstances. I don't think that that's actually going to happen, but in theory it could happen. And something similar like that could happen in other kinds of countries, especially in Europe where there's all kinds of agreements between different countries on how to deal with these international issues. But to come back to your first question, the developments in other countries is that they are seeing the light that the path of the Netherlands has shown them. I was completely surprised by the US Department of Justice putting out this framework just last week because I thought that they were really not a fan of this whole idea, but they actually did and they actually outlined very good ways of companies to actually safeguard security researchers from prosecution by the companies but also by the government. And I mentioned before the Belgians, the French, Italy, Hungary, Estonia, those kinds of countries are seeing the light and they're doing stuff. In relation to China, it's difficult. There used to be some kind of disclosure platform there which had a pretty progressive fixed publication date of 90 days. So you reported it there, the organization was notified of the vulnerability and after 90 days, no question, the stuff was published. The weird thing is that last year that organization that was responsible for maintaining that platform and the platform with it disappeared off the map. It seems that they pissed off the wrong people and disappeared. I have no idea what happened. But similarly, on the other hand, the Chinese network manufacturer, Huawei, they actually have a disclosure program in place. So even in China, this idea is accepted and they seem to be making the right movements. And there are others here who have more experience disclosing internationally. He's sitting right there and he may help you. If you want to discuss that, I think he'd be happy to tell you his experiences in doing that. Does that answer your questions? Yes, more or less, I just can't try to add my understanding that if you want to stay on a safe side, you don't want to hack anybody else's network.  Then you are on a safe side. Yes, but even then... Because no data is compromised. But even then, that doesn't mean you're protected in every country in the world, unfortunately. Or even in that case, you can successfully get sued and held responsible. That's bad. I'm afraid that's the case. Next question, please. Hi. When a company has a vulnerability disclosure program, it explicitly states that you are not allowed to do this. For example, you are not allowed to use automated scanners. And you use them and find vulnerabilities. Is it legal or isn't it? It will depend on the case. And this is a very lawyerly answer. But it will depend very much on the consequences of what had happened. Usually companies put that in place because they're afraid that their equipment is going to fall over if every hacker around the world is going to scan their whole network. And it will. If everybody starts scanning everything, then things fall over. If you do that in a sensible way and you do this in the least invasive way possible, then I think you're probably on the safe side. But you shouldn't do this everywhere. Okay, but you are on the safe side. But imagine the company is still mad and reported to the police. Yeah. Well, then the same thing applies, like I mentioned in the court cases. If you acted responsibly, then the public prosecution office is probably not going to prosecute. Yeah, that's the best answer I can give you. Okay, thank you. Next question, please. The explosion in the Netherlands was, yeah, happened from a lot of different fronts at the same time. Media attention and politicians. Can you put up the mic a little bit? Yeah, awesome. There was a lot of attention on the subject from different fronts. Media attention, the civil servants like the NTSC or also big companies who gave a good example. Is there anything from different crown to do to improve the situation? So, yes, so we should be really thankful to you also because you did a lot of hard work for the situation in the Netherlands. But I think that that process also showed that it takes a broad support within a country to actually make this into a policy. And I think that this actually happened in the U.S. as well, where there are a lot of companies in Silicon Valley which already had disclosure programs and even bug bulging programs. They're paying researchers to do this. And they were saying we would really like this to be legal because we want more of these kind of reports. I've had discussions with the policy departments in the Netherlands and all kinds of other lawmakers about how we would approach this in Europe. And I've advised them against making this regulation in Europe. Making vulnerability disclosure programs mandatory in all of Europe is not going to make this into a better place. What really should happen is that we talk to the governments, that we talk to all of the international corporations and organizations, and we talk to the security community to make everybody receptive to the idea that this is possible. And once a country reaches a certain level, then they will automatically start looking to other countries that are doing the same thing. Hopefully look into the documents that we published, that the GCCS published, that the American government published. And they will see that there is a lot of help already out there in how to do this and how to implement this and how to make these kind of processes so that it actually works in the right way. And we now have a couple of years showing that this is actually very successful and that everybody is happier now that this is possible. We have time for one more question. So if anyone's interested, I see a hand running. And please eat the microphone. The people watching the stream will be very thankful. Hi there. Hi. I do quite a lot of physical vulnerability research and it seems that the physical world is lagging well behind the digital world. There's no such thing as vulnerability disclosure programs, nothing like that, no bug bounties. I was wondering, is there any way we can sort of push those, push the physical security industries the same sort of way as the digital security industries are going? Yeah. How do you drive that, you know? Because a lot of the companies don't want to know. No. The physical companies that are doing software are usually lagging behind in all of the developments and they are unfortunately also behind in the developments of vulnerability disclosure. One exception is car manufacturers. The car manufacturer Lobby actually put out a statement saying, we take responsibility for all of the software that's in cars. And that's actually a pretty big step because there are hundreds of vendors supplying equipment, parts and software for cars. And the car manufacturer taking responsibility themselves for coordinating vulnerability disclosure is actually a big step because they own the process and they're actually very receptive to it. And one of the things that happened in the US before the idea of the vulnerability disclosure is they have the Digital Millennium Copyright Act that's loathed here. But one of the good things that's in there is that every once in a while it's reviewed and one of the reviews happened last year and they put in an exception for cars and medical devices. So as a security researcher, in the US you are already allowed to research cars and medical devices. The FDA in the US has also taken a big step saying that if a company, a medical company becomes aware of a security vulnerability in a medical device and they are able to fix it within 90 days, they don't have to do a recertification. So those kind of nudges in policy or in regulations can really help make other kinds of organizations receptive to the idea of vulnerability disclosure. And I'm seeing that more governments are starting to realize that this is actually a way to go. That we have to help other kinds of organizations that have to do with physical devices to make them more receptive to vulnerability disclosure because physical devices tend to be a risk to people when they become vulnerable.  Thanks. Thank you, Jeroen. Let's give Jeroen a warm hand.


Enrolling learners There are a number of ways in which your students can access your course. For example, your administrator might enroll them for you, or they might be automatically enrolled from outside, or they might be able to create their own accounts on the Moodle side and then enroll themselves into your course. In this video, we look at how you, the teacher, can manually add students to your course and how you can find out the other ways in which they can access your course. If we click the Participants link in the navigation drawer and then click the gear there, we see Enrolled Users and Enrollment Methods. Enrollment means how students get into your course, whereas Authentication means how they get onto a Moodle site, which we don't look at here. If we click Enrollment Methods, we can see the different ways that they're able to access or be enrolled into a course. The first one we see is called Manual Enrollments. This means that students can be enrolled into their course by the teacher. Note that, for security reasons on our Learn Moodle basics MOOC, Manual Enrollment is not available. Guest Access here has its eye closed, so that means that currently, no guests can look at this course. But the teacher can enable it by clicking the icon. Self Enrollment means that users can enroll themselves by clicking a button. This is currently disabled again because the eye icon has a line through it. Clicking it to open it would allow us to give students permission to access the course themselves. Now, remember that on our Learn Moodle site, teachers are not allowed to enroll students directly into their practice courses. But if we go to a different site and a different course, we can see how to do this. To enroll users manually, we click the Enroll Users button, and then select the people we want to enroll, and the role that we want to give them will choose Student. As a teacher, we don't normally have the rights to enroll other teachers into our course. We have to ask permission to do that. From this screen, we can search for and filter users by role, enrollment method, group, status, and we can also edit and delete their enrollments. Now, this method is fine if we just have a few people to enroll into our courses. However, if you have many, it might be worth exploring other ways of allowing them access. In this course, when we click the Enroll Users button, we can see an option, select a cohort. A cohort is a group of users created by the site administrator or a manager, and we can add cohort members individually from here or from the enrollment method screen where we were before using what's called the cohort sync method. And note that as they are added, you can also add them to any existing groups in the course. But you'll only see this cohort sync method if your site has set up cohorts.


Hi, welcome to Seymour's World commentary on Think Tech Hawaii. You can find all my commentaries and Seymour's World episodes on the Think Tech Hawaii website. I welcome your comments by email, text, or phone. You can reach me at Seymour.Kazimersky at gmail.com or<PHONE_NUMBER>. Today our topic is Look in the Mirror. When you wake up and look in the mirror, do you smile at yourself? Do you inhale confidence and exhale beauty? Or do you sigh and look at the monotonous life around you and allow it to influence how you look at yourself? Mirrors are tricky. Not only do they reflect the things standing before them, but they do so while constructing false realities, sometimes not necessarily good ones. That said, what if we chose to speak differently and think differently? One of our expectations circled on something different other than imperfections we wish were non-existent. Here are six things that we should notice about our reflection. First, our eyes. Before looking into the mirror, close your eyes and as you open them, look directly into them. Don't waver. Look at your hopes and dreams and passions. Think about what matters to you in this day, in this moment. Think about the things you love and try to see all of the emotions and compassion that sums up your inner beauty come to the surface. Give yourself a chance to see your own humanity before looking anywhere else. Second, your breathing. Let your eyes then drift. See the rise and fall of your eyes and shoulders as you slowly breathe in and out. Look at the tangible signs of your existence. Breathe in hope and optimism and exhale them in your own image for all to see. Allow yourself to find peace in knowing and seeing that you are alive and allow yourself opportunity to enjoy this fact. Second, your heart. Look to your chest and put a hand over your heart. Feel the constant beating. As long as blood is flowing through your veins and air is in your lungs, you can be and do whatever you please. Imagine what you would like your life to be like and feel that flowing through your body before you go out and work to achieve it. As long as there is a beating heart within you, you keep dreams and keep these dreams close to that heart.   Run it through your hair, if you have any, and marvel at how your body continually grows and reproduces itself. Your body, no matter how it looks, what color or what anything is a near indestructible vessel for your hopes and dreams and things you love. What is beautiful is what your body holds, but never forget that your body continually produces opportunities for you to grow and change and progress through this life. Number four, your feet. Look downwards towards your feet. Normally, escaping your gaze and mirrors, they stand rooted to the ground they brought you to. You are capable of going anywhere in this world you want. Your feet are symbolic of such a fact. Appreciate their stability, their strength, and message of hope. Take a moment to build faith that they will carry you without fail and without complaint. Number five, your smile. Yes, returning to your face, the last thing to look at should be your smile. While looking in a mirror, your smile dictates how you will leave that mirror. Smiling to yourself allows you the opportunity to affirm the beautiful parts of yourself that no one else sees as often as you do. The ones that you just spend time focusing on. Number six, the takeaway. It is easy to blame mirrors for what we see, but truthfully, mirrors are unbiased. It is humans that carry the bias. With varying levels of negativity and defeat, we often approach mirrors with hopes to see something different from what we know is there. But lose this when we open our eyes. Love your reflection because you are amazing. I hope this helps you have a wonderful and a great day. My name is Seymour Kazimurski. You're watching a commentary of Seymour's World. You can find all my commentaries and Seymour's World episodes on the ThinkTech Hawaii website. I welcome your comments by emails, text, or phone. You can reach me at Seymour.Kazimurski at gmail.com or<PHONE_NUMBER> Aloha.


Let's record on this computer. OK, so it's a great pleasure to have Jacopo Mastromatel from CFM in Paris to give this joint work with Thomas Mehdi and Mikhail Benzakhen on what really is a financial instrument inside from cross-impact. Thank you. Yes, thank you very much. It's great being interest at least virtually. I hope that as soon as we can, I'll be able to be there physically for the moment that's more than enough. So yeah, try to come up with this title, which is meant to be a bit catchy. And what I'm trying to do is try to justify and to build some narrative around the research we have been doing. The target was really to build a model in which you explain how prices of correlated assets move as a reaction to trades. But you can give some sort of interpretation and build up a story of this. So the idea is to try to guide you through the story that comes out. And the other angle with which one could see this work, which is maybe more related to what you do, is that you try to fit these models of price responses and the world is noisy. And you have a large dimensional system. And so you need some general principles that guide you. And if you're able to pick the right principle, then you can vastly reduce overfitting. So I think that's the message of this talk that doesn't necessarily only relate to finance, but it's a bit general and it relates to learning more in general. So first, the plan is the following. I'm going to formulate a question. I propose a tentative answer in an infinitely liquid market. Then I formulate the same question for a market in which liquidity is finite and you have transaction costs. And I'll try to give some answers. Someone based on common sense and common sense takes the form of axioms and principles. And another one is data. And I'll try to emphasize how those axioms play with the empirical results. So I start with the first part. And then I come back to the title and tell you what really is a financial instrument. And the emphasis here is on what really is a financial instrument as opposed to two or three. How do you count the number of financial instruments? And this seems a trivial question, but it's actually not. So I have listed some corner cases. First, if you're trading stocks like Apple, you can trade it in multiple markets. You can trade it in NASDAQ and bots. In the US, you have like 13 venues in which you can buy it. You would like to say that you just have one single entity that you're buying or selling at this, the fact that you have multiple markets is artificial. So you would like a notion that is able to capture that. For future markets, say that you're going to buy oil, you could be buying oil at different maturity. So you can buy a contract that expires in 12 months. You're going to have the crude of 12 months for now or a contract which gets you crude 15 months from now and so on. So you have many of them, but you think that the intrinsic dimensionality of the space is much smaller than all the number of possible maturities. And options. I don't know how many of you are familiar with options, but you can build an example which is similar to the futures one. But in that case, you have a B dimensional space. Somehow you can, an options is a contract that entitles you to buy or sell something at a given time, which is dimension one, and at the given price, dimension two. And now it's like in the futures example, but you really have a surface of products and you would like to count how many extra products you have. And another kind of financing example is a calendar spread. Sometimes in finance, you have one financial instrument which here is denoted with, whose price is denoted with P one. And you have another instrument which is P two and sometimes you want to buy one and sell the other one. So you build a product which is the linear combination of these products. And so you have three products which are product one, product two and product one minus two. And what you would like to formalize is notion that actually you have just two products. And what I'm going to argue is that you cannot do this. If you have, as you have finite liquidity, this sort of naive intuition breaks down. So the simplest possible operational answer to the question of how do I count financial instruments? Let's say you're an asset manager. You want to enter in a certain risky position. And the way in which you count the number of different things in which you can invest is the following. You compute the correlation matrix of price variations. You diagonalize that matrix and you plot those eigenvalues. And you could say, well, how many instruments I have? Whatever mode is a zero, then you cannot put money on that. You cannot trade it because it doesn't fluctuate. But any other thing, it can be scaled up to target risk by using leverage. And when it comes to how should I name these financial instruments, you can label them with the index of the eigenvalue. So these lambda a's are the eigenvalues and a labels the eigenvalue. That's really a naive answer. And how it's usually written down in formulas is that when you try to solve a problem which you have some signal in which you want to invest, but you want to manage your risk, since the fifties people like mean variant schemes. And one way to build a principal manner to do it is to build some sort of Lagrangian, which is this L here. I don't know if you, do you see my mouse? So yeah, can I? Yes. Okay. And you have a predictivity term. So Q is your position, how much you own of that asset. And this mu is your expected price change in the future. But you could say, well, I want my risk to be capped. And so in order to cap it, you introduce the Lagrange multiplier, which is this gamma here. And you say, I want to have a constraints on how much risk I realize, which is a quadratic form that you build up of your position, times this sigma, which is a covariance matrix. And when it comes out to the solution of this problem, which I'm sure many people are familiar with, just a linear quadratic optimization problem, then you should take this productivity thing, put it in the numerator and divide by the risk. And as everything are matrices, you need to do it with matrix inversions. But the natural interpretation is in terms of modes of risk. So you say, I have a productivity, which is this mu. I project my prediction on the modes. So I get this mu alpha, this is projected productivity. And I divide it by this lambda a square. And what are those? These are the modes of the covariance matrix of prices. So I introduced the composition sigma equal, sum over a number of modes on the projection on these modes, which is this O matrix and the corresponding eigenvalue. So when it comes to how do you count products, I can say modes of zero volatility are actually infinitely expensive because they don't fluctuate. And in order to trade them, you need to put a lot of dollars on that. You need a lot of leverage and they're infinitely expensive. The modes for which the lambda is small, okay, you can leverage them up and get to target risk. And another point is that you should be careful to overfitting this lambda a parameter. And this is just a simple example to fix a notation. First, you compute the correlation matrix of price. Say in a two-dimensional case, you have one parameter, which is rows, the correlation between the two assets. You diagonalize that, so you have the projection. In this case, you just have one absolute mode, which is this one-one component of the O matrix here. And you have a minus one-one mode, which is the relative modes. So you have like a center of mass and a relative mode. And you have the eigenvalues, which are one plus row and one minus row, which are trivially related to the correlation. So this portfolio, which you can put money are the center of mass. Its risk is square root of one plus row. And the other portfolio is the relative mode and the risk that you can put is one minus row. And for row equal to one, that relative mode will not fluctuate. You won't be able to bet on the difference between those two prices. And there are a number of caveats, which are quite substantial, which are abstracted away here. First, there is a statistical problem. How can you say if lambda is actually zero, which is very relevant in a case in which you have a large number of instruments? Actually, you can see easily that if you have more products than observations, then just out of linear algebra, you see that you have some combination of prices that never fluctuated, but it's not because really that mode should not fluctuate. It's just that you never saw it fluctuating on data. This problem you can deal with that, it won't be discussed. You also have another subtle problem, which is dynamical, which depends on the fact that I have not been talking about time scales. I didn't say if there are price variation one millisecond, five minutes, one hour, or one day, and the picture of correlation that you could have might vary. For example, in the apple, which you might trade in different venues, actually prices, they tend to go out of line at the time scale of some micro to milliseconds. And so at that time scale, it does fluctuate, but if you go to say to the daily time scale, you can completely neglect that. So you have a hidden dependency of all what I'm going to say in the time scale. I'm going to consider one minute, but the actual thing that I'm going to discuss is the liquidity. What happens to this naive labeling of actual financial instruments that I have proposed if liquidity is finite, which is today's topic? So let's take the example that I have with disaster manager that wanted to invest. He had this predictor, Mu, and he wants to invest a fixed amount of risk, say $10 million, and the disaster manager would like to invest this $10 million on the June expiry of crude oil future. And then you see that the June contract is not very liquid. I say, okay, then the contract expiring in May and the one that expiring July, they're very similar. They fluctuate in a similar fashion. Can I dilute my trading in these products so that I add up liquidity given that they are almost identical? And here I'm using to ill-defined concept, which is adding up liquidity and almost identical. And I'd like to make this blurry notion a bit more precise. And the question is, let's say that actually May, June, and July, they're completely identical. Does diversification work between completely degenerate products? Do I reduce my transaction costs when products are completely identical by splitting my trades along them? And can I build an effective liquidity pool notion which is able to sum up the volumes trading in a number of products? And so if I want to incorporate that, in the previous problem, I had to add several ingredients. And so I am taking back the Lagrangian I had before with some productivity term, a risk control, but now in same, actually, while you trade, you're going to move prices and this is going to generate extra costs, which I call impact costs. It's the orange term that I have. I assume it's linear for simplicity. And you have a matrix that tells you how much you move the price of product J when trading I. And then you have another term, which is a cost that scales linearly with the amount of trading which you do, which is an absolute value of the quantity trading, sort of an L1 control on the trades, times a parameter which can interpret as a fee. Each dollar I trade on that instrument, I pay a given fee, which is Psi. And when it comes to notation, so this capital Q is the position that you have. You gain things which you gain dollars that scale with the position you have built up, but you pay costs according to the trade, which is the small Q. So small Q is the derivative if you want of the large Q. And the question I was asking before was what to do if in some space you has complete the generosity. Let's say in the two-dimensional example I had, what if rho is one, you have 100% correlation and you have a kernel of this sigma which is larger than the empty set.  And so if you have that two-dimensional example, you should do half of your trades on one products and half of your trades in the other one because you're somehow hiding yourself from the market flow that you would otherwise measure on those products. And on the other hand, yeah. Yeah, no, sorry. So if I understand so, the question you are saying, you are concerned with when you have finite liquidity is that you want to split your orders across similar assets, essentially. Exactly, yeah. Now you have that this similar asset as a correlation matrix and also a cross-impact, essentially, in the sense that you have one, also the other. Now, these two objects must be related, right? I mean, they are essentially one correlation, one the response function of the same system. It's precisely the content of the presentation. So how do you build in a principal fashion this lambda respecting some consistency relation that you would expect from such a system? Okay, thanks. Yes, and on the other side of this trade-off, you have linear costs. And this is quite easy to interpret because you say I have five products. And if you forget impact costs, which one should you trade? You should trade the cheapest one. If one has a fee of five bits and the other one has a fee of one basis point, then you should go for the cheapest contract. So somehow if you want to solve this trade-off problem, you need to find a sensible way of modeling this cross-impact matrix. And actually, the fact that dilution works, it means how should you put indices in the equation for this lambda, which is this cross-impact matrix? And you could argue two different things. You could say indices are in the space of physical products. So these matrix should be somehow diagonal. And in that case, dilution works very well. And you can prove that if your impact is actually non-linear, but if it's got an exponent delta, say larger than zero, then your dilution is n to the minus delta where n is the number of products. Basically, it's a convexity argument that if you have the sum of the n volumes that gives my target and I split as one over n over n things, then I gain something by splitting. But another way of putting the indices is to say, indices are actually in the space of modes. So my cross-impact matrix, first I project on the basis of these correlation modes, which basically means that this lambda and the correlation matrix they commute. And then you have no dilution because the market is understanding that you're actually splitting your trades, but actually you should put them together. So projecting in this O and responding with some common liquidity pool and then projecting back to the physical space. And so the problem in the most general, or say the simplest way to frame this problem is how should I build a consistent coupled theory of price variation and traded volumes? And if you want to have a simple and sensible model for cross-impact, it can be of this form, price change equal to lambda, which is this cross-impact matrix Q plus a residual. And I would argue that it's simple because I have no time dependence on doing a static model and it's simple because the impact exponent is one. So I have a linear impact model, which leads to a quadratic cost model. And we have some proposition on what to do with non-linear stuff, but I won't be discussing that. And it should be sensible. And sensible means explaining well data, which in some sense means that your residual should be small, but you also wanted to satisfy common sense. So this lambda should be compliant with suitable axioms. And here I am formulating the question of not fitting this lambda from data and saying, let's consider space of all possible models. How should sensible model be made? And the first thing is, how should I take the building blocks for this model? And let's assume that the world is Gaussian, price variation are Gaussian and the volume's traded are Gaussians. And then you don't have many things to measure actually. In a zero mean world, you just can measure quadratic variation and they completely specify your model. So you have quadratic variation of price, which is the price covariance sigma. You have volume covariance, which is this omega and you have a response function, which is a price change Q. And the question is, which reasonable formulas you can build by combining the symbols together? And this slide is just to show you how these building blocks look like. These are, I've taken three products. So it's a crude oil contract. I have two expires of this contract, the next two maturity and the following one, which are the first two products. And the third one is the calendar spread, which I was mentioning, which is just long the first contract and short the other one. It's a linear combination plus one, minus one. As you can see in the correlation matrix, which is the first plot, the successive expires, they're very close. They're 99% correlated. Whereas the third thing has positive correlation with the first maturity and negative one with the other maturity, which is what you expect by construction. On this omega object, which is the covariance of price variation, you see a strong diagonal pattern. So in the upper left corner, you see that there is a product which is much more liquid than the other, which is the so-called front month maturity. The product which is closer to expiry. The other one you can see in the center, which is much, much less liquid. And then you have the third one, which is the calendar spread, which has a very thin liquidity. And this reflects in the plot of the response function, because you see these vertical stripes and these vertical stripes are reflective of the fact that actually the price variation, they might have similar scales. So you don't have horizontal strikes, but the volumes are very different. Some products are very much, much more liquid than others. And actually, when I first presented this, the audience, they had like a device to answer to multiple choices test. So I was asked to do quiz, and this was the quiz, but I will save you from answering to this. But what I was doing is to build some candidates formulas and asking people if they like the formulas for cross-impact or not. The first one is takes the price covariation and do the third power, because why not? The other one that you can do is to say, okay, I don't believe in cross-impact. I think the cross-impact does not exist. It's diagonal. So it's diagonal of sigma, which has dimension of price, divided the diagonal of omega. So they build something which is one over volume and have a delta V over delta V somehow, which is something that satisfies dimensional analysis at least. You could try to build a cross-impact model, but I say I want to do the same thing, but interpreting the one half symbol as a matrix factorization. So this sigma one and a half could be a PCA decomposition and I have a cross-impact model that satisfies dimensional analysis. I can do the same thing, but based on response. So I can build the same combination, something that transforms like a price, but it's built out of responses rather than price covariances. And or you could do something which is along the lines of what I was doing. So respecting dimensional analysis, which is on the numerator, I have this lambda A, it's the price variation. And on the denominator, I take the volume covariance, but I project on the modes of the correlation matrix. And I build something that scales well dimensionally. So I have to take a square root of this omega projection. And then I have a mysterious formula, which I'm going to talk you more about later, which we like a lot and we've been working. So in this seminar people have been answering and I was saying, so the first formula does not make sense. It's a price to the sixth power. It doesn't make sense because impact should be a price over a volume. Second point is that you're assuming the cross-impact does not exist. Third point, we call it the whitening model because somehow you're going to a basis in which things are uncorrelated and then building a model in that base and then rotating again. But you can notice that this model is not symmetric. And the second mode is r omega minus one. I don't know if some of you recognize that, but that's just what you get out of a linear regression. Linear regression usually the regress parameter is average of yx divided average of xx. And that's precisely what maximum likelihood gives you, which is not symmetric. This is a model that we have proposed as first in which you basically assume that the cross-impact matrix commutes with the correlation matrix and modes of the correlation matrix are also somehow modes of liquidity. And this is another model which we like more that we have proposed later. But the take home is that you have many ways of combining those symbols and you would like to have some maximization procedure to classify model. And somehow another speculative message is that this model are not very important per se. Probably what holds more value is to validate on empirical data, which symmetries does the data satisfies. And because establishing the fact that the symmetry is valid or not has much more deep value than validating the fact that the simple instance of a model is good or not. And another point is that if you axiomatize, then you have a sort of a plug and play approach. You can enforce property in the cross-impact matrix by controlling how your inputs look like. Somehow you control your output cross-impact matrix by knowing what you're putting in a disciplined way. So how to enforce this common sense, this axiomatization? Well, first you can try to get dimensional analysis right or gauge theory if you want to somehow glorify dimensional analysis. First you could say, well, I want to predict price variations. There are costs expressing units of cash, say dollars which means that my cross-impact matrix, given it's a function of the sufficient statistics, price covariance, response and omega, if I multiply your prices by alpha, then the cross-impact matrix multiplies by alpha. And then you could go further and say, actually, the volumes, they can be multiplied by arbitrary factor. If tomorrow I say that all volumes are multiplied by five, I can divide all prices by five and the market will behave probably just the same. Which is true to a certain extent, then you have some symmetry breaking things like minimum tick size or microstructure related limits, but you expect the symmetry to be respected. I call this split symmetry because in stock markets, you sometimes have stock splits that volumes are augmented by a factor and prices, they reabsorbed that factor. And this reflects in the property here. Prices get a D to M minus one and volumes, they get a D. This gives you a functional equation for your cross-impact. And then you can, going even further, cost could be invariant under rotation of products. If you wake up in a world in which you can trade linear combination of products, you expect things to behave a bit the same. So the first axiom, we believe a lot in this. The second one a bit less and the third one even less. For reasons which are easy to justify and that relate to the fact that the money that you make or the P&L that you make have symmetry breaking terms sometimes and sometimes they're large, but to a certain extent it's natural to build models in which those symmetries are respected and then go progressively and break them. Another property that you expect is that for uncorrelated product impact should be additive. So if I take this cross-impact matrix, but everything is diagonal, then it's the sum of a single product impact on the products. And then you have very important properties which relate to no free lunch. No free lunch in the financial markets usually is the principle of no arbitrage. And in this specific instance, no arbitrage means that you cannot use cheap products to push prices of expensive products and gain money. So you cannot pump and dump, use pump and dump schemes in a real market, which means boils down to the fact that lambda is a positive semi-definite. And this prevents the existence of static arbitrages. Somehow if you also ask for the absence of dynamic arbitrages, then you should have a matrix which is also symmetric. And this is a reasonable thing to impose. One should you impose it well if you're using the output of this research for an optimizer, some algo trader that is taking decision, you want the thing to be controlled not to start to pump the market or to trade infinity. So you need something that doesn't believe that it can arbitrage the market away. So that's one good reasons to enforce this. And the other one is if you're a strong believer in market efficiency, which to me brings to the next point, which is this stupidity chart which was made famous by an Italian economist, Cipolla, that divides people in these four categories, people who benefit themselves and the other who are smart, then you have people who benefit themselves and not the other who are bandits, then you have people who bring losses to themselves and the others that you can call stupid. And an open question empirically is the one that concerns the presence of arbitrages. So it's not clear to me whether it's possible to arbitrage the market and saying that it's bad behavior because it's forbidden by law, prevents you from pumping and dumping, whether you could not doing that because it's forbidden and because it's unprofitable or because it's forbidden, but it could be profitable.  And this is, I would say, still an open point. So now I come to one of the properties which I think is one of the most relevant. So it's a bit technical, I hope not to lose anyone in here, but it's really central. And the idea is that if you have combinations of products that never move, so which means that if you have a kernel of this Sigma and you never measure fluctuations, you could have a condition called weak fragmentation and variance, your impact should conserve that. If something never fluctuated, even if you trade in that direction should not be able to move it. And in formula, you write it by saying that if you project your impact on the kernel of this Sigma, then that should be zero. And then you could ask for more, we call it semi-strong version, which is actually you'll have a projection also on the other side, which means that regardless of how you trade those zero modes, you have no influence on what happened on the rest of the world when you're on the thing that actually fluctuate. And the strongest version that you can ask, it is a requirement that actually everything just depends on how the modes that actually fluctuate behave, that those modes that never fluctuate, they have no influence whatsoever on anything. And so somehow the property I had before, this fragmentation in variance, it tells you that if you have no fluctuations, impact should not move and you would like to control it from the first principle in this way. And there are other properties which are, we'll see are a bit related, which relate on liquidity. And this again, it's a bit of a long shot, so I'll go slowly. Let's say that you have some products which are very liquid, their liquidity is proportional to epsilon. So this means that you have this vertical stripes as I was showing you before in the response and you'll have a strongly diagonal dependency in your Omega because you have this subspace of stuff which is hard to trade. And what you would like to ask is that illiquid products should have finite influence on liquid products. Somehow, if I have $1 invested on a very thin liquidity product, you should not be able to use that dollar to move disproportionately, like epsilon to the minus one, something which is liquid. This we call it weak cross stability. And we also have a strong version of that, which is the fact that illiquid product should not change interaction among the liquid products. Somehow, if I introduce a new instrument and no one is trading that, the behavior that you measure on that should have no influence on the stuff which is liquid. And finally, these two are nice properties and they have a property that you should not want to have which is a self stability, which is somehow dangerous, which is in the part of the plot in which you have the projection on the illiquid space.  Actually what you would like to have is to have a model in which things are self unstable, in that you get punished if you trade illiquid stuff. Somehow you would like a model that tells you that you don't want to be the only one in the market trading some name. So as this concept is a bit complicated, I'd like to illustrate how it translates in the model I was presenting. Let's say you have a two dimensional word with an omega. So there's a liquidity matrix which is diagonal. And then I say that the second product is illiquid. So I go from omega to this omega prime and the illiquid is this epsilon and this PS project projects on that space. If you have a model with direct impact, it is cross stable because your cross term do not explode. You have, it's not self stable because it's telling you that if you trade the illiquid product, you pay a lot something which is epsilon minus one which is nice. If you do a maximum likelihood estimation, it's violating this cross stability. Maximum likelihood is telling you that if you trade the illiquid product, some say small liquidity stock, which is small cap, you're able to move the price of Apple. You can use that product to move the price of Apple which you think is unreasonable because it would lead to arbitrages. So doing naive maximum likelihood from this respect is not good because it doesn't respect this common sense requirement. This model that we have been proposing, which we have called ELM, Eigen Liquidity Modeling, which you project in the space of modes, actually it doesn't have any epsilon whatsoever because it's telling you that anything with which exists are the modes of the correlation matrix. And even if you trade the liquid products, you're not going to move it a lot because it's correlated to something that has finite liquidity. So it's self-sable. Somehow it's telling you that you should not be afraid of trading illiquid things because they're correlated to liquid stuff. And finally, the reason why we like this Kyle model is that somehow it gives a non-trivial answer to this point. It builds an interesting scaling variable which is this epsilon minus one square root of one minus rho square combination. And how one reads this is that should you be afraid of trading the illiquid product? And the answer is no, but it should be very correlated. So somehow you have a limit which is not well-defined. You have a non-commutation of limits and you should ask yourself, is it more illiquid or is it more correlated? Somehow you should compare the level of illiquidity with the level of correlation. It's okay to trade illiquid stuff if it's correlated with liquid stuff, but if it's very liquid then you should be careful. And I think it's quite a smart answer from this model gives. And another properties that you might want to ask is that does the cross-sectional structure of returns is the cross-sectional structure of returns consistent with the observed ones? And this property relates to the fact that lambda omega lambda transpose should be equal to the correlation matrix, which to me is not that clear that it should be the case because you could have that a part of the covariation of prices of financial instruments comes from liquidity reasons and it goes in one direction and you have part of price variation that comes from news and goes in another direction. But it's good to know which models have these properties and which ones do not. And finally something that we discovered just very recently but which I find extremely interesting is that these two key properties for building a cross-impact models, these invariance fragmentation invariance properties and these stability properties on liquidity, they are related by gauge symmetry. And the connection goes as follow, you take a model which is split invariant, so in which you can rescale away the volumes, then you can map a theory with weakly fluctuating direction to a theory with illiquid direction. And this just goes in one direction but if you also have rotational invariance it goes the other way around that if you have a theory with illiquid direction then you can go to weakly fluctuating directions. And it's got a lot of very deep connections I think which is the fact that if you have fragmentation invariance semi-strong fragmentation invariance then you have weak cross stability and if you have strong fragmentation invariance you have strong cross stability. Somehow it's enough to ask for fragmentation invariance and you have good behaved models in terms of liquidity. And yeah, so there was the second part of the quiz and the audience could answer to which models they like the more now which shall skip. And the take home from this part here is that the best choice that you have might depend on the application. And the other point is that what you consider reasonable from first principle might not be met in real data which brings me to the last part which is the empirical one. Before that I just wanted to show you how these different models relate with respect to these different properties. You don't have to check all the ticks. I just want you to point out that the sky model it's nice because everything is green. Somehow all the properties that you could have are verified by these models. And actually just the total number of models we came up with is quite large was 11. And that's why this axiomatization is nice because you check which difference they have they have on the basis of which properties they do satisfy or not. So now data which I had promised I started with a set of illustrative examples and back to this crude case. If you remember I had three products front month future contract on oil the next to maturity oil future contract and the relative calendar spread plus one minus the other. And you see the results of maximum likelihood on the left. And what you see is that it's clearly non-symmetric and you see this strange vertical stripes as this trading this combination plus one minus one actually was able to have a disproportional effect on crude which seems a bit unreasonable. And on the other hand you have this Eigen liquidity model that tells you that basically you have just one single product. And the single product if you remember these two crude expiry they were 99% correlated. So let's assume they're 100% correlated. It's close to reality. And then you get one single product and you see that this matrix here is actually rank one basically. You just have one mode of liquidity. It's symmetric and you're not afraid of trading the calendar spread even though it's very liquid because you say well anyway it's one single mode then one single pool of liquidity. We have also run the same example by taking anti-correlated products which is a nice example. We take US bonds, these 10 year US notes. The first expiry and the second expiry which are very correlated as in the crude example. And then we take the standards and poor mean so it's an index future which is basically the state of the US equity market. The first expiry and the three months expiry also very correlated. And they are typically anti-correlated but always accepting this COVID related days as we have discussed before. And so that's it for the correlation matrix. And when you look at the volumes what you see is that the front month expiry is very liquid and the other ones they're less liquid which reflects in the vertical stripes in the response function. And if you see which are the fitted cross-impact models actually what you see is that in the maximum likelihood as in the previous example impact is driven by the illiquid product. Somehow the answer is that the impact is a relative phenomenon. It relates to how much you trade with respect to the typical level of liquidity in that product that what's the maximum likelihood gives. Whereas this Eigen liquidity model tells you that the impact is absolute. It's not about the relative level of your trading respect to the usual. It's about the overall number of dollars that you trade which means that basically in this ELM cross-impact matrix you have two blocks which are basically one single product and then you have this cross piece. And finally this Kyle model is quite interesting because it's able to give an impact model which has a negative sign in the off diagonal block. It's able to grasp the fact that effectively you have a single product for the 10 year US note and for the SP mini things. You have these two by two blocks but somehow it also understands that it's if you trade the thin liquidity contract you might move it by a lot. So you have 4.4, 4.3, 4.3. It's the same product but you have a 59 if you trade the illiquid thing. So Kyle is saying that impact is not relative and not absolute and it's something in between and satisfying all the actions that we had put. And the other final illustrative example is what you see on stocks. Which we used in order to emphasize the fact that you can run this machinery in a high dimensional space so we take I think 400 products. These are the correlation matrix for stocks. You see that you have a strong diagonal component. You have a strong market mode which has been removed from the plot because otherwise you would see a constant color thing but if you remove the center of mass you see things which can be associated with the financial sectors which are like blocks in this correlation matrix. And actually the correlations of volumes it looks pretty much the same. So once I want to make a slide that you have the strong diagonal component correlation in both cases. You have a market mode removed sector structure and somehow the flow correlation is less correlated cross sectionally which means it's a bit more diagonal which is not visible in the figure. But what's interesting is that you have a huge heterogeneity in volume. Some stocks are very liquid. You have Apple trades three billions a day and you have something else in the sample that might trade maybe $10 million a day. You have orders of magnitude of difference which leads to estimations of cross impact matrix that have big stripes. If you use this maximum likelihood estimation and on the other hand if you do this Eigen liquidity model prescription of projecting on the modes things are very much homogeneous but you don't somehow you don't conserve this heterogeneous volume aspect and this kind model is able to do a bit of both. And my next point is that one could ask himself how much do you overfit by doing this? You would expect that maximum likelihood overfits a lot and more principal things overfit less. And what you see in this plot is that in the upper panel you see an R square of these regressions at the time scales that we have been using like one minute and there is actually maximum likelihood that does good. The bottom panel is the overfitting and actually what we are quite surprised by is that you don't overfit that much or out of sample score with respect to the in sample is like between 95% and 100% but that's because we have a lot of instances each one minute it's a new snapshot and so you're able to afford a greedy model in terms of data as maximum likelihood but if you were sampling at slower frequency like 10 minutes or some minutes then you would see that the performance of maximum likelihood would degrade and models that have a more principled that they're building a more principled way they stand better the challenge of overfitting. As you can see the one that behaves the better in terms of overfitting is the direct model because if you don't have cross-impact you don't have N squared things to estimate you just have N and this is good in terms of statistical performance but on the other hand you're losing a lot and the point is that these techniques they tend to be greedy in terms of data and you need to have enough data in order to be able to say something and this general principle they're helping you in selecting things that make sense. Another point here is the comparison of the performance of all the examples I have I showed you three examples one was about this crude future one was the example with the anti-correlation and the SP mini and the 10 areas notes and then the stocks and the upper and bottom left panel they compare the performance of the models in the different examples what I think one should say is that typically at this one minute timescale maximum likelihood it tends to perform better and the direct model tends to perform quite poorly you need to have cross-impact but somehow it's not that much evident from the stocks which are this solid continuous line in which we've arrived the number of assets from say one up to 400 by bootstrapping in order to check the robustness of the results with respect to the number of products but if you check the points there they are the other two examples so the first example which should be the plus signs they're bonds and indices and the other one are the crude oil contract example and in that case the direct model does very bad and you expect that because they're very much correlated so somehow if you don't take into account this cross-impact you lose a lot you don't lose a lot on the stocks because they're very diagonal but you lose a lot on these future contracts because they're very much correlated and that's somehow expected and then I like to show this picture because I find it quite illustrative here I'm showing the R square so the statistical performance of different models in different liquidity trenches what you see with the blue curve which is the direct impact is that it does relatively well on liquid and illiquid products but it does bad in the middle the orange model which is this Eigen liquidity model does well in the intermediate region and somehow the Skyl model does is able to get the best of both worlds and the maximum likelihood again does better for this choice of time and this has an intuitive representation you can understand, you can rationalize it by saying that actually for very liquid and very illiquid product the volume heterogeneity is very important so things tend to be very diagonal and a model without cross-impact does the job quite well in the intermediate regime you would expect a model that has cross-impact and it puts all liquidity together to go to do well but if you want to have right both the illiquid regime and the strongly correlated regime you need some good way of interpolate which brings you to this green curve and the last bullet point I have is that yes maximum likelihood does a bit better of them but it over fits more at larger timescales and I had this very last instance of this quiz and the take-homes are that well you can rule out bad models based on data but noise often leaves you with a set of reasonable models and using principle criteria for model selection might be a good idea in those cases and the conclusions first on the theoretical side the initial recipe I gave was naive labeling the modes of sigma as true products misses the liquidity effect this Eigen liquidity model predicts that they are actually the same thing assuming that correlation and liquidity they are matrices that commute but in general if you have heterogeneous liquidity these things do not commute and what actually is a financial instrument has a hard answer because you have no commuting objects and in practice you actually have no clear winner on data the application should be a tiebreaker among models performing similarly and on the other hand arbitrage condition fragmentation, invariance and liquidity behavior they play a crucial war and you would like to understand what's going on from that angle when choosing a model for example a direct model is working if you're trading is quite diagonal but it fails miserably if you're trading say modes or relative modes of products and the Skyl prescription it's nice because it gives you decent performance and many convenience features which are guaranteed exactly and I think it's all, yeah Okay, thank you very much Now, are there questions? Maybe I can start asking a question while we read your disclaimers Yes, they're always very funny So, why do you call this model skyl model? Does this has to do with skyl? I think it's a backup slide yeah, how do we came up with this model so why it's called skyl because you might see this model as arising from somehow a variational principle as an equilibrium of some game it's a Nash equilibrium of a game in which you have some market makers whose job is to price correctly the information which is contained in an order flow you have informed traders received say correlated trading signal and you have noise traders that are used by the informed trader to hide himself and somehow you have an adversarial feature in that if the market maker is wrong about his pricing model the informed trader is able to exploit that weaknesses, right? So somehow if you find an equilibrium to this game you find an impact model that cannot be arbitraged by construction that arises so the idea is to put an adversarial aspect in that that ensures that the formula is robust in that whatever weakness you put in it it would be arbitraged by someone else who's playing against you and then you guarantee exactly that there cannot be arbitraged say evident arbitrages in that Okay, thank you other questions so if not I have another one so all of what you discussed is based on say correlation matrix say linear response and this say essentially it would be I mean seems like you are assuming a Gaussian word essentially now is this correct or so higher order connections so the first point is that I think it's very bad to always assume a Gaussian word which is I think one should do more than these on the other hand if you're not even able to build a nice model in a Gaussian word you're bound to fail in a more complicated setting okay first let's do this and then move forward and this Mehdi Tomah is working on these in building non-gauss and non-linear version of that for example something that we are seeing is that the correlation structure of volumes is not linear so conditional to the trading on one asset the average volume traded the average imbalance on the other ones is non-linear and you have saturation non-linearities and they bring in effects for example okay hello yes to run to another meeting so actually you also have to run to another meeting Jacobo right yes so if no one else has a burning question so I had one so can you use these to understand how much these axioms are true in real finance yeah again this is a project that we would like to start so not focusing on the models but seeing how well you're able to reskill things somehow what these gauge invariance and stuff they tell you that you can rotate stuff or dilute things and having collapse of curves so you could use the symmetries and check how much these curves collapse and at which point you break symmetries for example I know fees and express they break rotational invariance because you have this absolute value things so you would expect something that maybe has rotational invariance when you trade a lot but that's some microstructural scale you expect to measure rotation and one could do that yeah okay so I think close here so it's one minute to three thank you very much again Jacob and and thank you all for joining the seminar and see you next time yes see you next bye bye bye so Massimo yes stop the recording stop the recording