This is once again the tower and this will be Manos talk on security enhanced Linux virtual machines and once again, the IRC channel for questions, if you're watching at home is hash dc6-talks-tower. Thank you very much. Hi guys. Thanks for coming to the workshop. Well, actually this was initially targeted as a talk but now that it has been repurposed as a workshop, I find that the audience expectations have changed. People actually seem to be able to learn something useful from a workshop as opposed to a talk where we all just chat around on IRC all the time. So, well, in order to address this raised expectations which was slightly frightening, I have divided this presentation into two tracks. The first track is where we actually do things, build kernels. We'll be compiling two kernels during the workshop. I hope that everybody has a fast machine, but we can always do it in parallel and we can continue on because there's nothing on after this apart from our boss that starts at seven. So, if people are lagging behind, we have plenty of time to finish up and I'll be hanging around afterwards to answer questions. The second track is where I tried to scare the living daylights out of you about the security threats that we face on the internet nowadays. What I believe is foundational to securing our machines. Let me see if I can get this. So, the methods parts are where we build things. Did that work? That didn't work. Then we'll go into the introduction while I talk about what kind of attacks we actually are facing. Most of us who are sysadmins already know this part. We go back to building things. Then if you aren't scared enough by the talk of all the attacks, I try to get some more rah-rah motivation in the audience. We go back to building some more things. Then I'll present the solution that this workshop is trying to sell and then we will come to building more things. I don't know what happened here, but that shouldn't have happened. Then we come to the part that is traditionally sung directly to the audience. So, getting started, we are going to build things. If I can get this mouse to work. So, we are going to be downloading the latest version of the Linux kernel. If we don't want to go all the way to ftp.org, I already have put this on Homer. I sent a couple of mail messages around. I hope that anybody who wanted to has already downloaded it. I have put up a couple of other files in the directory, which is ftp, Homer, Share, Manoj. And one of those files is called, cryptically enough, script.sh. And if you run script.sh, it will spit out all the commands that you will need to follow along. So the first one is download. Once we download it, we will untie it in some directory out there, wherever you wish. And since we'll be building two kernel images, I always like to split out the build area into two parts. We use copy minus l, so these are just hard links. So unless you are really, really short on iNodes, this won't consume any more disk space than you absolutely need to. The reason we are building two images is one of the images will be on your host machine. Why can't we just use the image that you already have? There is this thing called the separate kernel address space. We are building a UML machine, mostly because demonstrating is then build is slightly more complex than demonstrating a UML machine. But the same techniques apply there. You want, for security's sake, to have the processes that run inside your virtual machine to be in a totally different address space from the processes that run on the host machine. That way there is no way that the virtual machine processes can infect your host machine. So we get and apply this patch. It is if people want me to stop if they're still working through this, just hold up your hand and I'll pause. Apply the SKS patch, this neat little one line invocation of bzcat, et cetera, is there in script.acid so that you can just cut and paste for your build. Then we just compile, configure, compile, and install. There's nothing special to be done about configuring your host kernel. There is one extra configure option that comes in, which is PROC-MM, which is on by default. So if you just grab your old config, run make old config or something instead of make xconfig and build. It's clean old vanilla make kernel package build process here. This is going to take a while. So we'll go back to talking about why we are doing all this. Everybody ready? Well, those of us who are following around. All right, introduction and attack trends. SIRT has put forth an overview of recent trends in computer attacks and vulnerabilities, the URL at the bottom of the slide. The report states that attacks are getting far more automated, blessings of modern technology. Used to be that you would have these running scripts that would scan networks and computers for things which were open and accessible. Then they would go in and see if they could manually, somebody would come in and try to exploit any vulnerabilities. Then it would be another manual step to try to propagate the attack further. And however, the trend has been that the same scripts not only scan, they exploit the vulnerabilities, and they launch their own attacks, getting all the machines in the near neighborhood which might not have been accessible from the originating point of the attacker. They're also fairly sophisticated management, almost typically like management tools that allow you to manage all these machines that you have managed to break into and exploit. Apart from automation, the attack tools are getting better. They're building in anti-forensic techniques. They hide from the old-fashioned detection mechanisms. They are using object-oriented techniques. They are becoming modular. You can pick up an attack module and you can pick up an exploitation module and you can pick up a module that hides all traces and pull it together and launch an attack. They are becoming dynamic because they have this nice new plug-in architecture. They no longer have a static signature that you can hold on to and scan for. The most violent attacks, which thankfully most of them have not been released, are not out there in the wild yet, usually launched by various governmental agencies, are almost impossible to detect because the signature changes with every attack. Every year, the vulnerability is reported to cert double. Every year, the report of vulnerabilities on a new version of a software that is released are coming out faster. The number of people who are there to defend against these attacks is not going up at the same rate. Those of us who are CIS admins will probably attest that the departmental budgets are not rising up exponentially at the same rates as the attacks are. Firewalls. There are all kinds of new technologies that are being touted as firewall-friendly, web dev, and so on.  Firewalls are becoming more and more permeable as technologies are being deployed to get real-time data flow. The threat is also getting increasingly asymmetrical. It's no longer one guy out there hacking away at your firewall and trying to get it to your machine. There's probably one guy out there controlling a botnet of several thousands or tens of thousands of machines and trying to brute force his way in, simultaneously attacking you from a gazillion originating points. So any defense mechanism that looks at point of origination and tries to lock that point of origination out is going to fail against a modern attack. Oh, the infrastructure attacks. This is what gets the press, the distributed denial of service attacks, the worms, DNS poisoning, and attacks on the DNS root servers, attacks on routers, the Cisco vulnerability. This is an attack that is not specific against any particular target, but anybody who is using the internet infrastructure and Cisco routers would have been vulnerable to that. These kinds of attacks are becoming more and more common. I hadn't even heard of a low-level attack about five to six years ago. These are the reported vulnerabilities over the since 1995 to 2005, a 10-year record. There was a drop in 2003 and 2004. I'm not sure why. Maybe the dot-com crash had something to do with the number of machines coming on board. But the trend has been exponentially increasing. It gets worse when you look at instead of the vulnerabilities, if you look at the incidents. In fact, the incidents are getting so bad that they stopped recording it post 2004 because it no longer makes sense to record the numbers because now a vulnerability comes out, a few hundred million people get infected, and the numbers at that point are meaningless. You might as well assume that a large chunk of the internet will get infected by the next successful exploit out there. And some of those people, if they are not running Linux, it might be your users, right? Let's see if we can get you back. Going too slowly is my machine trying to give me a message. Why does this matter? Because in order for us to conduct mission-critical business as usual, whatever our mission might be, there are certain security goals which are common or something we should at least strive for, even if we don't often achieve this. The first is authentication. This means that the actor that is initiating the action being taken by the system is identified correctly, and there is some assurance that this identity is not fake. You need to know who the guy is who is typing in the commands. Authorization. This means that the actor who is initiating the actions is actually permitted to perform those actions. Me, as a regular user, should not be able to shut down Apache. I need to have specific privileges. Confidentiality means information on your machine or on your network should only be accessed by people who are authorized to do so. There is some assurance that it's not broadcast widely to anybody who just wants to be looking at it. Integrity, the same thing except that the data should not be tampered by people who are not authorized to. No web defacements by some guy who managed to hack in. Domain separation. This is desirable often in highly secure, well, computer systems and organizations to separate out your information and your computers and your access based on different zones of security. You have high security areas and low security areas. Payroll is probably a high security area. You don't want anybody going into payroll and giving themselves a 3% raise every month. And a wiki is likely to be a low security area. You do want people to come in and add in stuff. Presumably you do have version control over there. Availability is you need to have access to computing resources for authorized users in a timely manner. When 100,000 users are hitting on your web server for any distributed denial or service attack, that is what's going missing. The bullet points in green, authorization, confidentiality, integrity, and domain separation is what I'm going to be concentrating on today. Authentication is interesting, but not something that a security policy can help you with. You go into weird things like fingerprint leaders and written scanners and hopefully things that look for a pulse as well as your written pattern. I saw some eyes beginning to glaze over, so it's time for a change of pace. We are back to building things. How many people are still compiling the kernel? Well, it gives you better security. It makes your machine faster. I think, I hope so. That code path has not often been exercised because most of us who build UML machines generally do SKS patches. But it used to work. I haven't done it in six months or more, but it used to work. If you apply the SKS patch, I can assure you that it was working as of this morning on my laptop. Do you think we need a break? Do people need to ask me questions or? All right. I'm assuming that most people didn't want the hassle of actually running an SC Linux on their laptops. So this is the first time I'm mentioning the SC Linux patch. We are going to be using it in our virtual machine. I happen to also run the SC Linux patch on my laptop in non-enforcing mode because I have a confession to make. Currently, the SC Linux tool chain is broken. So even though we will get a SC Linux capable virtual machine, it won't work this week. It was working three weeks ago. It was working with 2615. It has not worked with 2616. It will probably work with 2617. And we have patches scheduled to go in in 2618. All right. So download the SC Linux patches either from there or from Homer. We create yet another build directory, this time for building the UML image, by again using hardlink tree. Apply the SC Linux patch as shown. And if you run the script.sh, it will spit this out for you. Now, compiling the UML, this is slightly different because the UML doesn't actually touch your hardware.  And so you can use a really, really scaled down configuration file. There is a file on Homer which is called UML-config that has what I use for my virtual machines. So you can just copy that as .config and start a second compile loading down your laptop. This is probably a good thing because that means your machine will probably be too loaded to be on IRC anymore. And once that is done, you can just install this Linux UML file that will be created. You can install this anyway, even if you have not compiled the SKS compatible kernel because all it does is creates a user bin Linux executable for you and puts stuff where the modules are. Questions. This is not a good sign. If there are no questions, that means everybody is on IRC or sleeping. God, either you guys are good. Using UML for this, just wondering about the virtues of UML versus Zen for this sort of exercise. It's easier to install a UML machine right now in a demo. I don't have to worry about people having hypervisors and a domain zero kernel versus a DOM U kernel. The techniques, if you have a Zen virtual machine, you can skip this part. When we come to building the root file system, the same technique can be applied for the Zen virtual machine. The paper, actually, in the conference proceedings I talk about, you could use either. I've just been using UML longer and it is simpler to deploy. All you need is a user bin Linux executable out there and you've got a virtual machine that you can run. Anybody else? This slide, the previous one to this. OK, one more. If you go to on-homer, how am I doing on time? 4.45, so I'm half an hour into 100 minutes. I have a question from Yo. Is the state of SE Linux unstable good enough for him to install it on his web server? The question is, what is the state of SE Linux currently in said? Well, no, if you're running stable, I would not recommend putting SE Linux on there. If you're running said, I would not recommend putting SE Linux on any machine that you're using for anything production. Until, I don't know if new has been processed, because prior to this last week's new processing, we were about six months behind distributions like Gen2, Red Hat, Suzy, and so on in SE Linux. Now we are at par. We've got the latest versions of all the packages in unstable, which not to say that they are working, but we've got the same bugs that everybody else has. I hope to be able to get this fixed once 2617 comes out, because the patches are already in, and we need that functionality. I'm targeting SE Linux for edge rather than for surge. I am hoping to convince enough of the powers that be with proper sacrifices at the proper altars to get SE Linux options in the Debian installer and as a semi-hard, non-blocking release goal for edge. But right now, if you are adventurous and you want to get a fresh start and be ahead of everybody else and be the cool kid on the block, SE Linux is precisely the place to be. So if looking at the attacks hasn't gotten you motivated enough, and actually, I haven't made my case of why SE Linux, why do we need a defense? Well, we are getting increasingly connected to the rest of the world. A computer without a network connection is nowhere near as useful as a computer on a network. Unfortunately, attacks are trending up, as we just saw. I once put a computer just outside of firewall. It was not in DNS. There were no links to it from anywhere. It was about 45 minutes before I got my first port scan. Within two hours, there had been full-fledged attacks. If it was a Windows machine that was sitting outside my firewall, I would have been rooted within two hours. I don't think that this is unusual. Used to be that the guys who were attacking you were just kids sitting in the parents' basements trying to get bragging rights with their friends out there. Well, that's no longer the case. The guys who are attacking you are more likely to be doing it for the money. I read an article a couple of months ago about the Washington Post, I believe. It was an interview with a hacker.  The downside of it is that the virtues of a market economy are working against us. These guys are motivated. This is what puts bread on their table. There's organized intelligence gathering out there. This happens more to companies and government agencies than to the general user at home. But that doesn't mean that there's nothing coming out at us. There are people who are coming at us looking for credit card info to install keyloggers so that they can get into our bank accounts. And they are organized. People are far more likely to lose their credit card information either from a bank or from the personal machine on the internet than they are from people stealing their wallets. And there are seek and destroy missions out there. And this is more likely to be a concern for the companies out there. Most of the seek and destroy vulnerabilities are not actually reported as such. But there are more of them than one would think. Thing that's happening is that, used to be, you would get broken into. People, the guy would come in, deface your web face, and he would be out of there. You might have to reinstall your machine, but at least you knew what happened. And it was a pain, but you got over it. Now your machine is likely to remain under the attackers control for far longer, weeks or months. And it might participate in several attacks over the period on other machines. But most of them, not enough, high enough profile for it to impact you directly. Your machine might be compromised.  This is because the attackers are getting better at being sneaky and remaining undercover. Somebody did an analysis of all attacks based on certain bug track advisories since 1997 to 2005. And I'm sorry I don't have the URL up here, but I will publish a modified version of my paper, and it is in the references. These are just the categories of attacks. These category list doesn't actually give a number or how popular the attacks were. And I haven't been able to find a comprehensive list. However, for the subcategory of web decasement, they are published numbers from zonets.org. And these are the proportions of attacks vectors that they have seen in the last three years. The most likely cause of a break-in was somebody misconfigured or failed to configure at all a server. You leave the default factory root server password on. The next one is something the person didn't get the patch up in time. If you're running a Windows box, patches come every Tuesday.  Then there were zero-day vulnerabilities, things that nobody knew about. These top three attacks are what? 17 plus 12 is 29. About 44, 45% of all the attacks are just the top three. And you see how things go down. One of the things you should notice that we're talking about a web server defacement. Most of the attacks that were successful were attacks against the service that is being attacked.  There is a seminal paper in security which has been published in various places over the years in different versions. It's called the Inevitability of Failure. The thesis here, and I believe in this, is all the security experts, all the pundits out there, all that bug track insert are doing are doomed to failure. They're fighting a retreating war, and they are being overwhelmed by the numbers. The mechanisms provided by mainstream OSs, and I don't just mean windows, are inadequate. Most of the defenses that we array out there in terms of security are things like virus scanners, firewalls. These are reactive mechanisms. A virus scanner only works after you know that a virus is out there and somebody has figured out how to scan for it. If you don't know that a virus is out there yet, you're doomed to failure. The access control mechanisms provided by any operating system post-multics, and even multics wasn't really a mainstream OS in its day, have been very weak. Usually, you get two kinds of users. There is either a super user who has control of everything on the machine, or you've got an ordinary user. There are some token efforts to have power users versus non-power users, but that's the most that I have seen, three levels of users. Really, if you're trying to do real solid security, you should be looking at more than the identity of the user in making a security decision. You should be looking at the role the user is playing. You should be looking at integrity and confidentiality requirements of the data. You should be looking at how critical this action or this process is, whether you're likely to have data loss. Just looking at, okay, this guy wants the file, let him run the command, doesn't cut it anymore. None of the mainstream operating system provides you with a trusted path. What does a trusted path mean?  People talk about cryptography as a penacea. It is not. There was a SSH attack revealed in the last month where people were sneaking credit card numbers. But SSL wasn't actually compromised. What they were doing was they had injected a trojan that was grabbing hold of the data stream before it got encrypted and sent over the network. You have cryptography, but it was bypassed by inserting a hook either before or after the encryption stream. Indeed, any general network security, if it can be bypassed, is not going to help you against the attacks that are beginning to come rolling across the internet. We have already talked about how firewalls are getting more permeable, just plonking a firewall down at the edge of your network. People have been doing that for years. It's not helping. I'm told by Wall Street that we lost several billion dollars in lost productivity because of worms this last year. Okay, too many glazed eyeballs. We need to go into more interactive mode. Comments, I'm sure that there are a lot of us who are SIS admins and who are running internet-facing servers. Is any of this convincing? Oh no, we'll address that point in a minute. It's not really depressing. It's depressing if you don't take any action. For people who are trying to follow this at home, when I asked if this sounds convincing, I think the consensus generally was that yes, there's nothing in here that I've stated that is out of line. Comments, questions, anything? What's going on on IRC? Oh, they can't see me. That's too bad. I have a comment. I didn't know that the virus signatures are, you said they're changing now, like changing for each copy. Yeah. So they're unique. I use like clam antivirus scanner and that works pretty good for filtering mail, but if they're, you know, they implement something where it's different every time. That was a pretty darn good solution for the 20th century. But we are, static scanners are not going to cut it for you. Well, we saw the same thing with spam, right? Because people are doing spam checks on this too and spammers immediately overcame those. The virus people are even better at it than spam writers are.  Virus writers don't have that kind of a profit margin. They don't get, because in order to make a living off a botnet, you need to have at least 10,000 machines. So, and then you got to fight for control of your botnet because the other guy who is running his botnet would love to add your zombie machines to his network because the number of machines you control is directly correlates the amount of money you make. $6,000 US tax-free a month for spending five or six hours on the computer. Heck, I spend more than that. I don't even make, well, not tax-free anyway. Coming back and looking at our machines. What are the programs that we should be examining if we are looking to harden our machines? Well, privilege changes. Any program that starts off his route then drops privileges and becomes something else is a potential source of problems. Because the very fact that it is changing privilege, any set UID or set GID program that the user invokes does the set UID or set GID because it has got access or privileges that the user doesn't have. Now you, if a rogue user can subvert this program that's giving me extra privileges, well heck, I've got privileges that I wasn't supposed to have just because I've subverted this program. So any program that changes privileges is something that we need to be looking at. Assumption of atomicity. If there is any time interval between a security check and an action taken based on the security check, you've got a window for opportunity for the attacker. You're removing symbolic links in slash TMP. Check for the file, find that it is pointing okay. Then you go ahead and remove it. That interval that is there is the window that somebody has for retargeting the symbolic link and suddenly you're deleted, et cetera, password.  You get the idea. If there is a window of opportunity between a check and the action taken on a check, you have a problem. Trusting the execution environment. That program that believed that, well, I'm the program, I must have been compiled exactly as I was designed to be. So therefore, no need for me to check any, do any other checking. Because since I'm executing it, I'm good enough. Or, just a corollary to this is, I am a plugin infrastructure program. My plugins are obviously trusted. They are my plugins. Why do I need to check? For any problems with those? Any program that trusts the environment it is running in can be attacked by a hacker, a cracker, beg your pardon. Trusting user input, buffer or flow. I don't think I need to say any more. This is a new fangled, well, for old timers like me, this is a new fangled attack vector. This is mobile code, like you get Java codes, faulted over the network from one machine to the other. Now we are getting little Ruby closures being sent across and executed somewhere else. If you wouldn't trust user input, why would you trust code that's coming from somewhere else? It's handing the keys of the kingdom to, it's easier to exploit, you know, code that you've written than to figure out some input that you feed into the program that will do a buffer overflow. And then you've got to be really careful what you feed after the overflow to make sure that the program execution jumps to the bad code that you have injected. And that means you need to know assembly and you need to know the architecture of the processor and stuff like that. But hey, I can write a Java program that does what I want and I don't even need to know what machine it is running on. Okay, using shared resources by itself is not a security vulnerability. But if any program that uses the shared resources has been compromised and can modify the shared resource, then every other program that uses the shared resource is now exposed. There's, in security, there's something called information flow analysis. So you start with that, this domain, the information started here, it was read by that other program and went over there. Anytime you have information flow, there's potential for compromise of anything downstream. Shared resources are one way of suddenly fanning out all your information flows that went through the shared resource. Already then, we are back to building things, networking. I mean, we've got in the virtual machine that we are going to have out there. We've got to network it so that data can get to it and out of it. Otherwise, it's just as useless as any computer which is not on a network. Oh, I forgot. Any questions over the last bit? Comments, critiques, am I speaking too fast, too slow? Maybe I should be telling jokes. All right. There are two ways of doing networking. One is using something that they call UML net and UML utilities. It assumes that your virtual machine is going to have a fixed IP address. It's easier to set up. You don't have to do anything special because the UML net program goes and discovers all kinds of things for you and automatically sets up things. But it means that every single UML machine you have, you need to give it a separate 192.168 IP address. The other method which I kind of prefer is to set up a DHCP server for all your virtual machines. And that uses the daemon that runs and mediates packages. What you see here on the slide is how you set up UML net. The bottom line is what you give on the command line of the virtual machine when you set it up. If you've got script.sh, all of this should be there. This is setting up the et cetera network interfaces on the host. We set up a tab two interface and we say that it is going to be used by UML net user. The other method, if you prefer to set up a DHCP server, I see people yawning, is do that and restart the DHCP server. You're basically saying that you should listen on the interface tab two and in my paper, you'll see the full DHCP config file. This is the other method using the DHCP server for you have to run the daemon UML utilities that the advantage of the second method is that you can actually set up a virtual hub. So you can have an internal virtual network of four UML machines, all talking amongst themselves and talking out through a gateway, which the UML utilities, it pretends to be a router gateway for the external world for that internal virtual network. I don't think I'm gonna speak much more about this. Everybody knows how to set up network interfaces. Either static or DHCP, five, 11. Okay, we've talked mostly about attacks, attack characteristics, what are the vulnerabilities that we face. Now I'm going to present what I believe is proper security solution ought to have. These are the characteristics that you absolutely need. First is privilege separation. This is so that an attack against one subsystem does not spread like wildfire all over your machine and take over the whole machine. So if somebody compromises my Apache or bind daemon, they should not get root access. This moderates any attack vector that attacks any internet-facing service. Fine-grained access control. We need this as opposed to user group world kind of access control. In order to tightly control the system and yet allow it to still do what we want it to do, nobody wants to run, and hi, am I, a machine that has a Linux on it and which doesn't allow their FTP server to run. One of the examples of fine-grained access control is when I've run my mail user agent, I want it to be able to read and write to my mailboxes. However, I do not want it to have access to my SSH keys. I certainly don't want it sending those SSH keys off to somebody else, even if my mail user program has been replaced by a Trojan. Fine-grained access control allow us to do that. Role-based access control. I wear several hats. There are times when I'm working on my Debian packages. There are times when I work for my day job. I work from home, so it's the same machine. However, when I'm working on my Debian packages, I don't necessarily want the stuff that I'm using to have access to stuff on my day job, some of which is not entirely unclassified. So changing the rights and privileges that a user has based on the role or the function they are performing currently is it reduces the vulnerability, the security vulnerability that you have at any given time. Least privilege. Well, don't give a user rights that he doesn't need. Don't give a role rights that it does not need to perform the function. Add in fine-grained access control and role-based security. Least privilege allows you to lock down the system without compromising what a user needs to do. You won't get very far by preventing users from what they think they need to do. You need to also limit error propagation. Nowadays, when somebody compromises a box, the first thing that they do is they send a, they call home and ask, wait for instructions from the attacker on some website somewhere. They attack the machines nearby on the local net, which may not be accessible directly from the outside. We need to prevent that. We need to prevent an infection of one machine or one subsystem from spreading to other subsystems and other machines. Now, this seems to be kind of laboring the obvious.  There should be any security system we have needs to stop people from arbitrarily becoming rude unless they are authorized to do so. Assurance. There has to be some assurance that the security policy that we have on this machine is complete and correct. This was strange to me before I started delving in hardened systems because back in the day, in Unix, we didn't have security policies. You wanted to log in, you typed in guest, password guest, and you were in. So, but you can have a security policy and you can have an assurance by static analysis of the policy that your policy is complete and it is correct that there are no information flows that you did not want there on your system. They have already talked about protected paths. One of the neat things about AC Linux, this is a digression, but I'm excited about this, is that you can take the concept of a firewall. You can say that, you know, in a firewall you say that, I don't want anybody except that particular I node to be able to talk to my FTP server. AC Linux takes this further. You can say that I want only somebody on that particular class of machines in that security domain running in such and such a security domain itself. I only want to be able, those processes to be able to talk to me. So not only can you say that this machine, you can say only these authorized FTP clients on those machines can talk to me. If somebody has broken into that machine, install their own FTP server break-in vulnerability scanner program, they won't be able to talk to me because they're not on the authorized list. Any system that we have has to protect against zero-day vulnerabilities. We can't rely on us knowing ahead of time what the attackers are doing. We just can't stay ahead of all the... You saw the way the incident report went. It was an exponential curve. How many people work for IT departments or university IT who can say that their department budgets and the people they are hiring is going up exponentially? Heck, we are lucky if we get a total, a one or 2% budget increase in a year in these days. Any security system we have should not be bypassed. I mean, what good is a security system is you can just bypass it. The only way you can prevent something from being bypassed is if you build anything which is built on top of the operating system can be subverted. Something can slip between that security mechanism and the OS and bypass it. Given this, why security enhanced Linux? It provides us mandatory access control. What does mandatory access control do? It prevents security from being bypassed. It also gives us decent privilege separation. It gives you regulated domain transition. I told you about security domains. People have discovered that once you set up a security domain, especially if it is a large enough organization, you can't prevent people from wanting to run Windows. So you say, okay, you guys can run Windows.  What you do is you put an air gap between that network and everything outside. Even if they're running something vulnerable, as long as there is an air gap, nothing comes through. But you still need to get your information out.  So this is where you do domain transition gateways. And this is where I've been trying to put Debian as a ideal platform post-edge for a domain transition gateway. What this does is say, suppose you've got a document which is top secret. You want to bring it out into a merely secret or a public open through the public area so that you can send it off to the Washington Post. The first thing that has got to happen is all the little things that make it top secret have to go away. You've got to redact it. So you set up a scrubbing pipeline. Top secret document comes in, words get deleted, replaced by black squares, and a scrub document comes out. Can't do that if somebody can just take bypass your pipeline and just say document in, document out, just not do the scrubbing. And this is not very interesting to most of us out there, but I find this exciting. Protection from insider attacks. Surprisingly large number of disclosure of confidential data happened with the collusion of an employee. Fine-grained access controls, least privilege protects you from that. It's proactive and fail-safe. It says SA Linux is denying any action by anybody by default, like firewalls. You deny by default. Anything you want anybody to do, you've got to put in a policy rule saying, okay, this guy is allowed to take this action. Just like firewalls which said that, okay, we allow by default and deny some bad guys. Actually, I don't even remember the last time I saw a firewall like that anymore, but they used to be there, I assure you. I deployed one once. Those are unworkable. You've got to have denied by default and you allow only things that you need. SA Linux also allows you to sandbox applications. One application, if it is compromised, doesn't infect the system, cannot crash other applications. You can take SA Linux policy and statically analyze the information flow. You can mathematically prove if Program X is infected, what other programs could possibly have been compromised and may need to have their data looked at and be reinstalled. Okay, if SA Linux is this great, why am I here talking to you about setting up a UML machine? Well, UML and Zen actually in this case, any virtual machine gives you security in their own right. They provide strong isolation. One Zen machine or one UML machine can't talk to any other Zen or UML machine on your system unless it comes out through your host machine and goes through the network stack. This allows you to have one machine where you can have multiple virtual zones with different security, confidentiality, and integrity requirements. It would be nice if we could all deploy every one of these zones in their own big iron Debian running boxes, but not all of us have huge budgets. Efficient use of hardware. I mean, this is not just SA Linux virtual machines, this is any other virtual machine. Just since no virtual machine can communicate directly with each other or even to the host system without coming through the network stack through a pipeline that you control, this way if you set up two different zones on two virtual machines, you've got a verifiable information flow control. You know that this machine couldn't tamper with that other machine unless it came through the host machine, which I control. You can also use these virtual machines to play around with different security policies. Actually, you might have different requirements for different machines. A bind nine machine might, you might want to put a different security policy on it, then you want to put on, say, a secondary bind server, a backup bind server or something. The other thing I liked about UML, and I don't know if this is true for Xen, so somebody please correct me, UML has copy on write. That means you start with a root file system, any changes that you make, go on to the copy on write file and not, your original root file system is not modified. I don't know if you can do a Xen machine that works like that. Oh, so with Union FS, Xen does that too? Well, if you've got a C Linux Xen machine now, with Union FS, you can get the same benefit. So if something is compromised, you can roll back at least the data part to a known good state and then reinstall that particular application that was compromised and you're good to go. So, why else UMLs? Well, as anybody who has tried a C Linux on their machines on the desktop, you can tell you this is complicated. It's a fairly complex system. It's in the same class as setting up a Kerberos real and I personally think it is more complicated than setting up Kerberos. This is not for the faint of heart. It will improve by edge. It will be real good. Policy writing is an arc and art. I've told you all the neat things that policy can do. Not very many people can actually write security policy. Can you see a show of hands about how many people have experienced with or actually have written Mac policies? One, plus maybe some others from IRC, anybody? So the problems are made worse is that there is no really good guidelines for writing policy. This is one of the things that my research work is on trying to figure out how to do security patterns, but there's nothing out there on it as yet. Modifying policy is almost as complex. Suppose you do have, say, policy for Apache or some other server and the service changed. You'd think that, okay, I've already got policy mostly written for the old behavior. It should be easy to do it for the new behavior. Just design by example. Problem is, you can't. Any change that you make might have repercussions that are not immediately obvious. You really have to do a full information flow analysis on the old policy, the new policy, and compare the information flow and see if you have accidentally compromised any information flow paths. Well, not that I always do it, but I'm supposed to. However, secure virtual machines are viable. You don't normally run the same things on a virtual machine that you do on your desktop. For example, I don't have a free WM or Emacs on my virtual machines, which would be unthinkable on any machine that I actually use. I don't even have WIM, the root file system. Most of the rest of the stuff was really easy. There's a webpage out there that tells you how to network your virtual machines. And there are all kinds of people who have already created virtual machines and how to network them. Creating root file systems is slightly more complex. So I've created a script called createsh, createfs.sh. It is all found on Homer, and I'll give it a permanent, well, there is a permanent home for it. I will send out an email for people following at home. And if you have access to that script, it doesn't need any of the kernels that you have compiled or any of the networking things, you can just run it. It points to the local mirror of the repo out here at DevConfix and should take maybe 15, 20 minutes to run. If you had compiled your UML with modules, which I didn't, so if you're using my UML config, you can skip over this part. This is the point where you can install your kernel image modules inside your UML. And here is the big complicated path. I have set up a shell alias. I strongly recommend people who are using this to set up shell aliases. This is how you run the UML. There is only one slight modification. A couple of little modifications were regular UML, the regular way that you run UML and actually, what I've done is I've taken the serial console, shoved it out to file descriptor three, and I'm logging the serial console onto a file so that even if an attacker attacks my UML machine, they can't erase all logs because the logs are not there on their machine. I do have three different examples of attacks and how SC Linux can be used to moderate the effects of those attacks. Where it's 5.30, I still have 20 minutes. We can use that for a questionnaire answer session. Or I found that going through the remaining slides that I have takes about six minutes. So we can stop here, break off early and go for BL or? I'd like to see an example of one of the security policies if that's possible. Sure, can you do it offline because it's not going to fit on a slide. Oh, okay. And I have an example sitting here on my laptop. I wish that I had, we could modify. Well, I have got it in my people now. I have been modifying my people after I submitted it for DEBCON. I'll make it available to people as soon as I get back home and polish it up and add a few more graphics. Just grab me after this. Sure. One question, you said several times that maybe SC Linux will be prepared for edge. I just wanted to know how many people right now dev and developers are working with SC Linux? Like maybe either providing the software or writing policies for base packages and such? I would say offhand, I know of, well, there's Russell Cooker, there's Eric, then there is, oh, Greg Norris, and maybe one other person who, I can't recall exactly who. So with myself, I know of three other people, so there's four, and we would love to have other people work on this, but what we have been trying to do in the meantime is get these SC Linux packages in SID so that it is easier for people who want to play around with it to actually do so. Once we get the tool chain unbroken and have it possible for people to actually use SC Linux, maybe in a virtual machine, I'll send an email to devilannounce. We are close, and I hope to be there all the way by the end of June. Is there any utility like in C-Straice to find out which calls are made, which files are read and written that we can use to start a basic SC Linux policy? Yes, there is the audit subsystem is now fully SC Linux enabled once you get applied the patches. Every action that is denied shows up in the log. Every action that is permitted may also optionally show up in the log if you tell it to. Usually that means you've got several hundred megabytes worth of logs if you'd go out, highest log level, but you can filter it down. There are also tools that can take the errors that happen because SC Linux blocks something, and there is a highly deprecated utility that will convert those errors into security policy allowing it. I don't recommend it because there are times when you really don't want to allow that thing. Maybe it's a bug, maybe it was an attack, and allowing somebody to automatically put in allow rules just like in a learning firewall, you check your rules before you enable them in your firewall. Do you think that the SC Linux policies are going to have an effect on the design of executables? So as to things that do things monolithically at the moment may split up into multiple executables, the bits that do things that are secure and the bits that do things that general users want to do. Well, one thing that springs to mind is GBG, for example, can display your secret key, but it's really useful for decrypting things, yeah. Well, I hope so, but this is a chicken and egg problem at the moment. People writing applications do not want to go through the hassle of writing to SC Linux until there is a certain level of deployment. And we are not quite there yet, and partially because people say that none of my applications have been changed, so do I really get that much benefit out of SC Linux? One of the things that has changed in the last three months is now we are using the reference policy, which is modular. That means I can energize Apache shipping with a Apache package policy, and then part of the post-tint can be, here is Apache package policy loaded into the currently running policy and go on from there. But first, we've got to get SC Linux into the default system, working correctly, and with a policy that our users can live with. The way that Red Hat in Fedora has gone about it is, they've created something that they call a targeted policy. That means most of the users are not impacted, their actions are not checked by security policy. You just target a few servers. If you're running Apache, anything which is facing the internet can be targeted to be locked in a sandbox. It's not perfect hardening, but at least you have locked down the applications that face the internet. Because Fedora, and Suze, and Gentoo, and hopefully Debian, are coming on board with SC Linux, the impact on the application writers would hopefully be positive in the next year or so. Do you have a set of rules that could be used for somebody who has very little working knowledge about the computer at all? That is, a set of rules that will make it easy for them to just use the computer, set the date, and do normal things, but all the protection goes on and hides in the background. Security often doesn't come with no cost. We can get you part of the way there by giving something like the targeted policy. The targeted policy will not impact on users. If you do run some servers, they'll come with full security. If you want strict policy and the full security to go in the background, I think we need about a year. Right now, this is basically just, we haven't even gotten a working version in Debian. I'm trying to make it so that when we release Edge, you should be able to at least opt for a targeted policy by default. Most likely, we will ship Edge with a grub configuration such that SC Linux is present, but turned off. You'll have to change the boot options in grub.conf. You have to turn SC Linux is equal to zero into SC Linux is equal to one. And then it's up to you whether you want it in forcing, whether you just want to be warned when somebody is doing something you don't want them to do, or if your policy is not correct, and then you can, once you're sure about your policy and you know that nobody is going to be impacted and everybody who is likely to scream at you has already been taken care of, concrete boots or otherwise, then you can make the, go into grub again and make it in forcing. Really quick question, kind of piggybacking on the previous one, if I may.  I mean, as you said for Edge, we're probably going to have it disabled by default, even though it will be trivially enableable. But one of the problems, as you mentioned before, is that it's extremely difficult to write policy statements. And so what needs to be changed in order for us to deploy this, such that users who admittedly, from my interaction with them, have a rather difficult time, even doing basic things in Debian, are going to be able to set up their system and keep it working. There's a lot of work going on in the reference policy. Most of the people who have complained about this have been running Fedora, and Fedora was running the old NSA, SC Linux policy. The new reference policy is far more modular. It's not one giant spaghetti mess with everything, depending on everything else. Modular policy makes it easier to write policy. And there are about a dozen people working on reference policy. And we are looking for volunteers. I've just become a upstream developer for reference policy just last week. And any help that we can get. So to answer your question, what we really need is people who will help us write policy and get us to a stage where every package that needs something special has a little policy module for itself and which gets installed in the postage. We can't do it in a central policy, just like Fedora does. We have far more packages than Fedora does. We only have four people working on IC Linux now. We'll burn out if we have to write policy for 15,000 packages. So if we can pump off and dump writing policy over to the people who maintain the packages, and that will entail somebody, hopefully not me, writing up documentation on how to do policy right. Well, I figured that the people who are writing the O'Reilly books and getting paid big bucks for it ought to be doing this. One question, and you almost answered that already, but is your vision then that package maintainers write the policies for the packages that they distribute? Right, so how are we gonna make that when package maintainers, a lot of them at least, provide some diamonds running as route with full privileges? So without even thinking the sign of the package itself, how are we gonna make them write security policies for those kind of packages? Well, hopefully we seed security policy with policies written by people who have experienced writing security policies. For the most part, once you've got a package that works a certain way, the security aspects of its behavior do not change. Apache, for example, has been doing the same thing all the time.  It reads configuration files from a certain directory. It writes logs to a certain directory, and this basic behavior has not actually really changed. What has changed is what's in the configuration files and how things are deployed, but as far as file accesses go, and the security aspects of it, we haven't changed anything much. So I'm hoping that the security policies will not have to be changed by the package maintainer, and when they do have to be changed, and if they are unsure, they come to somebody who can do it. Or if you can get, I'm hoping for some, even though I'm against helper packages fundamentally, I'm hoping that we might be able to, with experience, be able to create a debhelper utility that you just describe what your security needs are. You describe what kind of files are log files, which kind of files are configuration files. These are the files I need to read, and somebody comes up and spits out a slnx policy for you. There's some people in MITRE who are already working. Paul Jain, I think we already have a version of Paul Jain in Debian, APOL, and there are a bunch of tools which are not quite there yet, but in a year or so, and maybe if we get lucky, even before we release Edge, we might have something. I'm not expecting us to have distributed policy in packages by the time we release Edge. For that, I'm hoping for Edge plus one. Are you looking at sort of, this might be from ignorance, but are you looking at sort of run parts type system for the policies, or is it more of like a licensed thing where it's user share, blog, traffic, name, license? Conceptually it is run parts. Conceptually it is run parts. It's a little bit more complicated than that. You don't want anybody to, you need to have proper authorization to run, to create the new policy. There is a tool called SCModule, which handles how module, it's kind of like the ModPro equivalent. Think about policy modules as kernel modules. Just like ModPro, there is a tool that allows you to load new policy packages and compile them and get them running in the current policy. The usual security rule supply. You don't want just about anybody to run that. You need to be sure that the policy package that you have, the module that's being loaded is in a trusted path. You know that it's being properly labeled. And, but yes, it is close to use, but that's why a list of policy modules, there's some checking done to make sure that they are, you know, have the proper penguin pee on them and then they are loaded up into the running policy. Time's up guys. Since there is no talk after this, I will contravene the little rule about everybody disappearing. If people want to hang around, I'll answer questions. If people want examples of attacks and defenses, if you've got 10 minutes. We will continue after this. Okay, thank you very much. This is the last talk in the tower. Everything else is in the hack lab. I haven't got the titles or times to hand just now, but I believe there's one starting at six o'clock. So thank you very much.


Welcome everyone. It's my pleasure to have today for this Integrative Research Seminar, Emilia Gomez. Emilia Gomez is an associate professor in the Department of Information and Communication Technologies. She's actually a Serra Hunter fellow. She got an engineering degree in signal processing from the University of Staviglia. Okay, that was some time ago. Then she got a PhD at our department and she has done research stays in all these years in a number of places including Stockholm, Sweden, McGill University, and the Center for Digital Music at Queen Mary University of London. So, without further ado, here, Emilia. Okay, thank you very much for the introduction and thank you very much for being here. I'm very happy to talk about my research with deals with music information retrieval. In this talk, we talk about challenges and opportunities of this field that was on classical music. First, I would like to convince you all of doing research on music or at least playing an instrument. Music has been found to be very positive for cognitive stimulation. Not only for learning but music is also very important or has important links with memory and emotion. For instance, memories which are attached to music can be retrieved faster and have a greater influence in our mood. Music in general has a strong influence in all areas of our lives, such as our social relationships or relationship with peers and also our culture. But my research is not only about music but about music data. One can say that there is one song for every heart, so there is as many songs as people in the world. In fact, only in online music platforms we can find a huge number of songs. For instance, in iTunes or Spotify we find up to 43 million songs which are listened by 100 million users. Not for each of these songs we can also find for instance that people do versions or play them in different social networks. For instance, in YouTube, if we try to find versions of Mahler's Symphony No. 4 as you see on the figure, we can find a lot of versions of this song. Not only that, one single version can be made of a large amount of data. For instance, if we have a recording of Mahler's Symphony No. 4, we can find not only audio recordings but we can find data related to video recordings. We can also find score information. We can find textual information which is related to the composer, to the piece, to the performers. So there is a large amount of data that is associated even to a single recording. So this is the kind of thing we deal with large scale music collections, not only audio but anything. So music information retrieval tries to find things in these 43 millions of songs. Try to get technologies that can help you to retrieve music. And in the same time, we would like to understand how a person will describe each of those pieces of music and how can we emulate those descriptions by computational models to do a large scale analysis. This opens up a very huge amount of applications and also research areas because you can do research in quantitative data, in large scale data. So this is the methodology we usually follow in our community. First, we try to talk to humans and model the way they would perceive music or the way they would describe it and depending on the application we will find some descriptors. So we integrate knowledge from human perception and cognition. Second thing we do is as we process data, we process for music signals, then we have to deal with acoustics and signal processes. So some of our research is related to this area too. In fact we are dealing with audio or with music so we have to incorporate knowledge about music theory. For instance, music intervals, how they are built, how people use consonants and dissonants or build scales and so on. And finally as we deal with large amount of information, we have to apply methods and also contribute to improve algorithms for machine learning or for pattern recognition. Our focus, and particularly at the music technology, is to build real applications for real people. So we are also concerned with very practical aspects of research such as if this can work in real world scenarios, if you can check about scalability, user-centered design and so on. Because we have discovered that maybe some algorithms are very good for a particular application but totally useless for another application. So the music information retrieval area is not a community of people that meet around a conference which is called the ISME conference and there is a society which now I am president-elect and it has organized a conference in different places. This year it's in New York and the idea is that there are people from different areas from musicology, from computer science, from digital libraries and so on. And it's also very well connected to the industry because many companies are building applications related to our research. So the evolution of the field and first from talking about a particular application can be summarized here. People started to describe musical scores because this was the only information we had at the beginning and then musical scores are usually found for classical music. Later on people started to describe not only scores but audio and then you can have audio signals from different styles and you can have large scale analysis. After people tried to describe not only music content but also context information for instance people doing a lot of research on web crawling, on incorporating Twitter or other social or other information which is on the web images from album covers for instance to describe music. But recently people are working with multimodal data with many different sources. They are trying to do deep learning or this kind of huge machine learning techniques in contrast with the featured engineering that we call trying to get good features. And then people are moving from system-centric to user-centered designs so incorporating also personalization for instance into the models that we built. In this talk I wanted to focus on a particular repertoire because I think it's easier to explain the challenges than with general problem which is this scenario. Imagine you go to a classical music concert you will see this orchestra. I don't know if you go very often to this kind of concert this audience is kind of old. You see there is not many young people and then you see there is a conductor like hundreds of musicians playing along with the conductor. So our project was trying to ask ourselves which are the main challenges that people find when they go to this kind of concerts where they go to a piece of music they have never listened to and they want to discover. And if our technologies are ready or can help to provide meaningful or to enrich this information help people better or in a personal way enjoy music concerts. So this is an European project I have been coordinating over the last three years here I will focus on the work that our group has been doing and it incorporates partners from music for instance we have one of the best orchestras in the world which is the Royal Concertgebouwork Orchestra in Amsterdam we have also a conservatory from here that we were very close in this project we have also some technological partners from Austria mainly and we are having so much fun that even if it's finishing or it was finished we continue we have some spin-off projects from there. So the idea we have is to transform a music concert into what we call a multi-model, multi-layer and multi-perspective digital artefact. What does it mean multi-layer? That you can see the concert in different modalities you can check video files, you can look for audio files or text what does multi-layer mean? That you can describe different things within the concert you can describe the structure you can define the look of the musicians and so on and finally multi-perspective is that we want to access this content by different ways for instance by different locations of the auditorium or by different perspectives for instance the perspective of a naive user or the perspective of a musician. So then we have worked in trying to provide facilities to explore, enjoy and share the concert and we provide technologies that can be used before during and after the concert of course in a tablet or in a computer. So in particular we focus on this scenario which is virtual concert guide, overseeing the music focusing attention and switching viewpoints and joining the orchestra and I will tell you a little bit what have been done in this respect. Of course when we address this repertoire in the community it's very challenging because we usually work with songs and songs are very short even the most research is done in thirty-first seconds of songs so not even the whole song and here we deal with pieces which are one hour of duration in addition we have pieces which are very complex it's not the same to analyze maybe pop songs which is built by three, four chords then maybe this kind of music which is very complex and we have a lot of data associated and acoustically it is also very complex this is an image of a prototype I will show later on different instruments playing that so there are sections, there are also different timbres that we have to analyze and of course we have different modalities so this is the repertoire I won't get into details but just to have an idea of the kind of data we have we have different cameras, different microphones we have many information and maybe depending on the quality and also on the information you have you have maybe around 100 gigabytes for one of these concerts of course if you compare with different performances of the same piece then you will have it multiplied by the number of performances one thing I wanted to mention is that working with classical music has an advantage that in fact you have the score of the music this is not happening for instance in pop music where you don't have any score but the musical score is available but still not at all fully digital so we have problems in that so the first thing or the first thing we did is to try to develop a tool that you can have all this information accessible for building applications for learning training models and so on and we contributed to our repository which is developed at the MTG which is called Revobis which is a web browser where you can store information, multimodal information about the classical music concert and I will now get some time to play you some music and then you can maybe check or visualize different information that we do just have in mind that this is all in a browser so then it can be integrated in many applications using API you can always build applications on top of that so it's a web system and then I will play you one of the pieces we have worked in the project so this makes you maybe realize all the information we can get from the same concert and then different ways of exploring this information of course this was all the visualization and so forth for the framework but not all of them are used in the real application now I wanted to comment to you about five research topics we have been dealing with because our idea then is to provide facilities to describe this content, interact and we have been mainly focusing on these different topics and I will very briefly talk to you a little bit about that the first one is melody you know melody is the thing if you listen to a song or to music and you want to remember or imitate you will imitate or you will remember the melody it's the things we can easily remember better than for instance the instrumentation so the melody extraction methods try to simplify music into a contour this is if you see now there is this song twinkle twinkle little star and if you sing it you will see this is a melodic contour so we try to go from a very complex piece which is the symphonic music to this simple melodic contour with a difference that we don't have the score so we have only an audio signal so why we want to do that because for naive users it is more intuitive to look at the melody than look at the score so if you want to retrieve music you can sing it and then you can compare it with different songs or different parts and then you can find the part you want to make of course melody is related to pitch which is a perceptor attribute of people where we can differentiate sounds and it's linked to to fundamental frequency which is the signal property we want to measure which is related to periodicity in the time domain signal or in the frequency domain of course this is an example for a very simple signal the signal we have in orchestra is very much more complex so which is the goal of our research the goal of this research has been to extract from audio signal just the melody of course and this is the currently the piece the work by Juan Jovoz which is here in the audience so of course if you get popular music or songs for instance this song this is there was something inside of you something I thought that I would never find Angel okay so extracting the melody is kind of difficult in fact from the signal but from a perceptor point of view it's easier and this would be the melody sorry what happens? so the task of melody structure is equivalent to the task of describing the voice and then the pitch but if I play you an excerpt of classical music try please to sing along with it and define which will be the melody let me okay so you see it's more complex there is in fact there is a very challenging to describe melody in this type of material because there are many overlapping sources so the signal is very complex in addition there is a melody is played by different instruments sometimes they play together in unison that means the same note but sometimes they play different notes still you perceive the same melody so we have been working on trying to extract melody from this kind of data those algorithms for those of you that who do not know are based on three different principles and here I put a diagram of a method that Justin Salomon and I developed in 2012 which is mostly thought for singing but which incorporate the three principles that I was mentioning the first one is saliency so we might say that the melody source is more salient in the spectrum or in the signal than other sources so we try to estimate saliency in the time domain or frequency time representation of the signal this is a spectrum and then try to estimate which source is more salient I won't get into the details of the algorithm the second principle is to incorporate the continuity rule melodies are usually pitch values which are continuous you cannot have a jam of one octave from one millisecond to the next or 50 milliseconds so we use continuity rules this is also ideas from computational auditory analysis area where you try to incorporate those ideas to track melodies and finally we applied musical knowledge why because melodies maybe have some properties for instance they might belong to a scale they might be related also with other instruments etc so this is state of the art algorithm that we use and then from that we try to apply it to orchestra of course the first thing we do when this is the usual methodology we got many musical excerpts from different pieces of orchestra music you see here that this is a statistics of the melody play of the instrument playing the melody sometimes, oops, sorry sometimes there is a string instrument that play the melody brass or woodwinds but many of the times there is an alternation or a combination of instruments from that we try to annotate which are the melody or which is the melody and for that we ask people to sing along with the music so we ask people to listen to the music to listen several times to memorize or try to to learn a little bit and sing along and then we analyze aspects such as their perception, their agreement how this is related to different musical properties for instance if the melody is very complex it's more difficult to sing how this relates also to to our capability of singing of course because maybe we cannot sing a certain intro of us so we have analyzed that and I will show you just an example and from that after measuring the agreement we have a collection with annotated melody that we can use to compare to our computational models I will just show you an example of the melody you heard before let me play a little bit so that you'll remember and then how four different people sung this melody I guess some of you might have taken part in this experiment music ok so you see people do it pretty well they have some problems but they mostly agree on that but if you look for for algorithms they do much worse music so I guess you would agree that you cannot use that for any application so we have been working on that on improving that kind of algorithms we have also published this data set for people to evaluate their methods on our particular material and we have managed to get good results using algorithms maybe without getting into details more general not only focusing on singing which have also can characterize not only one melody line but several melodic lines and also that can be also using the score if we have it available and of course there is a competition in our area there is an annual competition where you submit your method and it's evaluated against others and we have very good performance not only for symphonic but also for other genres like we managed to also contribute to the field and we have developed also a prototype for visualizing melody let me show you where is it don't see my screen now ok but I will then skip this so you can maybe if you have time later you can connect it and then you can just visualize the melody while you are listening to the music ok so the second topic I wanted to mention is about structure musical structure is very interesting topic of research especially in this kind of repertoire why because you have pieces which are one hour long and then you want to know how it's organized for instance some people might want to know how to they have to applaud or when is the solo is going to play or when are the important themes which appear so musical structure is a very nice topic of research structure is related to repetition and also to changes so we use principles from pattern recognition we look for patterns in the music and we also use principle for continuity and novelty detection in the music so in fact experts also do structural analysis so when they analyze music they tend to structure into segments this kind of analysis and they name it for instance a or like exposition development and so on so structure in symphonic music is built by two different resources composers first use tonality for those of you who know music tonality is the way we organize pitch information like note information in fact tonality is used to to convey tension so when you have something which is in the key you are more relaxed and when you have something which is not in the key you are more stressed so people and composers use that to communicate the structure and of course in symphonic music you have a lot of changes in tonality in key which are called modulations for instance this is the typical structure of the sonata form which is used in many symphonies but not really as it is but more developed way where you always have an exposition which is in the tonic, a development which is in the fifth and then recapitulation which is in the tonic the other resource that composers use to communicate the structure is instrumentation so it's how you make instruments play of the instruments of the orchestra play and which notes do they play because by making instruments play different notes you can maybe get some dissonance effect you can get some effect also of very well known effects for instance the contrast between section or what we call the orchestral crescendo that there's always a change in number of instruments and orchestration so these are the two aspects that we try to model computationally and in order to evaluate of course we need to evaluate data and annotate it data so instead of asking people naive listeners to annotate one hour recording of a symphonic piece we got experts because you know many experts write about music they write about the structure of pieces and we got this piece which is the Oroica which is the third symphony of Beethoven so many people have written about it so we got like eight are the papers that have provided an structural analysis of this piece and we also gather notes from ten symphony orchestras the program notes you know the oops it was not my slides maybe it's and to find out which kind of information when you work with musicians or with musicologists they do not or it's usual that they do not agree for instance we tried to see and there was a lack of discrepancies so music analysis is also very personal but in order to arrive to some ground truth annotation we put a threshold of if three people agree there is a boundary and then we only with that we found 16 segments of boundaries in the exposition of the Oroica symphony so that means a lot of segments you have to detect we put those segments into this is the red lines you see here and then we use some of our computational methods for instance one the script we use is tonality we estimate it directly from the audio or from the MIDI by using information from cognition like how the tonality is defined and then we do it in a multi-scale way so this is kind of similar to wavelets where you have a global descriptors and then you have local descriptors for instance this will be the global key green maybe it's I don't remember the key but it's one particular key and then local key and then you can find segments of course there are segments that were found by using key information other ones were not found and in addition we also did some analysis of instrumentation orchestration and we found that that all relevant structural boundaries correspond to changes in instrumentation for instance here you see we have a simple representation of instrument and when they are active in the piece and we can then identify if they are playing or not playing we can identify in those segments so with all those symphonic pieces are very complex and additionally we can model the structure and we can represent it to people so that they have a map to follow when they listen to a musical piece of course there is a difficulty of large-scale evaluations so if you want to do evaluation over a lot of symphonies that's very hard because analysts will not agree they have different way of annotating and so on and also there are some features that have to be personalized for instance people that know music pay attention to something more related to music theory and people that do not know music maybe have attention to more instruments that are playing the three topic I wanted to talk to you about is instrument emphasis and this is the idea that if you listen to an orchestra there are 100 instruments you listen to the whole thing but we want to provide people the possibility of listening to particular instruments so for instance I want to listen to the cello or I really like the oboe or I don't really know how clarinet sounds so we would like to get people accessing the different instruments of the orchestra this is what we call instrument emphasis like I want to emphasize this instrument I want to listen louder so we have been working also on trying to from the acoustic signal trying to separate the different instruments after this separation trying to emphasize and trying to locate them on the scenario so that when we locate the instrument we separate them we can simulate different locations of the whole we can simulate that you are moving that you are going through different places in the auditorium those approaches usually are based on knowing at each moment which notes are playing each particular instrument so we need to align score and acoustic information for that and this is still a huge topic of research what we do in our approach is that we try and this is the word why Julio Carabias was also visiting professor here is as we know the number of channels and we sorry a number of instruments and we do the number of channels we try to estimate in which channel instrument is more predominant and from that we try to separate for this particular channel of course we have a multi-channel recording of the orchestra but the microphones are not located close to the specific instruments they are just on the top so in an array of microphones but they are not particularly targeting any instrument so this is the approach that we use for for source separation which is called isolating the instrument is based on non-negative factorization I guess many people use it for other purposes but we use it for that and we try to separate different channels I will get and we train with isolated samples of musical instruments to learn about the timbre to be able to separate on the mix of course it's very important to align very well the score with the audio because otherwise the results are very poor in terms of quality I won't play this example because it's very bad but the algorithms that do score alignment are still not very time precise so they have maybe one bit or two bits or have a bit of accuracy so we have also been trying to synchronize audio with the score and we have been done with this using some techniques from basic techniques from image processes so we try to look at the spectrum as an image and also not at the spectrum but at the gains for NMF for non-negative factorization you will try to really locate the starting of each note I will just show you now an example of instrument emphasis one minute of one piece so in principle you will hear one minute and then you will be able to hear this but play it only by separated instruments we have seen the quality is not the best one but if you mix it with other instruments for instance here you can emphasize the instrument and you can listen louder so you don't have to separate but just emphasize this is also a prototype we have on the web you can also check it where you can select the different instruments and you can listen to the different instruments of the orchestra and we can do things like for instance this video this was an engineering project where he created a way to mix up the different instruments to maybe visualize as you were in different position of the whole I will play a little bit of that ok we are not very good in visual at least you get the idea so now we are working in way to provide immersive experiences in classical music concert using this kind of technology of course we have been working with this material we were very proud of our results and musicians say this is very bad you cannot have it this is total so we still need to work on the quality of the separation and also we take advantage of multi-channel recordings but sometimes those are not very useful if they are not well placed so we will also to work which is the most optimal configuration for recording that the fourth thing I wanted to talk is about gesture modeling we have been all working also on trying to model the gesture of the musicians this is also a joint work with the smoke, with the conservatory where we have access to conductors and musicians and we have been trying to model the conductor because he is the boss of the orchestra so he is the one that is conveying with the gestures all the information to the musicians so we developed some tools to technology in real time to get gesture or mock up instructions for instance velocity or acceleration of movements of the conductor using Kinect and then also you can store all this information in ReboVis and access it later and for that we have been doing some things for instance we have been trying to observe how people move with the music, how people will try to conduct an orchestra and we have seen some strategies or common strategies of people that try to communicate different tempo or different intensity with different gestures and this was so maybe some of you took part in this study we have also been working on modeling expression for instance trying to model each person way of conveying for instance articulation like legato, staccato or conveying for instance different gestures and trying to use machine learning for predicting it so that you can control a synthesizer that will do sound that will be more legato or staccato depending on your particular articulation this will be presented in CHI also and you can see the video I don't have much time to go into that and now we are building a game which is called Becoming a Maestro which you may have played that try to do gamification of this trying to make people aware of which are the challenges when you have to direct a classical orchestra for instance giving the entrance to all the instruments or making sure they don't deviate from the tempo and we are building that now and we will have an evaluation large scale evaluation in the next future finally I wanted to to end up with the topic of music visualization during the last years we have worked a lot on extracting information from music so we can have information about the melody, the instruments playing, the gesture or the harmony, the structure but how do we present that to users so we have been interested also in visualization how do we visualize this information of course there is not much research on that so people just do research on how well do they describe music but not how well do they represent if it's and we have been collaborating also with some people that do analysis on users and so on and we also see that music is very challenging for visualization because there is some visualization that might be more suitable for certain people for certain user profile and also some visualization might be more useful when you are in the concert or after so this is also something we have studied I just wanted to show you a video of a concert that we did we have done several concerts in the project this one that I will show was in an event which is called Singularity Summit which is an event for people that do with technology and also there was a lot of young people so we wanted to make some educational educational concert to show different properties of the music to the people that were there and all the descriptors we strike we try to present it in a way people might understand so we got the piece and we added it into different segments and then we try to provide visualization of the score and the melodic lines the key or the tonality the instruments playing and the conducting gestures so all the aspects I have presented before and let me show you a video of that so that you can have an idea of the results thought about recording so the quality of the video is someone that took his camera but we had a very nice experience and also we are evaluating we evaluated with a group of users about their impression and we saw that for instance there is different needs of experts and knife users of the kind of visualizations they want to make there is the need to make the people want to have control over these visualizations so in a public space with a screen you cannot have control but maybe with computer you can select also there are some musicians that say it may be relevant as a learning tool to visualize music but of course people that like music they prefer to listen not to see anything so it might be even worse for these people and also we find that very interesting ideas on the need for a compromise between surprise and overview of what is coming and also between attracting attention towards over stimulation and this has been done in collaboration with the technical University of Delve which is one of the partners of the project so just to conclude I want to say that we have experience that state of the art algorithms are good for that of course there are errors we have to improve a lot our research method and we still have to understand many things about us as humans we want to understand music and how and also now we are seeing also that there is limitation of the state of the art algorithms for instance with this kind of material and also on a real concert because some things work but when they have to do in real time it is more difficult for instance the score alignment or the visualization of instrumentation you need some latency and it can be very bad for visualization also we have had the opportunity in this project that we didn't have until now to work on user center paradise because we have been we have been a lot working on the method the algorithm, the signal processing part and the classification part but not on the real user that how are you going to present that to the user and how is quality compared to accuracy so we have learned a lot from that and now we are working also to extend these ideas into multimodal description for instance together with Gloria Aro we are working and Olga she is working on the description of music videos also to to integrate visual cues into our descriptors of melody structure and so on and also we would like to continue on that project for enlarging our knowledge for instance to other setups I wanted just to finish with recognition to all the people that have worked on that these are photos we took that we need a new one because there are now some people more and also I would like to emphasize that we are opening our data and our technologies to the public domain and this is very nice because you get much feedback from the way people use that this may be very different to the one you thought and for instance this is from the melody instruction software so we want to also to keep that so if you are interested of some of these please let me know and then we can find out we are open to to provide all the data all the algorithms and so on so just to finish I wanted to invite you to our event that we will have at the end of March at our Santa Monica Ramblas and we will have a concert and an event which is called classical music in the 21st century so I hope you will have time to come and try out becoming a maestro game see a piano with some visualizations of in real time and just try to try out our prototype that we will also have there so you are all welcome to come thank you very much any question yes Emilia thank you for your talk very interesting one important part of the concert is the public so in your project do you take any consideration do you take any features any analysis of the public how they react to the music when they applause whether the characteristics of the public makes any difference they are all they are young and I haven't presented that because it's not really our topic but in the project we have been analyzing the audience there are some people that have for instance physiological sensors in other projects but in Phoenix we have mainly worked with in fact with evaluation and user behavioral experiment of users but the problem is that in classical music people applaud when they have to applaud so there is not such a thing for instance in other styles maybe analyzing the audio or the video you can get a lot of information but in classical music it's more difficult another thing we have done is to analyze for instance twitter and also how people tweet but of course in classical music you cannot bring your phone so this was the limitation of this particular setup in pop music and people we can ask people but for instance we had one concert and we asked people to record and then send the video so then we could analyze which are the moments they record which are the most we asked people to tweet and then we can analyze data so these are the approaches we are following like analyzing maybe audio information during the concert twitter or also from their videos or the photos they make during the concert thank you for the nice overview you start by or you say that you start by analyzing recordings and score and information that you have and try to extract information from it but actually can't you also extract information from the creation process itself for example I guess most composers use some kind of method in order to create music it's probably not a random process so using that information should also enable the analysis later on yes in this particular project as the composers are not available but we have used for instance information on the edition process for instance if the editors that are creating the final recording they use or the final video for instance the edited video they use information about different cameras for instance the way they do it is useful for learning about that or if we can for instance store multi-track recordings and also see how the producer created the mix we can learn a lot also from that in the composition process there are some initiatives that people can really document the creative process but still there are not much documentation but you are right that if we had a way to make it available it would be much easier and for instance some now there are some initiatives to make the data open when you are creating a mix so you will have all raw material to work with because then dealing with the stereo recording is very difficult but if you had all the information you could maybe do more things so yes maybe another question on not the formal analysis but sometimes you may know when a particular piece was created and the contextual factors and also the music was dedicated to someone the music came after a particular event in the life of the composer so do you use that in order to enhance for example the experience for instance I didn't focus but there is one use case which is digital program notes and there what we do we crawl the web with the composer, the piece and we look for music specific websites and then in order to build the program notes we use this information and we personalize for instance if we for we have three levels of musical expertise and then we can maybe for a music expert we provide more musical information maybe for a naive listener we also provide some historical context so yes we do we integrate that it's very difficult to personalize in a single person kind of thing but we initially try to use profiling personal profiling but now we have three levels of expertise Marcelo so yeah thank you for a wonderful talk so I have this this question personally I think a good part of the experience of being at a concert being able to hear the music coming from all around you as supposed to listen at home even if it's a good recording where you're hearing just stereo so the records you listen to at home have been recorded at the live concert say and but there has been placement of microphones and I'm assuming there's been some mixing going on so my question is have you thought or is it already been done or whatever using this kind of analysis of the performance to optimize the both the placement of the microphones and the mixing so that the experience at home is as close as possible to the experience at the concert yeah there's some research on automatic mixing then try to get the best mixing for each kind of setup so there is there are practical reasons also in the orchestras for not changing anything that you have so we have not experimented with that in the project for instance they had their own way to record and that's it you know and it sounds very good so but yes that you can estimate for instance the way they from the mix and the different channels then you can do the link is the same with the video you can do the link between fixed cameras and then the addition then you can know what will be better for each platform or for each setup of or here into that and with this virtual reality idea we are working with create experiences that you can be more immersive into the concert than stereo recordings okay thank you for the nice talk I have just a little suggestion is have you ever considered to look at ballet choreographers or ballet dancers to grasp in any sense how can you transform or how can you convert the music the acoustic music to the visual domain so maybe the gestures they use the things the moments that they emphasize the way they dance the music visualizes for ballet choreographers yeah that's interesting idea I have not addressed that but yes it will be a nice project to do we have been doing something on opera about also the movement of the same performer when they perform but not of dancing research but not we have done that thank you before we leave the faculty lunch has moved okay it's going to be in the tanker building ground floor room 22 okay the sector room or something like that so be there thanks again okay thank you