You probably know Julian Assange as well as his colleagues and friends and associates were extensively surveilled during his time at the embassy and our next speaker Andy is going to talk about procedural and technical details of that surveillance, which is what we are really interested in. That's why that was full, so Andy, technical aspects of the surveillance in and around the Ecuadorian embassy in London is giving a little round of applause to Andy. Well, let's start with the technical aspects of the surveillance in and around the embassy of Ecuadorian in London, presented by Andy. Okay, good morning everybody. Hello, actually the second time, well, this is going to be the second time I'm going to be here. The first one was the first day, but well, the video, I would say CAA, but the video failed, I think it was the CAA, but I don't think it's being paranoid, well, I'm going to repeat the talk now, or we have a problem with the, oh shit, we have a problem with the slide, we're seeing how we fix it, let's try it again, now it works. Well, the presentation is going to cover some, but no, this presentation is going to cover some, but no, but not all aspects of the surveillance. The first stuff that happened on the wireless network, clearly happened on the network, there were more things that made us think they might be, there were a lot more things going through the embassy, we thought something was going through the, with Meta Waves, with Onda Meta, in the embassy, and that gave us to know that there were a lot more other things going through, but I'm going to try to explain everything with the most detail that I can. The second thing I have to say, what I'm presenting to you is what Undercover Global has done, they are being judged, the owner of the company has received instructions from other parties, since it has been modifying protocols and other things, protocols and other information technology, well, and other of the causes that you have, which has been prosecuted, it is money laundering and bribery, and well, for having, for having been spying on people at the time of the presentation, at the moment of this presentation, this is not, he has not been judged yet, and the material that I am presenting is surveillance through the United States and Great Britain. Well, this is the structure of the presentation, I thought I wanted to give you a structure of the presentation in general, a general idea, well, the operation was called Hotel Operation, that was the official name of the operation, and so to be able to be in charge of the people who arrived, they had their first terminology, and most of this was instructed by the intelligence of the United States, what is what was instructed by them, what kind of data were collected by the embassy, and for example, there were introduced to update, for example, the type of furniture that was in the embassy to include more types of surveillance, what Julian did to be against, to be against the surveillance is what the basic idea is that Julian took the idea that any hacker should be free, and in this case has turned off one of the highest prices, having to be captive in an embassy, and has only been carried forward through the recognition of the community, but as it is, the general idea is to try to show you what is the history of what is behind in the first stage in which he was in the Ecuador embassy, and how it was, for example, that the surveillance cameras worked, which were pending the visitors who were going to see him, but nevertheless there were three large parts in this story, three phases that I would like you to understand the first was under the mandate of President Obama and that in the season when President Correa was in Ecuador, after that there was a phase in which there was a international security conference focused on protecting the Ecuadorian embassy, and of the Spanish company that had a great client, we will see that he was particularly interested in the Ecuador embassy, and how this company was selling to third-person people and receiving higher to take care of a yacht, so there is ongoing investigations also where the money came from and so on, and however in phase two, after some publications have been said that the Ecuadorian government, well, they began to change the protocols and the instructions, and there was also pressure from the American friends of the United States to have bugs, audio users and cameras to have much better surveillance. It is documented that this has been in the name of the United States, because we have the electronic emails and the lawyers have analyzed these electronic emails and we know that this was how these instructions were given. Sometimes his English was not very good, but sometimes his electronic emails had a very good English, and that is suspicious that it was not him, but it was other people, in this case the CIA. Well, and in the end I am going to explain the other phase. What I want to explain first is also that in April 2018 two things happened. The new Ecuadorian government made an agreement with the United States about the investigation against Assange to give them a reason to be able to do this. It was justified by the sense of avoiding the manipulation of the election, and the second thing was that it hired a second company, an Ecuadorian company that began to reinforce the security measures and the idea was that the information could be used in court, because all the information that had been obtained before was basically illegal. It was also an attempt of extortion by the director of WikiLeaks, the director at that time of WikiLeaks, and there have been publications about these attempts to extortion in April 2018. It would be good to point out that all the belongings of Julian Assange, his computer team and his equipment had been confiscated in the name of a cooperation agreement. What you see here is a facade of the low plant of the Ecuadorian embassy. It is not all the building, it is only what goes to a street. But the commercial center, Harrod, was nearby. On the side of the building is the emergency exit of the embassy, and what you see on the right is the patio. This is the entrance. In this same building there are several embassies, for example the embassy in Colombia is on the right, and there are more embassies, I do not remember any more. This is the entrance vestibule, and we have different camera perspectives. What we have here is a general view of the cameras installed before the updates in December 2017. What you see on the top left is the room and on the right is the office together. This is where most of the meetings with visitors, for example with the lawyers, had a place. And the green area is where all the sectors that were being watched by cameras were shown. Under the right there was a small bedroom where he had his bed. You can also see the entrance. On the outside there were some cameras, one for example of 8 megapixels, a monster that made me think that I also want a camera like this. But as you can see here the cameras that are outside. Well, you can see what the idea of this is. You can even capture some events that I am going to show you again. Well, as you can see the mall is on the left and how are the entrances. Well, the hotel was the official code of the security measures. It had its own terminology. The director was the ambassador. When they were talking about the embassy hotel, when they were referring to the hotel, they were referring to Juliana Saunsch. And this involved who could have access to the building and how the visitors were going to be treated. There was an official representation, an official person of the security service of the Secretary of the country, Gabriel Apaís, who was Secretary of the Ambassador, and was in charge of supervising the companies and how they worked, the security companies. We are talking more or less about 20,000 people who were going to the embassy for a month. 20,000 dollars per month that had been subordinated for the company to have access. This gives them more or less an idea of how the company accepted these modifications and how they made it known that they were in the name of Ecuador, but not of the United States, in fact. Well, there were other things that were also fun that I'm going to show you. The standard procedures that were there, they were that in the door they had to show their passport, there was a metal detector and they had to leave all the electronic devices in the lobby at the entrance with a diary in which they had to sign with the passport, then a guard would direct the visitor to a conference room, then the door would close and every week there was always a report where everything that had happened was mentioned. What the visitors would have was that the name, the reason for the visit, all of their pages with entry stamps and visas would be copied as well. All the visits and all the entry pages also appeared maybe in a description of how they looked, if they came alone or not, a description of how they got there and how they left. Here we have a lot of material of passport copies. This is an example of what happened at the entrance in November of 2016. It's one of the lawyers who arrived, who played the timbre, entered, was checked, was checked his passport. And finally on top of that you see one of the guards doing some upgrades on the cameras. How it was done was shown in the whole video, but he needs to give that to you. He needs to give that to you, he needs to give that to you. He needs to check his wallet, his bag to see if, to make sure that the visitors didn't bring a recording device or other type of device. What we see is documentation of one of the meetings. This is actually me, but this is something boring. Let me show you something else. This is the only positive that I have, unfortunately. This person from Sri Lanka, who also brought a cat. And yes, of course, the meeting that I like the most, but the others are pretty boring. The modifications for third parties that were made to the building in the beginning of 2017, after the communication changed to a system of encryption that had to do with the change in the US regime. The administration of Donald Trump was taking possession of the US government and it began to complicate the relationship of power. They started to pick up, in the middle of the year, the original Huttus from the network. They started to take over half a year of new recordings and then, morally, he flew from Spain later to the United States. And he would meet his American friends and send new instructions. They also started to optimize some parts of the system. For example, there were some problems with the camera filters, which later I will tell you a little bit more about how General Sacha had dealt with this. Also, the United States had asked them to have direct access to the cameras. So, what was done was to install them in the embassy so that they could have access to the Ecuadorian people and the people from the United States of America. This is one of the emails that I have. I don't speak Spanish, but here it is talking about the access to video transmissions where you have one for the North Americans and the other for the Ecuadorians. And I show them here only as a reference to the North American service. And then they started to do some other things. For example, they installed a phone on the doors so that they could have a phone or a telecommunicator. So there was a desk in the vestibule where the security guards were. And then if they heard the name of the person who wanted to enter, they would record. And then the people who were in security started to take recordings of the visitors' voices. They would open the door, they would take their passports, they would close the door with the person outside. In that time, while the person was outside the visitor, they would take copies of the passports and then they would enter. They would take all the email and MSI of the different mobile phones. And for example, here they would photograph all the items. For example, there are examples of a national security guard. This is my phone and a cryptocamera. And this is an Italian activist.  And as you can see, they took out the SIM cards and to photograph them all. Here you can get an idea of the documentation of the meetings of December 2017 with the new cameras that also had audio. You can imagine that you can see that the quality has increased, clearly. However, that was not enough. Some of the meetings included white noise. Well, they had some problems with the white noise. The cameras that were in the ceiling were not enough. Therefore, they made other modifications to improve. Here you can see where they ended up later. As you can see, they played the main role. There was the extensor and one thing down there. Some of the plastics out there, they took out the plastic and put the camera, the vigilance, on the microphone. The only issue they had was that it was a battery drop model. They had to replace the battery and the SD card, which they did that every 12 hours, 26 minutes, actually. That was the capacity of the thing. But you get the idea, they looked into all of the things here on the left side. By the way, you see the noise generator, but they looked at all the art objects, all the kid stuff, to see what could be suitable before they identified the fire extinguisher. It's just the best thing to do. Here you would see that Julian also installed a loudspeaker next to the window. There was not only one loudspeaker above the table for the white noise generator, but also directly at the window. Here you see very briefly on the top the loudspeaker over the table. But they also opened during the survey all the electrical wiring things, the light switches, and so on, to see if there would be any sealable places. They also did that in the bathroom number two. Bathroom number two was originally the woman's bathroom, which also had a shower. It was then later more or less Julian's personal bathroom, where he also made the most critical meetings with his lawyers and associates while running the shower, as you would see it in every better spy movie. However, what they did was behind it, you can see here these boxes above the cabinet. They installed the microphones above all these boxes, and we don't know if they had good audio quality. Through the things in the kitchen, for example, this could also have a microphone in the kitchen. The other aspect that Morales discussed with the people in the embassy when he came back from his travels, is that he asked them if they could accidentally leave one of the doors open and kicknap him. And if someone could come and kidnap him. Or, for example, if they could put poison in the food or other possibilities of poisoning. But, for example, this opens the door to something that would be completely illegal. This is not the United States. What do they care about laws? Well, they don't really care about the law, what is legal or not. So, for example, they went through all the doors and analyzed how a kidnapping could happen, for example, or the movements that someone could have, how they could kidnap him through the embassy. I have this video, for example, that you can imagine, you can have an idea of how it worked. Well, we can identify the person who did it. We have three videos of this type, of the rooms back and forth, and they also considered what happens if it is through the window, instead of through the door. This is still part of the plan. For example, to identify all the windows and to name them. For example, how they could identify all the windows, how they could enter above, for example. In 2017, they installed all two microphones in the exit and above the toilet, in the second bathroom. They also discussed that they wanted another room, where Julian had a small library and a place where that room was connected to his people. But to be able to watch it, they would have to have watched it through the embassy's room. But for that, they would have to have heard it through the doors. To be able to do that, the United States introduced them to take out a spoon, a sample of the wall, to know what kind of surveillance they had to install to be able to go through the wall. Well, here we also see other places where there were microphones and how they documented their work. Here, for example, there is photographic documentation or video of the update of the cameras. This is the updated diagram of the camera installation. You can see all the cameras now are four megapixels, and now the conference room has two. And so they can cover the whole room. No, I don't show you the two positives, but now we can see that all the rooms have been covered by the surveillance. This noise generator that Julian installed was really a headache.  It's pretty tough to do that for two hours. However, the Americans had problems with that, and they had already been trying to penetrate the embassy from the outside with lasers that shoot through the windows. You can detect vibrations in the windows, and that way you can take measurements of the sound that is reflected. They were able to analyze it this way. It was one of the measures to counteract the white noise. We have to see in which places these stickers were placed. This was not only in the conference room, but also in other windows. In fact, the cameras that were outside, we can see a recording of a North American person walking outside with the laser machine, with the laser ray. What you see here is an idea that is not the camera. There are other things that have happened. I would like to show you just for entertainment, that some times this was not only spying on Americans for the benefit of the North Americans, but also for the security system. At the end of 2017, these things happened. During this year, the new government of Ecuador was elected, but they had not taken possession of the government. That is, the situation was still a responsibility of the previous government. They had agreed with some countries as a diplomat in one of their other embassies, where he would look for a job in other embassies. On December 21, he had a picture from that meeting. What we see here is a meeting on December 21. At that time, the Americans were on alert. There they made sure that the people were not near the extinguisher. They even introduced that the guards were outside the door and heard through the door. That same night, the next day, they issued the international arrest warrant for Julian. They knew that the plan to arrest Julian as a diplomat, which was the previous idea, would not work. They knew that it would not work. This is how this term of active intelligence comes to be produced. At the end, I have brought some resource links of cases that have been happening on the back-planet. If you have been reported on the back-planet.info, if you want to support this type of investigation, you can check it there. We also collect money to help all these cases of surveillance. That is all I want to tell you. We have a little time for questions. I forgot to mention that the sticker is on the windows. This building is under CCTV and Morales has told his people that the British law has changed and they would require the sticker. What happened was that these stickers were now required to be arrested. That was the justification they were given. The justification was that the British government had changed the law and that the windows had to be defined. The building was under CCTV surveillance. If we go with the questions now, we also have questions on the internet. In this case, I am not only going to answer questions, but I will also suggest how to solve this. We have a question from the internet. A short question. About the lasers. What is the goal of the lasers and the technology of the lasers and the stickers of the stickers? I am not going to say it clearly, but the stickers reflect everything from the outside. The way they are attached to the window separates them from the internal noise. The truth is that I do not have details of how this really works. In a way, it helped the Americans in their laser technology to capture the vibration and capture the voice of what was happening. It seems that there is no... Maybe I want to... Yes, we have another question. Yes. Hello. It is not necessary that you tell us the source, but without telling us the source, you can tell us where you got all these videos and this information you showed us. Well, as a lawyer and lawyer, the truth is that I am an activist journalist. Therefore, I cannot reveal it, but this is in fact being investigated. I have discussed this with Julian. And I should say, before I go into detail in all this and look at this material for hours and days, of course, Julian and I would have... Julian and I would have had a good situational awareness if we had thought that we were paranoid, but in reality what we had was a good... We realized that we had something good. We have another question. At the beginning, it was mentioned that there are several embassies in the same building and that is what happens. It is about the vestibules of the building in general and this can be considered a neutral area because there are several embassies. Well, the corridor is from Great Britain, so all the apartments in that building are private. Well, maybe there is a better... In English, I don't know, but the diplomatic police, which is better than the regular police in Great Britain, they are not authorized to enter the embassy, but they are in the corridor in front of the door. But for example, they could also hear the conversation if one is talking to a guard, for example. In the first months, there was a lot of police outside. And as the police was on the side, they installed a vehicle that said police conference in which they had this vehicle that was constantly in front of them. We have one more question from the internet. Have you been approached by the CIA or by some other agency about these security operations? Not in a direct way in which they have talked to me directly, but in terms of surveillance, for me, when I'm, for example, coming in the morning on Heathrow, in London, they asked me the same question. The official, for example, asked me the same questions three times and repeat and repeat the questions without even listening to the answers.  So what happened is that after all those questions someone came behind me, because I followed him all day. And that's a form of intimidation that made me realize that I'm not welcome, that that's what's happening all the time.  We have another question. Thank you very much for the chat. I would like to continue with what you just mentioned, to be under surveillance. And I would like to talk a little more about the fire extinguisher. Well, I'm going to give you a chat in the next few months about the surveillance of my microphones that you may have asked me. For example, there was an experience that occurred to me, and at some point in the night, of course, it's a beautiful loss of quality of life and of becoming a little bit different in your behavior towards other people. And, well, change the behavior towards other people. For example, this is how you realize the situation. Basically, if you look at it from a balance of... Basically, if you look at it from a balance of... between the measures and the options, they have an endless astronomy military budget that is incredibly military, very big, and we have cryptography and encrypted phones. My guess is that I'm not doing anything illegal. I'm just studying this material and this case. And that includes, for example, photos of the bathroom. And for me, it's very clear that this is not about him having violated any laws. So it's very clear that he has violated the law, but that it's a political case, it's intimidation, and they want to give the president a... like hanging him from a tree to give an example that he should not be doing. On the positive side, he has shown that it's possible that the internet can provide more information about the government and about the secrets for all of us. And that has consequences, it has a price. This seems to be becoming a bigger dystopia, bigger, bigger. Let's take another question from the audience. Thank you very much for the presentation. It was very important to see the details of these images and the details that we were told about how the embankment was being mixed up. And as you mentioned, all of this was done with complicity with the embankment, with the Ecuadorian embankment, even though you tell us that there was no long-term relationship. And I wonder if there is a way that you can defend yourself or that you can try to counteract it and that you can consider this as a violation. Well, what I can say is that the Spanish lawyers of Julian with a lot of pressure and also with the Spanish judges have examined the evidence and the owner of the company is under arrest. The computers and the hard drive, USB sticks have been confiscated. That's what Julian's main extradition case looks like. That's going to be very important for Julian's main extradition case. Some entities of the United States, or some entities of the United States that are located in the United States, have not been able to develop their defense in a legal way. And that would be the first argument. They say things like in any normal case, this would be the end. The lawyers have told me that in any normal case, this would be the end. This wouldn't happen. No one expects a fair trial in this case. Well, we also have to say that he was under surveillance restriction for 24 hours a day. And he clearly is not a terrorist. And clearly with this we see that this is not a normal case. The regulations of the United Nations of Torture and this is completely out of what has happened. Clearly this is not going to be a fair trial. Of course, there are problems with this, but with the United Nations and the United States. What we have to do is to demand from the British embassy that things have to be more human. When the cut is the fear is that he's kind of being killed right now as we speak. The risk, the concern we have is that he's going to be killed before the trial ends. We have another question. Do you know if the Spanish company continues to be in charge of security in the London embassy? Well, the contract of this company has been canceled since 2018 and that's not happening anymore. What I can also say is that an important aspect of this is that the embassy had their eight hour shifts there because they were protected witnesses of the Spanish government and they were not happy with what they were doing. They didn't want us to have a problem with the official content of what happened. There was a certain resistance to what had happened. Something of this resistance did not share that money. The other resistance did not share that money. This company started to charge more money, but that was not shared. The people who were working there are unhappy that they were sitting with Julian all day. We have another question from the internet. How do you explain all the effort that has been taken to intimidate you, what you mentioned? How have you been watching it? I have a question for you. In your opinion, how many security services and information from the governments can be in this room right now? I don't know if I answered the first question completely, but the first question could have introduced me the other way. I could have said, go to the Mueller report and go to page 46. Look, I brought Julian food. I brought him clothing. I brought him clothes, food, packages, sticks. I could never prove what I had not brought him. They told me that I was the one who brought the USB sticks. I will never be able to prove that. I am in a small whole foundation that works for the freedom of information, that has worked in different cases. And the truth is that I would not be surprised if they wanted to kill Wikipedia instead of Julian Assange. And of course, I'm not the only one that's being followed. How many security agencies in this room? Out there on the stream, I see 23 different. In the stream, I see 23 different. Five to eight. It's been quiet the entire time. The man next to you seems a little suspicious. What can I say? From being blackmailed, maybe that was Laos' idea about it. But of course, not all aspects of WikiLeaks can be subject to full disclosure. Because there are protected witnesses, information that has been protected, that we can't... And other sources that we can't comment on, and clearly as a medium project, are things that we can't... that we have to protect. We have another question. Is there someone from the CIA? No, let's move on to the microphone. It's not brutal. There are some diplomatic consequences in this case, because we have the British government, the CIA, and it violates the Ecuadorian embassy. The Ecuadorian embassy, installing cameras, is not something that I understand as being correct. I would think that in any normal European country, there would be an official investigation, for example, of other embassies in the same building. Great Britain has not only decided to leave the European Union, but also to get closer to the United States. The truth is that I don't think the current government of the West is allowed to... I think the current government of Ecuador is quite pro-US. They have made this agreement with the United States to start the investigation. So, I don't know if the strongest thing going on is the Spanish investigation. There have been things against Victoria about secret services, but this is ongoing and it's still in process. All these links are in my WikiProp, where I'm collecting all the material, especially, for example, of Spanish news. And I think they will learn a lot more in the next few weeks. All of this is in progress. We have now a question from the CIA. That was the truth of the CIA. I would be very surprised to find evidence, because one of the main characteristics of vigilance is that you cannot prove who it was. That's exactly the confirmation. And so to say, the proof... We have evidence that things are going on, but we don't have evidence. Well, that's almost the definition of vigilance made well, because... I don't know why I'm telling you that, but, well... Well, what happened is that the way the cash was transferred from... Well, from this person who transferred money, the truth is that... Not all things were transferred by Sheldon Eddison, but the truth is that this was quite... an international intelligence operator, quite rare. He can... He has cash in Las Vegas. He has... He has made espionage operations in... Macao, accused of wikip leaks. He's a very interesting person, he's 93 years old, he's involved in business in Macao, in business in China, in business with the North Korean press. Actually, I would also like to know about some of the fantasies of a friend of mine who's really in trouble. Yes. Well, if you have fantasies of how to get him out of there, not only of how to fight against the CIA, but also of how to get him out of where he is. We have one more question from the internet. What is it? What would happen? The question is, what would happen with wikileaks if Julian Assange is in a perpetual chain and for the last comment from the internet. It's a thank you for the talk and a request that I can give again. I don't speak for wikileaks, but I know the editor-in-chief and he's publishing things. The last publication, for example, was two days ago, on the 27th. Of course, they decided to continue with the work. They think it's the only thing that they can do to demonstrate that this is a publishing organization, a publishing organization, a publishing company that has been working with media and so on, that has collaborated with other people in the media. Where they are located, they are much more supported by where they are located. Iceland is a very small country where there is a bank scandal where Cristian is now. He is in a more friendly environment with wikileaks. Well, for wikileaks or for the Great Britain, if someone has ideas of how to improve this, we have to collect ideas to get Julian out of jail. We can take another question. Since a long time ago, we had the suspicion that there was a request of extradition from Julian and now we know it. And well, this already seems that we can be sure that this is not a conspiracy theory. And now, how can we help the next Julian who we don't know if he is under surveillance and what we can do about it in the future? Well, as I know, what I know is that the publishing activities of wikileaks are not illegal. They have not been illegal. Yes, there is an investigation that is taking place, this investigation is about how the sources have been convinced of being sources. For example, the idea that Chelsea Manning was instructed to do something and to publish something later, the fact of publishing was not illegal. And as you know, Chelsea Manning is in jail again for not having testified against Julian exactly this question. It's not only what is in danger, what has been published, what has been published, but also the sources and how they are put into words in the collection of sources. It's difficult for me to say, difficult for me to say how this should continue. But in terms of transparency, it has been proven that this can be done through the internet and it has been fought by the people who are in power. And the truth is that we have the right to see what is happening. How are we going to do that with the new law in the United States, even in Great Britain? Well, here we are in Europe. Germany has a rather strange history. Other countries have a constitution that includes the human rights. I see that at least in countries like France, Germany, it's not wrong. We can't allow other countries, for example, the United States, this is still happening.


Hey everyone, we're going to talk about data science for infrastructure. So three steps, what we think are the three steps, observe, understand, and automate. Before I get started, just a really quick overview. I'm Zane, and this is Natalie. I'm a GM at New Relic, and formerly the co-founder and CEO of Pixie Labs. We got acquired by New Relic, and both of us are working on the Pixie project now. And Natalie is a principal engineer on our team. So we're both machine learning people by background and have basically spent the last decade building scalable data and machine learning systems. And we really wanted to try to understand how to apply our knowledge from that space into observability and infrastructure. So when we started in the space, we started to think that this is just another data problem. It's easy for machines to really very quickly generate gigabytes of data, but it's really hard for us to actually get complete coverage, especially in distributed environments where it's hard to instrument all your applications. It's also hard to make sure that all the data you collect is relevant, and it is difficult to then distill this information into something that's usable to automate workflows. So we wanted to take what we learned from the data space and apply it to infrastructure. And we kind of realized that there's a few different problems that need to get solved. So one of them, I think you've probably heard about this and everybody is collecting the right data is very difficult. And we say it's half the battle here, but in reality, it could be way more than half the battle. And one of the other takeaways from our data and machine learning days is that simple models on good data or relevant data usually outperform very complex models that are built on skewed or inaccurate data. The other thing we really learned is that it's really important to be able to audit and understand what's happening in your data pipelines and that you don't just have ad hoc systems running your infrastructure. So for this, we kind of think about how to break down the steps in the three different stages. The first step is to observe your data. The next step is to understand it and the third step is to automate. And we'll kind of walk through each of the steps and see what that means. So for data-driven automation, the first step is that you gather the raw data. So this might be something like getting the CPU utilization. It might be capturing request information. It might be trying to understand what are specific parameters in your request and how they impact your latency. In reality, most of the time is actually spent over here, even though a lot of coverage is given to the next two steps we'll talk about. So the next thing you typically do after you gather the data is you do some kind of data transformation. And this actually gets the most disproportionate emphasis. A lot of times, you will talk about machine learning models and all sorts of sophisticated tools that can build over here. And something over here could be a simple statistical machine learning model. It could be a regular expression. It could be aggregates. There is a whole gamut of things that it could actually be. And the goal over here is to be able to transform the raw data into some signal that can be usable downstream. An example of this is you basically take, let's say, CPU information for a bunch of different replicas of your application running, and you say, oh, look, this particular replica is extremely high relative to the others, what's going on. And the third step is that you take these signals and then you do something to automate it. So it's pretty common to be able to generate limits and alerts with existing tools. But one of the things that we've been pretty excited by and we'll talk about over here is the possibilities by using the Kubernetes API to really connect the first two steps and then close the loop. And by doing that, you can ultimately take data all the way to the entire pipeline, have Kubernetes go adjust the actual application parameters and then go back into the cycle, gather data, transform it and automate. So you can create like a full closed loop system. So there are a lot of steps over here. And I don't really wanna go into the details of exactly what types of metrics or what types of information is relevant. But just quickly, for raw data, typically people are looking at logs, metrics and requests. For transformations, you're looking at aggregates, anomaly detection, regular expressions. And on the signal-based stuff, you're basically looking at trying to update some systems or be able to do things like allocate more resources. There are a bunch more things you can do that will leave for a reference, but not that important for this particular talk. So there's a lot of stuff to do over here and how do we actually build a system that can handle this? So when we started to build Pixie, our goal was to build a data system that we could deploy on clusters to really understand what's going on and then automate workflows. So Pixie was built to solve these problems. We kind of have three core principles that we follow. The first one is to try to do as much automated telemetry as possible and we utilize this new kernel technology or maybe relatively new kernel technology called EVPF. And there are other talks that talk about this, I'm not gonna go into details, but in industrial it kind of allows you to go insert code dynamically into the kernel to be able to observe applications. The second thing we really wanted to do is make everything scriptable and API-driven. So you can actually write scripts that can work with the data that's captured automatically. And the third thing is to have a Kubernetes native system so that we can actually understand the entities and be able to work on that. So just a quick overview of the features or feature set of the project. Yeah, I think I forgot to mention, but Pixie is an open source CNC of sandbox projects. You can check it out. But in terms of auto telemetry using EVPF, we can capture application network and infrastructure data. This includes things like application profiles and full body requests. And we have pretty low overhead while doing this. You know, we say under 5%, usually under 2% CPU overhead. We basically understand all the Kubernetes entities and connect things together. And the third thing, which will spend a little bit more time on in this talk because it's relevant to the latter part, but not only will we be working through some demos, is that it's 100% scriptable and API-driven. So we really think about this as infrastructure as code. We can basically write code to be able to generate all the metrics we need to automate. And then this can be accessed to the API. So instead of Pixie, everything is a script. Our UI actually runs as a script as well. And we can easily integrate in the downstream tools for our API. And they're pretty active, Grafana plugins and plugins for other SaaS products for Pixie. So like I said, we'll talk a little bit more in detail about Pixel, which provides, which is Pixies like language that provides us with the capability of building these data science like tools. So we kind of have like three principles when we're starting to think about what Pixel would be. The first thing is we really wanted to be able to use the query data and build workflows. The second is we wanted to be able to automate some more data collection that we didn't think about a priori. And the third thing is we really did not want to invent a new language because everyone starts off starting to invent a new language and it gets very complex for users to use. So you can actually see a little snippet of Pixel code on this, I guess on the right side for everybody. And that particular code, not to go into the details, but it's based on a Python, it's based on Python, it's actually 100% compatible with Python. And it's based on this library called pandas, which is for data processing. And it's very commonly used in data science. And you can basically see over there, we're saying, oh, look at all the HTTP events over the last 30 seconds and then capture what taught information's there and then pick out some specific columns. I don't really want to walk through the details of the language, but it basically gives like a pretty simple way to work with data in your system. So just to recap, Pixel is an embedded DSL. So it is still like a domain specific language. It doesn't work like generic Python, but it is valid Python. And in particular, it is valid pandas, which is very common in data science. And it's built for data analysis and machine learning. So we have core capabilities like, running TensorFlow models inside of Pixel, being able to run aggregates and joins and stuff pretty quickly. In particular, just to go into one more level of detail, Pixel is what people call a data flow language. So it specifies what is a logical flow of data. And it's really Pixie's job to figure out how to optimize and execute this so that it's efficient in your system. So the next step you're probably interested in is what can Pixel actually do? So Pixel's scripts can be used to transform and analyze your data. So some of the things we support are aggregates, joins, filters. There are a lot more things to do. But the good thing about Pixel is that it is composable. So it's a declarative spec, which means that it's kind of like SQL. You say these are the steps that are done. It is fully functional with no implicit side effects, which means that you can build workflows off of each other. And if you take a look at Pixie's UI, if you get a chance to check that out, you'll actually see that we pretty much compose multiple scripts together to be able to generate our entire user interface. So to recap, Pixel provides an interface to work with data. It allows us to construct powerful composable workflows. And the next step over here is that we have a couple of demos that will actually demonstrate those capability that Natalie will walk through. So the first one is doing slack alert on SQL injection attacks. So it's like, Natalie will talk to what SQL injection is, but if there's a SQL injection attack in your cluster, we'll be able to flag that and generate a slack alert. The second demo will actually take a look at HTTP request throughput and auto-scale your Kubernetes cluster. Oh, thanks. These are, thank you. So these are all like pretty simple examples, right? But our goal is to be able to build these up and get more feedback on what we can do. With that, I'll hand it over to Natalie for the demos. Great, thanks, Zane. Okay, sounds like this mic works. So as he said, I'm Natalie. I'm gonna be taking you through some of these demos that are kind of meant to be proofs of concept to illustrate some of the principles he was talking about before about how we can automate more of our workflows using the raw telemetry data that exists in all of our applications. First, we would need to deploy Pixie to a cluster. This takes about three minutes, so I've created a video to kind of fast forward that for you. But I think what we're trying to illustrate here is how we're trying to set a path where it's much easier to collect data with EBPF. Instead of spending a lot of time manually instrumenting your application, you can just run something like PXDeploy or other EBPF-based tools and automatically start collecting this raw data from your cluster. Okay, so as Zane said, the first demo is going to be creating a Slack alert in order to detect or creating, sorry, creating a pipeline to detect and do a Slack alert when we think that there may be a possible SQL injection attack. So for this demo, before I get into what a SQL injection attack is, we're gonna be using DVWA, which is a well-known security project. And I'm just gonna pop up for you. Oh, I think my port forward has stopped.  Basically, this is kind of like a test security application that is used in the security industry that has a lot of known vulnerabilities in it. And so basically it is intentionally built to be insecure. And so people can use it as a testing ground for running different kinds of attacks. So basically, we're using this in order to demonstrate the SQL injection detection that we're doing. I guess I'll go back to sharing. Oh, actually, one thing first. So just to kind of show some of the data that Pixi can collect, let's run a SQL query with DVWA. That isn't actually a SQL injection. So let's say I just want to run a single SQL command. It should have executed. Now I'm actually gonna use Pixi in order to see that this query has been executed. And this is a very simple view in Pixi, but it basically collects all of the MySQL data on your cluster. So I guess I should say that we've deployed this DVWA application to our Kubernetes cluster. So that's how Pixi is collecting this data. So if we were to kind of inspect this row, we could see that we're actually able to capture the full SQL request that I just executed right here in Pixi. The font may be a little small, but that is the foobar123 that I just executed. So let's go back to what actually is a SQL injection. Well, I didn't want to come up with a definition for you because security experts are much, much better about that. The way we think about it is it's just a way for people to try to run malicious SQL in your application in order to get information that they're not meant to have access to. So just to take a very, very simple example, it's worth noting there are many, many kinds of SQL injections, so this is not a cohesive, you know, complete example. There are many kinds, but an example one would be, let's say I have an endpoint where I pass a user ID, like the one I just showed you in DVWA. So I pass in user ID equals 123. On the back end, my application executes select star from users where user ID equals 123. This is intended to give me the information for that user, but a malicious actor put something that isn't an ID in there. Let's see what happens when that occurs. The application would execute if it was naively implemented, select star from users where user ID equals 123 or one equals one. This would effectively return all of the users in the table. This would be really bad. So I'm going to talk about SQL injection detection, which we think is really important because no application is perfect and free from vulnerabilities, but it's worth noting that prevention is really, really important when it comes to SQL injections. So detection is only part of the story. We really want to be as application developers thinking about security and doing prevention as a first step. But let's say that we have an application out in the wild and we want to make sure it's not running any SQL injection attacks. There's two main ways that we can do this once we have a list of all the SQL statements that the code is running. The first is a rules-based approach. You might say, I'm going to parse all my SQL queries and make sure that it's not actually using any syntax that I have deemed prohibited, such as unions. You could also use reg-axis to detect this type of syntax. There's a complication with this approach, which is what if there is a legitimate use of that type of syntax in your query? Another way you can do it is using machine learning. You can train your model on real-world examples of things that are legitimate queries and things that are SQL injection attacks. That could theoretically learn that using a union in this scenario is okay, but using it in a different scenario is not okay. A major complication with this is where to get the data set. You have to have a list of data that has known SQL injection attacks that reflect your application. So this can actually be a major blocker to doing something like the machine learning-based approach. And so for this demo, I'm actually just going to be doing the rules-based approach with reg-exes, but as Ian said, Pixel supports running TensorFlow models. So if you were to have the data set available to train a model on detecting SQL injection attacks using machine learning, you could easily plug that in instead of the reg-ex-based approach. I just want to call out, in addition to DVWA, a tool that is helping this demo is SQLMap. It's a really cool command line utility that basically tries to attack your database and figure out are there vulnerabilities in it that exist. And so we're going to be using that today. Okay, so let us get started. So first off, we want to have the script that actually detects the SQL injection attack. So what this script is doing right here is it's actually just showing you, hey, what are the SQL queries that exist in my cluster? But we actually want one that runs pattern matching to detect what ones could be a SQL query, or a SQL injection. So let's first populate the table with some SQL injection attacks using SQLMap, like we just said. I'm going to point it at my DVWA. I'm going to say yes to various things.  It's trying to attack the database and execute malicious code. The data tends to pop up pretty quickly. So what we can see here is that the SQL commands that this thing has been running are showing up in this view. So I've written a script, or I know I have not written a script. We have written a script. We asked actually the help of a security team who wrote a great blog about this, gas blog on our blog website, to basically look at all of the request bodies for the SQL queries and try to label them, like which rule has it broken. So what we're going to do is instead of the script running here that's querying the MySQL data, we're going to add a little bit more functionality to this script and actually try to classify possible injections. So basically this script has successfully given us a list of these injections. Now some of these might be legitimate queries, but we just want to flag things that are breaking the rules that we've said aren't allowed. So what we can see right here is that someone wrote a very, someone aka SQL map, wrote a very suspicious looking query right here. This does not look like a user ID to me. And we have the rule broken, unmatched single quotes. But the thing is that I don't want to have to actually try to manually run these queries every time. I want to actually be alerted when there is a SQL vulnerability in my application. So because Pixi has an API, we've actually created a Slack bot that will alert my Slack when there is a problem that looks like it could be a SQL injection. So basically the Slack bot took the exact same script that I just showed you, and it's going to use Pixi's API to ping Slack or to create the dataset that will then ping Slack if we see it in attack. So we're going to let's first create some more attacks. And if the demo gods are with me, we should see, oh, it's detected a possible SQL injection. So this thing has told me that a similar, I guess it just always runs this one first, unmatched single quote. It's running this one a few times. And so what we can see now is that I have a full data pipeline that is looking at my SQL data and pinging me on Slack whenever it detects a possible injection. So here's a new one, comment dashes. This definitely looks suspicious. So to recap, we've basically applied the same thing that Zane has talked about, which is we use Pixi to gather raw data and transform it into a signal by collecting these raw SQL events and using a script to try to use regXs to establish which ones might be SQL injections. And then we're doing something based on the signal, which is pinging Slack because we see that there's been a problem. All right, let's move into the next demo for a completely different use case, which is basically workload scaling. So I think a lot of us who use Kubernetes might run into these questions often. How big should my deployment be? How many pods are necessary for the service? How many pods are necessary for the service if they have a lot of CPU and memory associated with them? Both the size of the pod itself, in terms of the resources it gets and the number of pods are important factors in sizing a deployment. It's also worth noting that the size that my deployment needs might not be the same all the time. 2 a.m. in my time zone versus like Black Friday or something like that, I might have completely different needs for the size of my deployment. So there's various things that we could use to size these deployments. One of the most common ones is using the CPU and memory usage of the pod. If my CPU and memory usage of my pod is high, I should probably add more pods because that one looks overtaxed. I might want to look at things like request latency. This request is taking too long. Maybe the pod has some kind of resource utilization issue. Maybe I want to look at the latency of my downstream dependencies. Maybe I want to look at my number of outbound connections. Maybe I want to look at application-specific metrics, like how long this particular function is taking. There's so many things that we could use in order to drive the auto-scaling of the deployments. Also, all of these slides are part or uploaded to this session, so if you want to look at them for reference later, they're available. Kubernetes is amazing. It has such an API-driven approach and it makes auto-scaling based on these complex factors possible. Right off the bat, it supports both horizontal and vertical auto-scaling. Horizontal basically increases the number of pods in a deployment. Vertical increases the amount of resources allocated to a pod. So built-in has the ability to auto-scale on CPU and memory. Like I said, these are very common ones to use and they're very valuable. But it also allows you to define your own custom metrics, things that you know your deployment cares about. Something like CPU or memory applies to everyone, but you may have something very specific for your application that you need to scale on. The Custom Metrics API is really useful because you can hook in your own metric into the auto-scaler to add more pods or add more resources to a pod. We're going to be using a very sophisticated demo app. It's basically just an Echo server, but it is enough to kind of make the point about auto-scaling. That was meant to be a joke, people. No. Anyway, you know, the Kubernetes SIG did this amazing, basically like demo app of how to make a custom Kubernetes metrics API server. So we're really appreciative of the work they did in putting that together because we based this example off that API server. If you want to build your own, you should totally use this project as a starting point. Also, we're using the tool called Hey for HTTP load testing. Rackle, aka Yana, who's one of the board members of the Pixi project, created this tool, and it's really easy to use. So let's go into it. I'm going to go into K9s, of course. And I see that I'm running this Echo service on this external IP. And these are the number of pods that it's running. It's just one pod right now. And I've deployed this Echo service with an auto-scaler that basically uses Pixi metrics in order to say, I would like more pods when my application is serving a lot of requests. So I'm going to launch this and then show you a little bit more about how that works. Let's give it some load. Let's give it a lot of requests to the same endpoint that we just saw. And while we do that, we'll keep it over here in the background. We can see that these containers have just been created in response to the request. So how does this look? Basically, we have this horizontal pod auto-scaler that we've defined. It says, I don't want any more than 10 pods. So this is the max. It's kind of like a switch in case you accidentally steal to 300 pods or something crazy like that. And I want to use this custom metric I've defined, which is PX HTTP requests per second. And I've set this target value to be 20. And so what happened was this pod saw that it was getting a lot more than 20 requests. So the auto-scaler decided to add a lot more pods. The load has died down. So now we see that these pods that were just been up are terminating. And you can configure using the auto-scaler API the amount of seconds and, you know, how many pods can get added or taken off at once. I just made it really fast for the purposes of this demo. The metric provider that we have is pretty simple. It basically takes this pixel script, which computes the average number of HTTP requests by pod over the time window. And it fulfills these endpoints. These are defined on the custom metrics API. Get metric by name and get metric by selector and list all metrics. So you can see that in just about 200 lines of code, I've easily created a metric server that will auto-scale my request based on, or auto-scale my deployment based on the number of requests. So to summarize what we just saw, we collected the raw HTTP requests in Pixi, although you could do this with anything that taps into HTTP requests. We've calculated the request per second by pod, and we've plugged that into the Kubernetes API at auto-scale or deployment. I'm passing it back off to Zane. Thank you. Awesome. Well, thanks for those demos, Natalie. So in these demos, Natalie basically showed how you can take some simple data, or run some simple data workflows on Pixi. In particular, for the SQL injection, there's a blog post on Pixi's source blog that goes into a lot more detail if you're interested on it. And then we're going to be posting a blog on the auto-scaling project next week, which will have a lot more of the technical details about how this works and code that you're going to look at. Actually, sorry, one quick thing is that these examples are in our Pixi demos repo today if you want to check it out, but as you said, the blog, what are we coming? Correct. So they're already all in the public, but there is a blog post coming next week that I'll talk through the information in more detail. So what are we working on next? So we're working on doing some stuff with cross-site scripting detection using FIXL, and really what we want to do is learn from all of you about more of these cases and things that we're interested in seeing. You can find us on GitHub or Slack and either create an issue or send us a message on GitHub that could either be the CNC App Slack or the Pixi Slack. And that's it. Thanks a lot for listening and we'll take questions. Thank you.