Hi, so I'm first a bit about myself. So I have two sites one is an entrepreneur and CO on my other side during the nights. I really enjoy participating in in deep learning competitions, and this been recently going on since 2017 and typically I spend my time in a place called gobble, which is a Website where people from all over the world compete So typically companies have a problem and they want to crowdsource if you will the solution to different teams from different Perspectives, and they have been really getting momentum in the past few years So you can see I got really excited when I was a browsing through Twitter And then I saw this guy who I follow so he's a Chinese guy who happens to be doing his PhD In the US and then he said something on the lines of hey You know so this is what he said the way I read it was okay So, you know you guys do these fancy competitions in the West in Kaggle, but in China You know we have these places right so he List you know different websites which you had no idea that existed then I went into one of these websites And actually I wanted to do a few of them and they were really really really Challenging problems in China. There's a whole world of deep learning and the learning competitions that we at least in in the West I didn't you know, I didn't know anything about them So I want to one of these places and of course I see this Okay, so everything is in Chinese, but then I hit translate.  So, you know, I decided to join So I kind of find the the join Button in this place and then you know the registration said basically, you know in China everything happens on the internet You have to have a cell phone. So in China is really mobile first. So the authentication required you to have Chinese mobile phone number which I didn't have so, you know, I was a bit frustrated, but I am Persistent person.  All right, no response. I got a bit frustrated then I then I read Also on the MIT Technique review that, you know, China has this great master plan to become, you know, an AI powerhouse in 2030 and they said, you know, we are be we are a bit isolated, you know We really want to be more collaborative with the world, etc. And I'm thinking to myself, you know, how it upgrades these guys You know who you know, they are like, you know boasting about that They are going to be open and I cannot join the competition. So I write again, right? I write again No response in the first email. I read again. Hey guys, it's me again. The guy from Spain I really want to join this competition. No response.  I cannot join and I'm thinking, okay You know, Andres, hold on because if you ever want to go to China, you know, if you do this You may not, you know, be able to come back And I may be sent to a like a education camp or something. I don't know So Then I go back, you know, I compose a tweet, etc. I was about to tweet it but then I go to the website again and oh my god, they added Spain So there's like China, Hong Kong, Taiwan and Spain Hopefully we don't become a province of China So I joined and you know, I joined and then you know, when you join this competition You have to pick up a name for your team. I have this friend this friend So I live in Alicante. I have this friend from Madrid. So I'm not from Madrid I have no idea about almost anything in Madrid and he tells me like he lives in a neighborhood called Sanchinaro and I'm thinking to myself These guys in Madrid are crazy Do they have also like a San Korean arrow Or some Japanese arrow, too So it's you know for you English guy Sanchinaro could be almost translated as like a saint You know and Chinese and big Chinese guy Right, so I'm thinking so this is why it's all Sanchinaro So, you know, so this really is taking my head. I don't know why so I was like whole day thinking about something I was Sanchinaro and then I joined this competition and then I put a You know that in my in my name Now the deal they know that eventually we were going to win So, you know, usually I got asked right like, you know, the Chinese guys really wanted to know what Sanchinaro meant, right? So it was very difficult to explain.  So there's the in China, you know, the same way in the same way as in the West we have Aristotle, you know They from the Greeks so they have Confucius, right, which is a very wise person So they have a pronoun to name, you know wise people, right? So I said, you know, these first two characters are like the equivalent of wise person, right? And Sanchinaro right China in Spanish could also mean stone By a long stretch And then the other part is, you know, do you know this stone, which is a precious stone green jade stone? So so I tell the guys like Sanchinaro means wise jade stone And then the guys said, oh my god So that's just for the name So that actually I Competed so this is kind of my bio and I joined this competition with a friend of mine who I've never made So I've met this guy with color in competitions So, you know, we embarked in this challenge So now going back again to the competition what this is really about is to they give you Later point about what is called a point cloud if you see self-driving cars They have this device on top of them except for Tesla which for for cost Reasons they cannot add a lighter Almost all of the others especially Self-driving fleets like taxes. They do have like Uber has this, you know ugly thing on top Way more has to etc. So everybody uses lighter. There's many reasons why you want to use lighter Essentially it works during the night So the way in which it works is it has, you know, multiple beams of lasers that go and then Do like a cycle like a turn they turn very quickly and then they project Laser beam it reflects and they measure two things. They measure distance and And they also measure the amount of intensity So they're they are not able to see colors. They're actually in infrared But you know the different reflections different materials have different reflections. So this is so they're also the good way in which A good characteristic of lighter is that of course because it's emissive. It's not reflective In the sense that we know we don't we do not depend on the Sun it works during the night and also give you natively 3d representation of points. So what I'm showing in the In the in this video is like the top view of the lighter But in reality you have like a 3d points You have the x y and c coordinates of each and every point that you get plus So you have the position and the intensity That's what you have. So this competition is really there's only I don't speak Chinese But hopefully there are formulas which are universal. So they ask you to Classify each and every point. So this is for you for those of you who know about deep learning It's called a segmentation problem So for every point they ask you to classify them in one of seven categories like pedestrians cars Motorcycles crowds, you know vans, etc. Seven categories and then they Wrang you with an objective score. So this is what I read like about competition It is not, you know, how elegant the solution sees, you know, based on some sort of subjective Judge, this is like a metric and if you do well, then you win. That's it Then the other obviously Alibaba is interested in this for commercial reasons So in this competition, so you have to get, you know, as As, you know, the best you can get with this score plus you need to be able to process this thing in real time And by real time they mean you have to process each frame each cycle of the lighter in In a less than 100 milliseconds with, you know, 1080 GPU and this I7 CPU So that's what they ask you and Also the training data that they give you is heavily unbalanced. What this means is that you may have lots of let's say Actually, the most abundant class is Really the floor and nothing like, you know, just the environment and then there's a less abundant classes like Like vans, for example, there was this abundant in China. There's lots of motorcycles But there are there are heavy heavily in balance, which, you know, has some challenges for deep learning. So So that's what we are asked to do. Now the first thing if you've ever Faced You know deep learning problem the first or even a machine learning problem Even more than a deep learning one Is to know, you know, what type of inputs you get now now here. The challenge is that the LiDAR You know is this right is well actually it's not this so this is a 2d projection So we are projecting a 3d Point cloud in 2d and from from enough on the outside now the beauty with LiDAR is that you can project in 2d in Many different ways, right? So this is kind of a perspective from the outside. You could also project it You know as you were seeing it from the sensor You can project it from the top you can put it from the side Okay, but the important thing is that each LiDAR frame a frame means like a cycle of the of the lasers has about seven, you know five 57,000 points per frame So each point is a reflection now something that is challenging is that if you are seeing this And if you've ever faced a deep learning program So the first issue is that so this is really the native format of LiDAR is just to give you points x y C coordinates plus the intensity, right? So and then there is every frame has a different number of points. So not every frame has 57,000 some frames have more points other frames have less points So now you can see this is different also the order of the points doesn't really matter Right in an image If you think about an image the the order of the pixels in an image is really important, right? Because if you shuffle the pixels you get coverage In this case because you're given an x y and c coordinate That's what really defines the actual order, right? Is the is the position is encoded in the in the point itself? So the other thing which you're given these six sequence It's irrelevant in that sense. It's not really a sequence It's really a set right a set of points that you're given So that's a first issue. So, okay, how do we deal with LiDAR point clouds? You know in this competition is not not of course not only in China, but here also in Kaggle To win these competitions or even to score in the you know one top 1% you need to do two things You need to do the first one is obvious, but it's difficult is to do everything right So whatever you do you have to do it, right? There's no room for mistakes in any of these competitions and the second thing if you really really want to be on top By top. I mean top 10 you have to do something different. So now this telling right you have to do everything right and has to be different so when I embark on this kind of challenges, I always try to Approach using my weaknesses as a strength and say, okay, what is everybody else doing? what is the state of the art in LiDAR recognition so instead of the art essentially if you score through patterns through Publications, you will see that what people do is They they take the point cloud and build Projections from the top from the sides so they build images, right? That's an image with projections so that's and then of course because In deep learning I would say that image recognition is really the most advanced Application of deep learning image classification is very mature Image documentation is also very mature. So if you are able to move your problem from a 3d point set, which is difficult To an image problem and you know, you know, you know, you are getting closest to something workable at least But you know, I said, okay I cannot do the same as Other people who've been working on this problem for years are going to do because you know, they're going to Really beat me on this issue so Okay, so that's easy. So I started like exploring what is really really the sensor Computing if you think about what the sensor does so this is kind of how it works. So there is like in this case, there's 32 Lasers so one laser is it's fixed like pointing in this angle And it's one is pointing in this so with with fixed lasers that it's they're spinning, right? They're spinning They were to capture the environment. So The estimate is is going, you know from, you know, that's a full cycle So from zero to 360 or from minus 180 to 180 So it's you know goes in this angle and there's one laser for each Synod angle for each like, you know, the kind of the elevation angle So that's how it works if we take the a frame and in this competition We were given frames as a CSV file So each line of the of a frame has like the three coordinates x y and z in meters and then intensity So if we convert the x y and z coordinates into polar coordinates in 3d So you have distance and two angles And we plot the the estimate angle Of course, you know, it's you know something we expect right it's spinning So it's going in this case it goes from From zero and then it goes all the way down to minus 180 Then it you know it flips goes to 180 and goes back would you really would expect now You will see that sometimes it goes to zero So you see that is it's a weird thing that that is for the most part is linear But sometimes it goes to zero. So, you know, but but we'll see what's going on later The important thing is that at least we are it looks that we are given the data order in the same way in which the sensor Capture the data. That's what really is of interest to us at this point so if we plot so this is the if we plot for The the the just the estimate angle, right in the same order as is being read from the sensor and if we plot it Into a let's say an image like I'm in we play into a matrix. So each line It's each horizontal line is one laser. So the laser. Let's say that points to the sky is the top Line and the sensor that points to the to the floor is the bottom line Then and then we plot in color the estimate angle. So you see that it goes from some purplish Color it goes to dark blue and then it goes to yellow which is 180 then it goes back to the same color. So it is this is just visualizing in a different way the You know the 1d representation. So we are just stuck in in a 2d kind of Matrix and visualizing this matrix Well, and we see something so we see it's kind of expected now. We see something weird, right? So we see this in the center, which is like They're like kind of shuffled So what this what could this be? so if we zoom in and then do again the same 1d representation so we this is this pattern so this the so the sensors each sensor goes down But they're kind of shuffled I don't know how you guys remember in Spain. We had, you know, many years ago. This is from maybe in the 90s We have Pay TV like cannot plus right and it was scrambled, right? So you have to have your decoder to be able to see it and back then I was in the university and With a few friends we build a decoder for this. So, you know, we kind of cracked The the algorithm and the way in which we cracked the algorithm was not using cryptography Although that was one of my passions back then was really by exploiting Correlation between the images and because a natural image sensor correlation you could basically if if people are just resurfing the image you could Find the Reordering that maximizes the correlation between lines. So it's very simple to this so this What I you know what we did which is find the Shaffelin angle. I don't think that this was intentional and this is not really an encryption But the same principle applies I think it's just that the angles the wing which are mounted they the way they wait you with in with the Lasers are mounted. They are sifted because they have to be close to each other. So they may need to have You know some you know arrangement so that they fit in a small space So it's very, you know, you do a few of a few correlations and you can find out The reordering that makes it that maximizes correlation again, and you do some fancy non pie Shaffelin which is very simple and then, you know, you can rearrange, right? So this is now if if I basically shift every line in the right way, you know, we get this now so we can see that is very gradual and you know every The angle the azimuth angle between one lighter and the other lighter You know, it's almost the same. So, you know, we've arranged this in a nice way now The question is so this is the original one as we are given it from the In the competition from the sensor and this is the rear the one now here in this case I'm plotting the distance, right? So I have it realign and what I'm plotting here is the distance now You can see that in the color so that the yellow is closer to you of course the floor, you know Is closer to you and as as you know, you are measuring the distance in you know in the in the sky So you see darker blue so it's farther away, right? And there are some blocks that you can see there are some blocks There you see those those things are like at least three and partially one of them occluded on the side That are closer or farther And this is the ground truth. So this is really so it's now is a much much easier way of You know dealing with this problem. So in the top image you have the rearrange reshuffled lighter Measurements in a dense almost dense matrix. There's holes. We'll see what those holes really mean and on the bottom We see the ground truth and you can almost tell visually that there's you know that there's high Even obvious correlation, which is great things even easier to see Lighter in this way than to see it in these very sparse, you know point clouds that are really really sparse So this is a super dense matrix. So Now what are those? black points so the black points are really where the measure the the sensor measured almost zero and It may be because you know, you are spinning the lighter and there's like nothing, right? And then the sensor may time out and then it speeds a zero, right? That's that's what we think and then So basically we took every point that had a very small distance and we put it in In a some dark Point and then we'll see how we handle this in the deep learning model now because it's Equivalent as if you have some image with noise like with points that you don't know really what they what they are But there's all solution that you can that you can come up with now if instead of putting the Asimod which we already did we plot the synid angle Now there's something weird going on We would expect that because we have a discrete amount of lasers. We have 32 lasers Why you know why so we see those spikes which look cool There's there's a more lasers closer to zero because typically there are more objects, you know, they're you know in your line of view But then why you know, we we see this like almost noisy Synid measurements and if you you know so so if you look at this image There's something funny going on in this image. So you can see that There's a discontinuity Right there, right? So what could this be right because obviously there's no It is not, you know a glitch in the matrix like in the movie. There's something going on There must be a physical reason why There's a discontinuity and the reason is because the car is moving, right? So if you You know if you pretend that you slow time and then you see the car moving So the rotational speed of the lighter is in comparison, you know Relative to the car moving. So when you start measuring, you know, distance you're moving And then when you almost going back the car is in a different place So they are changing in this way the frame of reference, right and the lighter is not accounting for that and actually you can see that The it's fine the way the way in which with a lighter with this lighter at least you can measure the movement is because The top of the car which is static relative to the car is the only thing that is static right at the top of the car Versus the environment is encoded so we can derive movement of the car Using the top and then you can re-reference the whole thing with the top of the car and when you re-reference You change your your reference system to to one point and we just starting taking measurements Then boom you get discreet the discreet angles which were what we were looking in the first place So So this looks pretty good. So at this point, what do we have? So we have so if we reshuffle the stuff so we have the distance You know and we can we can already see stuff, right? So this is really you really have to think about you have this in in this goes like a circle Right, so though we are seeing just the front. This is from minus pi to pi For minus 180 to 180 You have the intensity and then you have also, you know optionally You have the SM or angle if we if we think that there could be some correlation because Let's say that typically right cars are in your line of sight. There's hopefully remember these things work on correlations and You know, we can assume that there's more cars You know, maybe on this angle and on this one that on the sides, maybe on crossroads You may seek out from the side but not so often same with pedestrians, right?  So but you know, we'll see what we do But at least now so now what we've done so far is we've come with a completely different representation of the data data that if you've worked with a Three with debilitating models this almost looks now like a very very easy problem now hold on because The issue here is you have to enter all these things that we've Mentioned so we are we add the four things the distance intensity as a mode and then The we'll see why we enter the the ring coordinates and also the fake points at the end You have to get a semantic segmentation Label of every valid 3d point, right? So this looks, you know, very very similar So we use the workhorse Workhorse of segmentation problems in the learning which is called unit So unit is a very popular architecture for segmentation problems now the issue with that unit Like with the canonical unit is that there's a few issues one is that it does the The max pooling if you are familiar with the unit You see that the convolutions that they come up of convolutions and then you gradually reduce them And you get like a latent space at the bottom and then you reconstruct again with upscaling Convolutions or interpolations so you can use the convolutions or transpose convolutions Or interpolation is doesn't really matter in this case too much But if you think about it in a regular unit in a unit you do it in the two axes right in X and Y now In our case, this is tricky because yeah, the X axis looks that we can squeeze right, but the Y axis is really the lighter Lighter angles so there's no really in the same way in which in an image We have like a trans translational invariance in an image in this case. We have only Translational invariance in one direction in the in the X direction in the estimate directions. We do not have Translational invariance in the scenic direction because this is like a different This is a different Laser and a car doesn't look the same here than in the very far not only it looks a smaller in the in the You know when it's far away, but it's also typically it's been shown in different lighter lasers so But anyway, this is the architecture based on a unit. It has roughly 10 million parameters with a special dropout to increase generalization and We did some ablation analysis that basically ablation is a very boring task of you know Running the same experiment taking out stuff, right? I'm measuring the relative impact improvement or worsening of your performance if you take give or take Some characteristics and we found that when we You know when we you know added or removed the estimate angle it really this didn't change too much So not you know not surprising, but anyway, we also tried many different things in these competitions You know it's not that you have of course You have to have a good idea and be able to implement it That is critical that you are able to try experiments really really really fast Those of you who've done deep learning should know that deep learning is really a very empiric discipline Right, although you see equations and everything looks like Fixing stone is really not the case and you need to run experiments and see really you know You may have like an intuition of what make work better than other things but it's really about having a pipeline that you can test very very quickly and The speed is critical on these competitions Models staying long to train so you have to balance between you know complexity of your architecture number of experiments you run etc That's critical. I cannot emphasize this enough. So Now there's a few things the most important things what I already mentioned that there's Translational invariance in this direction, but there's no translation and invariance in this direction Right, there's two ways you can deal with this problem the most elegant way But we didn't do because again, this is finding a shortcut to do quickly is to build any type of convolution that has shared ways in what direction but not in the other that is in in pytorch This is very very well. Well, no, it's not very easy, but it's doable relatively easily Because they give you some low-level primitive to build your own convolutions in TensorFlow It's a bit more convoluted so We behind this competition would use TensorFlow and Keras So we basically did a shortcut which is super quick, which is just adding an extra channel That basically tells the a regular convolution architecture Where it is in the image in the image, which is really a dense matrix, right? so so if you think about in an image in a in a convolution the What a neuron at the end sees it doesn't really know where it is Where right because it has shared ways all across the image But in this case it's critical to know where it is in which at least in which laser It is and the way in which we do it is we just add a new channel which just telling In every line the in a number meters my between minus one and one something that basically tells the Convolution weights, you know where they are and they can learn a combination of this number plus the actual contents of the image So it can learn from the positionals location so and then so this is You know one one trick Actually uber did that pretty good paper on this trick and then we also we have this issue with with Images that are different, you know, remember I said the different different number of points So it looks they have different number of width in this So there's two ways you can basically do one net that in one shot give you all the predictions or we'll see why later You can split the the the whole frame into, you know Small segments with some overlap run in Rantating or inference in all of them and then reconstruct and then you get with this you can do TTA which is text and test time augmentation and you can do kind of a You know runtime ensemble and and you know incredibly the secretary is super fast and we were able to do Your augmentation in real time in real time and we still being within the limits of the competition requirements So How we optimize them all nothing really fancy Really, I think the we just did a validation split of 10% of the of the images We have to regularize these with a dropout and then we did flips and some small alignment from Produration some out in Gaussian noise and then for a 4d optimizer. We didn't do anything fancy Just Adam. That's it. Maybe today. We will use one cycle which is you know more You know, it's newer. It's the cool new on the Block and then I think the important thing is here For the loss function we use Remember this Intersection of our union metric that we are given is not differentiable So we cannot use it directly as the loss function. There are two ways. We use a very new Loss function called love us which is very sophisticated mathematical function that is a good proxy of I O you but surprisingly didn't Didn't really get as much better performance and it was much slower So we did a mix of cross entropy class a softened version of the I O you which you know It's very easy to basically soft, you know make a Soft version of the hard I O you and then also the same with soft dice coefficient So there's actually there's a loss called it to work Chiversky loss got something that is named after a Russian guy. The math are good, right? That's guaranteed So this is a traverse key loss. It's a generalization of the F1 F2 I O you and dice And then you can with this formula you can basically soften it and Do closed up with it and then for those fake points that we see we basically Void those function. So then that's how we take care of those like fake points. They are, you know, kind of a nightmare And then we train for 100 epochs, but not much Looks a lot, but it was relatively quick to train Maybe it took one day, but much for the whole thing Which you know incredibly it's given where we are today with the learning that some models takes weeks this pretty good and then In terms of performance so this competition what this competition was crazy because we had like a different We didn't even know the rules Because they were in Chinese and they kept changing everything but but in any in every part of the A stage of the competition, which we really didn't understand we are scored On top and significantly better than the rest of the teams I think this is why because what I said we something very very difficult and very Native to the sensor without losing an inch it's early an inch of accuracy of the sensor so the winning submission had a runtime inference of 65 milliseconds well below the limit of 100 so and this is with four TTI overlaps So pretty good. Even we tested with no overlaps and we will still have one with just 20 milliseconds So this means that these guys could have better lighters and the deep learning model will not be the bottleneck actually funny because once we were in the actual competition the Alibaba guys Told us that they tried something similar, but they couldn't make it work So again, so what I said is you have to do something different but do something right I think they made a mistake somewhere and then they thought this wouldn't work But it did if you do everything, you know carefully then You know what once you train a model and you want to optimize it for inference You know because we were it was very very fast already the only thing we did it says, you know trick that is very easy You know that you to train deep learning architectures typically you use batch normalization Right to help you with you know the vanishing gradients because of the stats of the activations now if you think about what is really batch normalization is really Everything in deep learning is multiplications and additions everything right?  It is just multiplications and additions So what is a convolution? It's the same right is a multiplication and addition so you can actually once the model is trained you can actually mix the two of them you can Get rid of the batch normalization Those those additions and so take the bias and weights and then shift them to the convolutional weights very easy and then you remove many layers performance and And that's it. That's I would say that the summary is that we created a very novel representation of a LiDAR sensor and then we Of course once we have this in a dense like Main tricks we could leverage existing segmentation architectures with a few caveats and tricks and then in every part of the Competition we scored much better and to our surprise not only we want the objective price of the competition But then the organizers so this happened to be so this I didn't know when when I joined I thought that this was just an online competition and because everything was in Chinese This happened to be one of the biggest like a conference about did learning in China with lots of people you know in China physically in a in a in a city in China and we got the Also an extra price for the most innovative Solution of the whole you know competition We have many different tracks for other tasks and that's it One two and that is well It's fascinating that you beat so many people without really knowing what the hell was going on. It's a very very impressive Does anyone have any question? I did you say what you won what the prize was or is it secret? Well, it's money. Yeah, okay Chinese citizenship or no I'm not sure if I wanted this price You don't have to reveal the money. No, well, it's it's it's public in the competition rules So was money in in June? Uh-huh. Yeah, good exchange rate Okay, any questions out there for under this about this this experience of Incredible story Okay, I don't see any. Oh, here we go. Here we go. Is that Jesse up there? I see maybe maybe not. No, it's not We have a question here in the middle Fatima is hitting towards you. Keep your hand up Hello I'm I guess that you didn't Got to this solution at the first time. So I would like to know How much time did you spend to achieve? all of this So it's funny. I always get this question like how much time you get so Actually not much coding. So the way they typically goes for me is on this so I've done many competitions And typically like I'm giving the problem and then I don't do anything. I spent like a few days Thinking about it. And then I guess the moment in which I knew that We had a great representation Maybe only took one day once, you know, I did the exploration because I have a background in electronics So I was I guess like the hidden secret I was going to to I was seeking for and actually found was how would I Send the data of the LiDAR sensor if I was a firmware engineer Right. I thought he myself how I would do it In packets, etc. And then try to then but this could have not worked right, but it did in the first try so that's that's the I guess the The answer which may not be very You know, maybe a bit surprising, but I guess we were a bit. We had a bit of luck in that regard Awesome. Thank you Any other questions out there? Here we have one right up gonna make you work. Fatima. I have to come down to the front now Just if you keep your hand held up so Fatima can see you Just down here in the front So I would like to ask you because I know that you have been engaged in many other competitions since this one And I know that you have evolved a lot Even if this is fascinating the way that you have envisioned the representation of data to be embedded in the in the model Will you now that you with all the experience that you have with other competitions What would you have changed if you will approach this sort of this problem now in the data representation? This is one question and the other question in model architecture. Would you have changed something? Well, I guess the So the answer is in the data representation I don't think I would change anything and I also made a recommendation to the hardware community to send the data In this way because there is very very complex Substandard in my view architectures to deal with point clouds, right? And a point cloud is really like a set of points in in let's say pretty good space now But a lighter although easy it is a point cloud It has less entropy that you know many arbitrary point clouds for example It's impossible to have a point if you're shooting in this direction It's impossible physically to have a point here and a point, you know beneath the point right because it reflects Right, so there is entropy. So there's a you know precondition, right? This data representation is the native data representation So I guess Maybe if this was like an image competition like a pure image Then I will give you a better answer although people in deep learning architecture. We used to feed RGB channels You could also feed In JPEGs for example, there's luminance and chrominance and there's a sample. Why don't you feed? Luminance and chrominance, but in this letter, I wouldn't deal with that now What would I be doing differently, you know if I wanted to further improve this one will be the validation So the validation we did a random percentage. We did a quick test To because we were given the the images like shuffled like the images the frame shuffled, but we were able to using correlations Reordered them and build videos of The cars and so of course if you do this then you can build a better validation model that Basically doesn't have any overlap with other videos, right? So this this this will be one difference and this is for architecture. There's many new things coming in segmentation Architectures, there's a pyramid network, etc. So, you know, I will use, you know, the coolest new thing Which there are few they give you tiny percentages So with that alone, you may not be able to win now. We will improve a bit but not winning that's with that Okay, thank you very much. Well, thank you very much. It's a fascinating story and thank you to Andres and Congratulations once again. Thank you


Good morning, John. Every time people get frustrated with YouTube, I get a lot of messages saying, Hank, you make stuff, why don't you start a whole new competitor thing to YouTube? And ultimately, I feel like this comes down to a pretty central question when you're upset with a situation in your life. Whether this is about your government or a relationship or your online platform, do you revolt or do you reform?  If you want to revolt against your government, you probably have to take down your government because you can't just leave.  But on the internet, if you want to revolt, you could just go somewhere else, right? Maybe not. Here are six reasons why I have no interest in leaving YouTube. One, it's actually quite good. Like the video player works pretty much every time. It transfers really seamlessly between my phone and my computer and my TV. And as a creator, I get to benefit from a gigantic audience and also the ad sales team of the second largest corporation in the world. Two, and I thought a lot about this one, ultimately my problems with YouTube are my problems, not my audience's problems. You people. And while I appreciate people who say that they will go to the end of the world for their faves, and I believe you, you are also my fave, and I want to make a good experience for you, and to me that is the experience that you're comfortable with and that you've had for the last 11 years. Three, YouTube is really good at getting people to watch videos. That's what that somewhat insidious algorithm is designed to do. And while I have some philosophical problems with a company training neural networks to hack the human brain into watching more content, I recognize that every social media platform is doing that, and the ones that don't are gonna end up losing out to the other ones. Which is the situation we're in right now, so buckle up for that over the next 10 years. And on that same train, number four, I want new people to find my content, so I have to put my content in a place where new people are going to find it. And YouTube shows my videos to new people. Hi, new person, if you're one of them. Here's the big one, number five. YouTube's problems aren't really YouTube's problems. They are the problems that any sufficiently big social video platform would have. You have to get advertisers bought in, you have to monetize for creators, and you have to be open enough that there's so much content being uploaded to your platform that there's no way you can review it all. The moment a new platform got big enough to matter, it would have all of the exact same problems that YouTube currently has. And I don't actually know how to solve those problems. And finally, number six. I'm sentimentally attached, just like I'm not going to leave my country when things start to go bad. I just won't abandon this platform to leave it to become a worse place without me. When we criticize stuff, we don't actually have to imagine a solution that works. We just have to imagine a solution that works for us. But communities in real life or online are more complicated than that. They have tons of different stakeholders with different needs and different wants. When you get further into the details, you find that it's complicated. And I'm not saying give in, give up, we're stuck. Obviously not. When you love a place, you should want to work to make it better. Imagining that you can just run away from all the problems into some new place where no problems exist is a self-indulgent fantasy. I care about this a lot. I'm willing to fight for a better YouTube. I know a lot of other people who are too. I think that's how you make a better YouTube, not by starting it from scratch, but by caring enough about a place to work for it. Hashtag the studio is also about America. If you have ideas for how to make YouTube better, please leave comments. Ideas are born out of people. And if you want to talk about this stuff in the real world, VidCon Europe this year is focused entirely on creator and industry stuff. So these kinds of conversations.  But it's March 22nd to 24th. You could find tickets at VidConEurope.com. And also, if you don't live in Amsterdam, you get to go to Amsterdam. It's an opportunity to get together with other creators of all sizes. Talk about craft and how you make it work and what you like to do. And also talk to brands and platforms and the kinds of people who enable people who want to make stuff. John, I'll see you on Tuesday.


And I'm an Asian, I'm a shot over just like George C. Baderick's is a Corvik. I'm a New Yorker and Nick Felstein is going toe-to-fingering cheesy content. Thanks everyone for watching. Also, you can subscribe to like button and thank you if you already supported on Patreon. You beautiful. There's one good thing about New Jersey. I have a really beautiful friend. Michael Pomerica who lives there and it's where the Sopranos comes from. But other than that, I've never been. Today, what are we doing, Rick? We are gonna do something we've never done before on the channel. Oh, wow. We are going to show signs of intelligence. No, I mean we need to be on brand. Okay. So. Today we got a video, this is a Sachin Tenduker's tour. My secret 3AM ritual before matches. Ooh. What do you think it is? Sachin takes this person on a personal tour guide of the destinations that had a profound impact on him. I wanna know what his secret 3AM ritual before matches. Masturbation. That's what I'm hoping. But other than that. Just like Wolf of Wall Street, twice a day. I mean, just got it at least twice a day. It was a rookie number. It was before lunch should happen. Here we go. My dream was to pay for Indian. I would come here while drinking water. I would always look at the idol and pray. So I continued doing this throughout my career till the last game of my life. You're still alive, man. This is the colony that I grew up in. Between these two buildings, we had to hit the ball to the main gate to score a boundary or a six. If the ball hit over the first floor, you were out. And these were the windowpills there. We managed to get the ball there and damage quite a few windowpills. Do you remember different windows you got? Yes. And that's why all the windows now have cages over them. Sick of all the, what a great story. Cricket destroying windows. Sachin, when he was a kid, broke my window. I would come down to this playground and spend hours on it. And this is where I lived on the fourth floor. My food, snacks, everything would come down.  I wouldn't go up. I was not interested in eating. Just wanted to play. Yeah. Yeah, as much as possible. I understand you guys would pull Franks on one another as well around here, right? All the time you would be doing that. We, me and my friends, we were notorious kids. So we would sometimes flacken the tires and all those kind of things, which I shouldn't be saying on camera. I was a... What a great picture. Really, really naughty. This is the spot where I spent four years in my school days. My uncle and my aunt stay here. This is that place where I would bat. And I would make my aunt sit there on a stool. And I had actually a golf ball, which was slightly out of shape. It was almost like an egg. And I would play with soft hands so that I did not damage the house. The other thing that I also did was... There is a cricket ball inside. This is a normal sock. I would put a cricket ball inside and literally here on the ceiling, I would hang the ball like this and keep practicing. So I would do that hard on him, comfortably five, six hundred times a day. I could be, let's say, filming an episode with Tom Hanks, as an example. And you would get a hello maybe some looks, but people would generally let us kind of do our thing. It's very different here with celebrity of your stature. How would you explain that? It's different. I mean, I didn't get the warmth of Indian people. And the emotion. I've always felt that I value that. There is not much privacy, but it's a package D. That I wouldn't want to settle for anything else. So, you know. In 12th grade, you had cops outside of your classroom when you were taking your exam. I was. Right? I mean, you've had it. Yes, but by then I had already played for India. I played for India after my 10th grade. So my life had changed. Wow. He wasn't a normal 16-year-old boy. Seriously, he was already playing for India at 16. That's crazy. Much faster than what a normal 16-year-old would. I do find it crazy that the host is not driving. Yeah. That is for music. When I'm listening to music. He probably just deferred to whatever he wanted to do. He started a phone call or a message deep like this. That's how India is though. He's the guest. Just one phone only for music. Okay. That's when you know you really like your music. I love my music. So this is Shivaji Pakham. I spent five years of my life here. Do they know he's coming? Did you imagine? It's a coincidence that today in 1989, 32 years ago was my debut. So it's a coincidence that we are on the same very day we are here where I played the first ball of the season ball. So I had to bat in all these nets. In the last net, my coach used to keep a rupee coin on top of his stump. Right. And you see the number of children here practicing. You can see at least 200 fielders. Anyone caught the ball, I was out. There was no discussion on that. So there were a few points that I lost but more that I won. What is it about those coins that are so personally significant to you because you've kept them? It was like a medal for me during a practice session. Our coach would allow us to have a glass of water just once. Oh, God. It was three hours. To be a stamina. The choice was ours at the beginning of the practice.  We were allowed to have one water. That's how football coaches used to be. Yeah.  Yeah. No, we're human. This is where the tap was. People need water. So I will be drinking water from this tap. And there was this feeling within me that that belief was there that this water energized me. It had some power to help me achieve my dream. My dream was to play for India. And I would come here while drinking water. I would always look at the idol and pray. I continued doing this throughout my career till the last game of my life, which was in 2013. Sometimes odd hours, like 4 a.m., 3 a.m., when nobody was there and I could actually come here and offer my prayers in peace. I would do this before every tour of mine, every international game. I would come here before I left home. So I became richer. The practice session was over. The tiring day was over. This is where we would kind of sit together and fool around. The pockets were not deep at all. So we would kind of share everything that we got. What got you so into food? Because I know you're pretty passionate about it. My mother was a very good cook. So whenever I was traveling with my friends, I would take recipes from her. And then I would help register those things. Those were the days where no more boys were around. So you had to keep everything in this computer. What is your go-to dishes? I would say my bangun bhartha. Okay. How would you? And what about if Anjuli is cooking? Anjuli? No, Anjuli doesn't cook. No? She eats. So you're the cook in the house? I'm the cook in the house. Anjuli, she's not too fond of cooking. Not a great video. Yeah. It's really interesting to see more about him. Yeah. It's so funny because it's such a universal thing. Back in probably your day, honestly, it was slightly, I guess, when I was a kid, but it kind of changed around my generation time. Just coaches just didn't want you to drink water. Yeah. Because they thought if you needed water, you were weak. You got water when practice was over. And it's won. Yeah. So bad for you as an athlete. No, I remember. I can tell you every time whether it was high school ball, college ball, playing softball with guys on Saturday morning with my dad and Rich, or just going and doing stickball with my dad. We always had gum. We always had sunflower seeds. We never had anything to drink. Never had anything to drink. We would get something to drink when we were done. That's crazy because it's so bad for your body. I know. You do not have water. People hydrate all the time. Because there's science behind it now. This was pre-gatorade days. Yeah.   But then people would make fun of you. Oh, you need water? Yeah. It really was. Oh, you thirsty?  It is. Yes. I'm a human being. Oh, you need to breathe. Oh, you need food. You need something. Oh, you need to sleep tonight.  I swear.  I would love that. And if that's the case, also go over to Graham's channel and see anything else that is on that channel as well. Yeah. I'm surprised he was able to talk with all those cricketers. I know. Maybe they were all told he was coming. Or they had just a big entourage around and protecting him from telling people, please back away while we're working right now. So, yeah. Have such and just show while you're playing cricket. It was a really nice, I love the personal aspect of it. It really felt like we were just getting to see these two. There was no one there on the street. That's what the greats do as well. My brother and cousin played soccer longer than I did my youngest brother. And I hear about the great soccer players or the great basketball. They always had a ball with them. Yeah. They would carry the ball around with them in the house. And my cousin, especially, he was great at it. He would never let the ball touch the ground some days. He would just, he would juggle it all day long. It was actually insane how talented they were. But like that's what Michael Jordan probably, you know, just all day long after free throw after, you know, not baking his high school. No, there's people who do things I remember because I wanted to get better at catching fly balls. I would have my dad. I'll give you some fly balls. He would, because we live by a golf course, I'd have him or I'd have Rich. They would tee off with the golf ball. I'll tee you off on it. And I figured if I can, if I can track and follow a little golf ball twice as far as a baseball, that'll be good tracking a, tracking a baseball. And sleeping with the baseball glove is a standard thing, just having the smell of the baseball glove right with you.  In that movie. He can track a falcon on a cloudy day. He can find you. Night's tail. Nope. No? Mm-hmm. I don't know. Uh, how did it do? No, he, um, really. Princess bride? Yes. He can track a falcon on a cloudy day? Mm-hmm. Who says it? Uh, Buttercup. When, uh, Wesley is still in the mask. And he, she thinks she's captured her. Oh. And Humperdink is trying to be chronic. Okay. And so she's trying to intimidate it. He says, yeah. He can track a falcon on a cloudy day. He can find you. That's a random line. That's not a more common line from Princess Bride. Every line from Princess Bride is iconic. No, there's some that are far more iconic. No. Yes. No. The Cliffs of Insanity. It's probably one of the most quotable movies of all time. Anybody want a peanut? It's one of the most quotable movies of all time. If you've never seen the Princess Bride, do yourself a favor. I'd love to actually see somebody that didn't grow up with it, like, because obviously Nostalgia plays a big part. Oh, yeah. A lot of people saw it when they were really young. The R.U.S.'s. Yeah, R.U.S.'s. I don't think they exist. Anyway, uh, fantastic video. You heard of Socrates? Plato, Aristotle, Socrates? Morons. Morons. You really are that smart. I could quote that entire movie. And, uh, let us know what other videos we can react to down below.


To welcome to the annual DEF CON convention, this meeting was held at exciting Las Vegas, Nevada from July 9th to the 11th, 1999. This is video tape number 14. Some of you, I'm sure, are here to find out what not to do in order to not get caught. Some of you are here to find out what to do to catch the ones who want to find out what to do to not get caught. You may take from this what you wish. We're going to talk about this evening and then I want to thank you for all coming. I realize the bands have started. We're going to talk about cyber forensic analysis and there are all types of forensic analysis. I make some distinctions. Forensic computer analysis deals with specifically the removal of information from hard drives or floppy drives, floppy disks for that matter. In such a way that the information can be used as evidence in a court. I'm as caught up in hype as the next guy so I had to figure out a way to use the term cyber. I use the term cyber to refer to things that have to do with networks. Cyber forensic analysis then is the forensic analysis of information on a network that we can use then as evidence in a court of law. Then we have one other kind of forensic analysis and that's software forensic analysis which we won't be discussing tonight because it's black magic. And that is the forensic analysis of code to determine who wrote it. Alright, change. Thank you. First of all let's define what we're talking about here. We're talking about crimes against a computer that's crimes where the computer is the victim. And the FBI tells us that they define that as computer crime. They say that everything else is in fact traditional crime and it just happens to be done with a computer. So we refer to that as computer related crime. And that includes crimes where the computer contains the evidence and crimes where the computer is used to commit the crime. Now let me put everything we're going to talk about in a little bit of perspective. Probably 90% of the investigations that I'm involved in are corporate investigations. They haven't got a thing to do with some hacker or cracker or a thing that goes bump in the night breaking into my clients from the internet. Probably 80% to 90% of the investigations I do have to do with inside people who are doing something naughty within the organization. And so there are a number of things that I can do in the course of an investigation that are much more difficult to do if we have to do them over an internet network like the public internet. However, the techniques are not dissimilar. With that in mind, let's simplify things greatly. Let's say that there are only four kinds of attacks. There are denial of service attacks, social engineering attacks, technical attacks, which is where I put the whole gamut of exploits. Whole gamut of attacks that exploit bugs in a system. And then we have sniffing. These are the kinds of attacks that an investigator needs to be prepared to deal with. Yes, sir. Sniffing is passive. But one of the things that an intruder wants to do, probably the most important thing that an intruder can do, is to gather information that will let that intruder enter the system without triggering an alarm. Therefore, the intruder wants to see if he or she can get passwords and accounts. And that becomes part of a total attack and therefore we're concerned with passive sniffing. Furthermore, one of my favorite attacks when I'm doing vulnerability testing for a client is to compromise a machine on the ISP, put a sniffer there and watch. But nobody here has ever done anything like that. So yes, sniffing is passive. But sniffing is an important part of any attack. So, if we're investigators, put ourselves in the position of the investigator. And if we're investigators, we need to deal with the attack as the attacker dishes it out to us. Well, the first thing any good attacker is going to do is gain information about the target. And they do stuff like internet searches, who is, NS, lookup, dig. All sorts of those little things, which everybody does, doesn't break any laws. Isn't there more unethical or legal of fattening? But they do it. And we can't tell as investigators. The only way we're ever going to know that the attacker did these things is if we're able to eventually do a forensic analysis of the attacker's drive, the attacker's disk. And then we may see evidence there. And then again, we may not. The next thing the attacker may try to do is do some preliminary probing. Now, they want to get passwords. They want to get information. But they don't want to trigger any alarms at this point because it's too early in the game to get caught. And so they'll do things like another one of my favorites is a pop password attack. Just run a little attack against the popper, do some password guessing. It doesn't usually log. Maybe a DNS zone transfer just to sort of figure out what's out there mazes me how many ISPs leave their name servers unprotected. Maybe do some simple mail probes, other simple probes, and these probes may or may not log. And on a big system, they're going to log in such a way, probably, that they're going to be very hard to find. So forensically, as investigators, we're going to need some tools that are going to be able to parse these big logs. Anybody ever seen a firewall log that's been running for about three days on a big company? Yeah, you're going to need a tool to parse those logs. And then, of course, some attackers will get frisky and start looking for backdoors and do something like a ping scan. That ought to show up in a log somewhere if it doesn't, your logging's not set up right. And then we're going to try some technical attacks. We may try a little social engineering. All of these things are things that the investigator needs to look for. And let me tell you, folks, there is no such thing, and I've been doing this for a very long time. I have never, never seen a smoking gun in a forensic dump of a disk. Never found the big clue. It's dozens upon dozens of little clues that you put together, and they're buried inside of these big logs. Next thing that's going to happen if the intruder is gridded what he or she does and does in fact penetrate the system is they're going to try to clean up. They're going to try to delete their tools and work files. They're going to try to modify logs. Here are some UNIX examples of logs that they might try to modify. And we're concerned then about what's left on the computer. In a few minutes, I'm going to tell you a little bit about criminal profiling and how we use it to investigate a security incident. Now if you do like I do and you deal in a corporate environment, the people who are in that corporate environment have an expectation of privacy. And if there are no policies to the contrary, you're not going to be able to investigate this incident from the inside. Because we've got a law called the Electronic Communication Privacy Act that says that without a policy the only thing we can look at is source and destination address. We can't look at content. Well, that may make it very difficult to investigate an incident. So we need policies. So those of you who are here from the corporate side, go back and tell your people we need policies. Policies define the actions you can take and they have to be clear and simple to understand. And the employee needs to acknowledge them. Usually in writing. Now there's been a trend lately towards putting little snippets of privacy policies in login banners. And that seems to work. However, here's a caveat for you. And the caveat is that we're dealing in murky waters legally here. There isn't a lot of case law. It's not well understood. It depends upon what part of the country you're in. And the courts may or may not interpret those login banners as sufficient. So from the corporate side, see your lawyers.  Yes, sir. Yes. The question is, are there not provisions in the law that allow for troubleshooting and for monitoring? You may monitor for the normal course of business. It's called the business exception. And in the normal course of business, you may monitor for maintenance for those types of things. But what you use as evidence is strictly limited based on the laws. Now, if you have a policy, and again, corporate people, if your policy says that the employee has no expectation of privacy and they consent to monitoring, then you're okay. You're okay. Now, here's the Electronic Communications Privacy Act. This is the law that enables you as a corporate investigator to investigate. And for those of you who may put yourselves in a position to be investigated, this law is one that you really need to know about. First of all, it says that the owner of a system can intercept communications between an intruder and the owner's system. That means that if you come in from the outside, the victim can monitor you. The next one deals with inside. You can intercept with prior consent of the user, and that's where the policy that's been acknowledged by the employee comes in. Once the employer acknowledges that policy, that constitutes prior consent. You can always intercept the portion of communications necessary to determine origin and destination, and you can intercept where necessary to protect your rights or property. That's murky. So ask your lawyers. I had somebody when I gave this talk over at Black Hat, I had somebody say, well, that covers everything. And my answer to that is maybe yes, maybe no. It depends on how the court, in the particular area where you're located, interprets it. So ask your lawyer. Okay, now we've got that tool. Let's figure out what it is we want to do when we do an investigation. First of all, we want to ensure that all applicable logs and evidence are preserved. And I can't emphasize this too strongly. That is among the first things you do, and you preserve the evidence, not by walking up to the victim computer and typing down, copy such and such a log to another disk. Don't touch that computer. So, corporate investigators, if you walk up to the crime scene, the victim computer, and the screensaver's on, don't wiggle the mouse. Don't touch that computer. I'm going to tell you what to do with it here in a while. And those of you who may find yourself in some small amount of trouble for minor transgression, watch to see if you happen to be fortunate enough to be there during the seizure. Watch to see how that computer sees. If the cop walks up and starts typing on it, you win. He loses. Because the cardinal rule of forensic analysis is you must not, you being the investigator, must not contribute to the evidence. And a good defense attorney will know you every time if you do. Trust me, I know. This is the voice of experience talking. Okay, the next goal of our investigation is to understand how the intruder entered the system. That's the critical goal. The next one is to obtain the information we need to justify a trap and trace of the phone line if the intruder is using the phone, or to subpoena information from the ISP if they come in over the Internet. Of course, if this is an entirely internal attack, we don't have to worry about those things. Trap and trace is something that you will need to get law enforcement involved in to do. You can't just go put a trap on a phone line. You can call a phone company and ask them to do it, and they'll do it. But you'll never see the information. You'll never see the data unless you have a warrant. Yes, sir. You know, a lot of PBXs now have called for or called ID, and the answer to that is yes. No. Unless, unless you have a policy. Okay, an acknowledged policy, and that's prior consent. Yes, sir. I don't know. It's the same as this? Yeah, I don't know. I've never been involved in an investigation where we were doing that, so I can't answer the question. But if it's the same as this, I'll buy that. That makes sense to me. Okay? All right, the next thing we want to know is why the intruder has chosen the computer. This is our first little bit of criminal profiling, and I'm going to come to that more in a minute. It helps us to decide what kind of a person may be doing this, their skill levels, and it helps us narrow down our suspects. Now we need to have as much evidence of the intrusion as possible, and we're going to talk about collecting and preserving evidence here in a minute as well. We want to obtain information that will help us narrow the list of suspects, document the damage caused by the intruder. This is important because the amount of damage determines a lot of things, among them the threshold as to whether law enforcement will investigate. Another one is what the penalties are if there's a criminal prosecution. And we need to gather enough information to decide if law enforcement should be involved. Our immediate objective in any network-based intrusion is to preserve the evidence. It is extremely fragile. The evidence lives in a variety of places on the Internet. Most of those places, if not all of them, will be outside of our control. Many of those places do not log. If they do log, the logs may turn over very rapidly, and the information will be gone. So remember when you have an Internet-based attack, you need to do your back traces rapidly as you can. And if we're talking about something like harassing email or cyber-stalking or something of that nature, the data that you need to trace back that email header may disappear very rapidly. We contact system administrators on intermediate sites and request log preservation. Now, if you're dealing with private sites, generally the system administrator will give you what you want if he's got it. If you're dealing with ISPs, it's a whole different story. ISPs are terrified of being sued. And so in order to protect the rights of their subscribers and their own pocketbooks, ISPs will not give you the information you need without a subpoena. And with a subpoena, they usually cooperate very well. So what we do is we call them up and we say, look, here's the information. Please preserve the logs that contain what we want. Now, a word of caution. Again, when I gave this over at Black Hat, somebody raised their hands and said, well, you know, we've had a lot of trouble with getting logs preserved, getting response from the ISPs. We send them email and they either don't respond or we get a form letter. Two things, okay? Number one, don't send them email. You will eventually, but you're not going to do that until you've got a human being on the other end instead of a bot. Next thing you do, you always call. And I think there isn't anybody here who doesn't know how to find out who the contact is for an ISP. So you always call. Some of you have found out without particularly wanting to, I'm sure. So you always call.  You can't call them up and say, I've been hacked. Save the log. Because you'll be able to see over the phone. Yes, back here. They have every right to say that. You can't force it. You can't force it. There's no law that forces an ISP to keep logs. And in fact, there are ISPs that purposefully do not keep logs because they believe it's how they protect the privacy of their subscribers. So if they tell you that, I've tried crying. I put my wife on the phone once. Anything you can do to get them to do it because they don't have to. Yes, sir. How long does the process take with law enforcement, ISPs and so forth? First of all, I'm going to answer your question, but let me digress just a little bit so that we're all talking the same language here. Warrants and subpoenas are different. A subpoena is something that an attorney can draft. It has to be pursued to a filed lawsuit, but an attorney can draft it. A warrant is issued by a judge. Law enforcement generally does warrants. Several cases generally do subpoenas, although law enforcement can do subpoenas also. If we have to do a subpoena, it can go very quickly, and I'm going to show you how to do that here in just a couple of minutes. If we have to do a warrant, law enforcement is involved. It can go on forever. My advice is, if you're a corporate investigator, first of all, make sure you know what you're doing. That's real important because you can really screw up an investigation royally by not doing it right.  You've got a team that knows what they're doing. You've got consultants that know what they're doing. Conduct the investigation and then spoon feed it to law enforcement. You'll have a better chance of getting it investigated. So it can take a while. It can take a while. The reason, by the way, you may have trouble getting it investigated is because there's a lot more incidents than there is law enforcement to do it. And so they set thresholds. And the other thing is that it isn't really the cops that decide what to investigate in this particular case. It's the prosecutors. If the prosecutors don't believe they can prosecute the case, they won't take it. And I've had that happen many times. Yes. Yeah. And you can do that, but there is a caveat. And this is all a matter of judgment here. The caveat is that once you do that, you've essentially given the case over to law enforcement. Now, some organizations have policies about that and some don't. Myself, the type of work that I do, I prefer, frankly, to get law enforcement involved because usually we're dealing with something, we're not dealing with simple hacks. We're dealing with fraud. We're dealing with some non-computer crime. In other words, a traditional crime where a computer was involved. And those kinds of crimes, especially in a big organization, need to be turned over to law enforcement for investigation. On the other hand, law enforcement does have thresholds and some companies have policies about not bringing law enforcement in because they want to keep the noise level down. So frankly, what the lady says is absolutely right and it may be your only hope of preserving those logs. On the other hand, the ISP may say, hey, we don't got no stinkin' logs. And then it's between law enforcement and the ISP. But they better really have no stinkin' logs if they say that, too. Excuse me? Okay, so next thing you need to do is contain the damage. And then you want to collect local logs, any logs you can get your hands on. And finally, we're going to image the disks on the victim computers. And we're going to talk here in a few minutes about disk imaging, so let's pass that for a moment. Now, crime scene analysis is a branch of criminal profiling that uses standard investigative techniques to analyze crime scenes. This is what normal investigators are comfortable with. They're used to looking at crime scenes, they're used to analyzing the evidence. And it's good in computer incidents because we can look at what kind of activities the intruder performed on the computer. And we can learn a lot about the intruder from the kinds of activities that they performed. So as we develop a profile of an intruder, we'll start with crime scene analysis. How is access obtained? What kind of skills were required to obtain access? It's quite one thing to take a canned program off the internet and run it against a system and cause the system to go down. Simple denial of service attack, maybe a simple sin flood or a mail bomb or something. A monkey can do that. It's quite another to go into a system as one that I saw, a mainframe where they zapped a single word of a program in memory that caused that system to go down. And they bypassed all the security in the mainframe to do it. Those are two distinctly different skill sets required to conduct those two attacks. And we can learn a great deal by analyzing the crime scene as to what kind of skills were required. And we can know a range of suspects considerably. Now how did the intruder behave on the system? Did they do damage? Were they able to clean up? If they're a skilled intruder, did they clean up their tools? Can we find nothing? Or do we go into on a Unix machine slash temp slash dot tools? And in there we find a copy of our canned shadow. Oh yes, this is a skilled intruder of the first order. Did they steal anything? Was it worth stealing? What was the motive? Why did they break into the system? Are they a disgruntled employee? Or are they like a group that seems to have gotten into the DEFCON website? Oh, probably none of you have had a chance to look at the DEFCON website today. You'll find it somewhat different. It's been changed. We should be able to tell a great deal about that intruder. And the personality types. Also we want to consider opportunity. In the case of the mainframe attack, the attacker had to have been on site. The only access to the mainframe at this particular level of security had to be local. So it had to be done at a time when the intruder was physically in the computer center. So this tells us a great deal. It helps us build a picture of the event. So we're going to build now an incident hypothesis. We're going to try to figure out what happened. And we start with witness accounts. And what I do is when there's an event, the first thing I do is I talk to as many people who are present at the event as I can. System administrators. Users who are on the system and observed some event occur. How did they know that they'd been attacked? How did they know what happened? How did they know when it happened? All of you I'm sure have heard stories about eyewitness accounts of automobile accidents where there were 15 people saw the accident and not one of them got the color of the car right. This happens here too. Absolutely happens here too. So we need to begin to get a picture. So I start with witness accounts. Then we consider how the intruder could have gained access. And we eliminate the obvious. I mean, if we know for a fact that the intruder dialed into a maintenance modem on a unit's host. And we can establish that unequivocally. There's a pretty good probability they didn't break through the firewall and come in from the internet. We can pretty well eliminate that as a possibility. It helps us when we do a back trace. Then we again consider skill level and inside knowledge. And we want to create mirrors of the affected computers. I'm nibbling around the edges of this collection of evidence thing here. And I'm going to get to it here in a minute. But a mirror is a perfect physical copy of the disk that we're investigating. Because we never, never, never work on the actual evidence. Never. Because we don't want to contribute to it. So we've developed the profile of the intruder. Considered a path to end of the victim computer. And now we're going to try to recreate the incident in the lab. Using our mirrors are perfect copies if we are fortunate enough to have them. Typically we will have a mirror of the victim computer. And consider alternative explanations. Let me tell you. It's re-embarrassing to go up before your boss with a report this thick of an investigation. And say, we have absolute conclusive evidence that Joe did this. He had motive, he had means. And somebody in the back of the room says, but wait a minute. Joe was canoeing in the Rocky Mountains when this occurred. And it was an inside job. Well, it could have been Joe. So test your alternatives. Now we're ready to start looking at some back tracing. The elements of the back tracing are the end points, obviously. Any intermediate systems. And if we're dealing with harassing email or something of that nature, we're going to need email headers. If we have a sniffer online, we're going to want to look at packet headers. And obviously we want to look at logs. Our objective in a back trace over the internet is to get to a dial-in point of presence. Now maybe this is coming from another network. And if that's the case, that's what we'll do instead of the dial-in point of presence. But the reason we want to get to a dial-in point of presence is because that's where the logs are. It's like, well, he's setting, why do you rob banks? Well, that's where the money is. So we want to get to a dial-in point of presence because that's where the logs are. The only messages that we can't back trace when we're talking about email, harassing email, those are who's a true anonymizer and those where there are no logs. And let's define a true anonymizer. A true anonymizer is something similar to a mixed master. Something where encryption is used to obscure the source and even such things as the time of the connection are changed. And even the owner of the anonymizer couldn't back trace this if he wanted to. Those are not traceable. Yes, sir? I'm going to show you in a minute. The question is, how do I get around email that comes from an unobvious place such as somebody masquerading as him or using hotmail or something of that nature where it isn't directly traceable to an individual? In that particular case, your ISP will tell you, oh, yes, we can do that. It comes from Joe Fabeats. And Joe Fabeats lives at 1234 West Avenue, New York, which is, by the way, a parking lot. Because we know for a fact that nobody would log into hotmail and put a false name and address in there, wouldn't it? But that's who did it. No, we have other ways that we have to do that and they don't always work, but I'm going to show you one here in a minute. Yes, sir? It depends upon the community. If I'm dealing with a community of sophisticated computer users similar to the ones we find around here, I see a lot of anonymizers. And I have to tell you a story. I generally don't go heavy on war stories on these shorter presentations. We had a situation where we were tracing harassing email messages where the victim had had her life threatened. And the person who did this was doing it through a service-like hotmail. I mean, we had 13 messages and we had taken those 13 messages. We'd gone through the subpoena process. We'd gotten all the information from the ISP. We knew who was doing this and then they started using an anonymizer. Why? They were caught. Oh, well. So these are not traceable. Yes, sir. Anonymizers, they can be, but they can be effective. But I find that when I use them about half the time, the anonymizer delts my message and it never pops out the other end. Yes, sir? So it's about making a decision on a target machine between a senior or a father-in-law distribution session. Wow, that's a tough one because it's hard to take that machine down. One of the things I'm going to tell you in a minute is that the way you do it is you pull the plug. And there's a good reason for doing it. It's kind of dangerous with a firewall. And if your firewall is built on a computer, then we can take our chances and do that. But what if it's something like a Cisco PIX? You're out of luck. Yes, sir? Yeah, the question is basically if I may paraphrase. What happens if it's one of those machines you can't take down, like a server or something like that? This is a judgment call. We have had clients tell us that we cannot take their machine down. They have to keep it running. It's a file server or something like that. And we've had to make the best image of it. We can knowing that the evidence has been contributed to. It's not the best. But business drives security, not the other way around. We're going to do another quick one here and then we're going to move on. Yes? That's a problem too. The question is what do you do if a portion of the disk that you're imaging has intellectual property and they don't want you to take it off-site? First of all, let's make the assumption for a minute that I'm working for you as opposed to being your adversary. There won't be a problem. Obviously, we'll work it out. If on the other hand I am your adversary and I've subpoenaed that information, then what you need to do, and you need to do it real fast, is you need to go to court and you need to get an order to stop me so that you can have the judge can issue an order that will protect the information on that from being revealed. That's intellectual property. This happens fairly frequently. I've had it happen several times. We've done some popular cases where that has happened. You may lose the case because of that. That's right. The judge may step in and quash the whole thing. That happens. Yes. Yes, one more. And then we've got to move it.  If it involves national security, you can bet that the situation is going to be such that that information is going to be protected. You can just absolutely guarantee it. Other than that, it's whatever you can work out with the owner of the information. You know, they may get the judge to issue a protective order. I've had situations where I've imaged a disc that had medical records on it and those medical records were deemed to be private to the individuals. And so a representative from the organization that maintained those medical records was permitted under supervision to delete them before we imaged the disc. It's tricky. It's very tricky. Okay, we've got some enabling relationships when we go after a back trace. And we'll start down at the attack victim down at the bottom. I don't know that I have enough cord here. I guess I do. This is the first place that we're going to look for logs in the victim. The next place we'll look is intermediate hosts. Now, here we have a router. Those don't have discs. Some of them log to an external device. Some of them don't. They have cash in them, though. And they cash their routing tables and they cash the traffic that goes through them. That information can be extracted. Then we want to get up to the ISP where the point of presence exists. There are going to be logs on the ISP's dial-in router, like an Ascend or Cisco dial-in. There's also going to be logs at the telephone company. Now, you've got to subpoena those logs. You're not going to get them just by asking for them, but they're there. So that's where we want to end up. And that's the key to the back trace. Now, obtaining subpoena. The way we do it is we notify the organization, usually an ISP, that we're going to subpoena and request they preserve the evidence. And we ask them who to send the subpoena to. Most of the larger ISPs have people that you send subpoenas to. So we always get or usually get cooperation there. Then we get our lawyer to file a John or Jane Doe lawsuit, which basically says, somebody hacked me. I don't know who it is, but I'm going to sue the bastards. And you've got to let me go and look for information so that I can find out who they are, because if you don't, I'll never be able to find out who they are. Or even if not, the court will let you do this. This is fairly controversial, but in most states they let you do it. Then subpoena the logs you need and get everything on the first pass. Don't expect to go back for more. And you may have to depose the people who collected the logs to find out how they collected them, because we have issues of things like chain of evidence and chain of custody. Okay, if we're going to use logs for evidence, they can be modifiable, which of course isn't possible. But we want to raise the bar as high as possible. If they've been spilled off to a protective log host, if they've been saved to optical media, if we have corroborating copies and backups, then we can probably use the logs as evidence. However, there will be questions of authenticity and there will be questions on accuracy of the logs. You must be prepared as the expert to answer those questions. And when the opposing attorney says, could the logs have been modified? What's the answer? Yes, absolutely. The logs need to be complete too. And complete in my mind is we have to cover all super user access, log in and log out times, attempt to use controlled services that's an M.T. thing, attempt to access critical resources, and I like to cover email details as well. That makes big logs. And there are organizations who do not want to log email details because they don't want them to be subject to discovery. And then there are those organizations who have been subject to some minor mischief where the logs just disappeared. Appropriate retention, assuming our lab is usually six months or more because we can go months and not even know that there's been an attack. Okay, tracing email headers. I told you I was going to show you how to do this. Everybody knows how to trace an email header when it's easy. Okay, and everybody in here I think probably knows that the farther down towards the Dear John part of it is the source and the farther up the stuff that nobody seems to pay any attention to is the destination. In this particular case, you'll notice that on number one there, the grade-in part, it was received from Read at Local Host on Web03.iname.net. Now what this is, for those of you who have never used I&A, this is one of these outfits where you can get one of those free email accounts you see, and you go into their web server and you compose it on a form and the web server uses a post and sends it out. So the reason that it looks like it came from Read at Local Host is because that's the process. Okay, but it's worthless. There isn't a piece of useful information there except for the time and date. That's the only useful information that message ID number is totally useless. The log that contains that probably had been gone for days by the time I got to this. So what we need to do is we need to understand the system and the system is a web-based system and since it's a web-based system, we hope there's an access log, an HTTP access log, and an HTTP referral log, and I'm running out of things to cross up here. Turns out in this case there was an HTTP access log and we were able to look at Web 03 and see that on Saturday 12th September 1998 at 1825-13, Eastern Daylight Time, don't forget the time, don't forget the time zone, that there was a particular connection. We had about 13 of these and turns out that the connection was owned by the same ISP in every single case. We were able to go to the ISP because we had the source IP addresses which turned out to be dial-up points of presence and we were able to find out who owned the account and we went to the person who owned the account and she said I didn't do it and her husband said I did and we had him. So the trick here is, the question that you asked is forget about who it's from, it doesn't make any difference, they're probably lying about who they are anyway and go for the time of the connect and start working back a step at a time until you get to the dial-in point of presence. When you get there, you've got your best chance, I don't guarantee you're going to win, but you've got your best chance. So in this case we contacted our own security officer, he gave us the connect account name, time, message ID and source IP address, all of which were totally worthless except for the source IP address. We located the ISP, contacted their security officer, got their logs from the source IP which was the dial-in point of presence. It happened to be in a Sun router and found out who was connected at the time of the email. Yes, sir. Cried. I had a lot of time invested in that. It took me 15 minutes to do that trace. No, we'd have been in trouble. If the logs were gone, we'd have been in trouble because we had gotten back to the ISP but we wouldn't have been able to go further. Yes. If the question was if I had broad enough number of instances, could I get telephone company logs? The answer is yes, I could anyway. But you have to, when you write a subpoena, you have to, and your lawyer will know how to do this, you have to write it very specifically. Not as specifically as a warrant. You've got some breadth there. The reason you write a subpoena specifically is to make sure that you get what you really want. Now in this particular case, in this particular case the ISP, and then this would be different from ISP to ISP and from dial-in to dial-in. But in this particular case, the IP address is dynamically assigned to the port and it was log, port, and IP address. So we knew which port it was and we didn't have to do all that. Okay, there's a question over here. How often in real cases did you work in if you encountered a problem of you could order it back to get it to the account and put it at the time, put it on the bottom here and discover that it's an American online user that basically used somebody else's account to get all the tricks that you know. It happens. It happens. And in that particular case, the only way you win is to have enough information that you can begin to start correlating things and start working through the maze. But in the case of American online, the maze is so big and there's so much stuff going on that it becomes almost impossible to get enough because where you really have to go from here is to the telephone company. That's really the next step back and find out what the dial-in was and that can be a massive problem. Massive problem. Okay. Yeah, there's one back here. This is the last one for just a second. We're not finished. I'm sorry. Oh, okay. I'm sorry. I was having trouble hearing. The public makes it. The question is if it was a publicly accessible port such as a cyber café or something like that, well, all I can say is that you're going to need some good old-fashioned cop work. You don't have anything to do with forensics. There's going to be just standard police investigation questioning people who was there at the time, that kind of thing because the forensics isn't going to do a bit of good for you there. The toughest thing to do is to put a human at the machine. We can trace back to the machine by putting a human at the machine and identifying that human is the hardest thing to do. Okay. Evidence collection and preservation. Forensic computer evidence we create. Physical images using a product called SafeBack. There are others. I happen to like SafeBack. It makes a physical image. That is the evidence. Everything else that you do with that is a lead. You protect the evidence, the entire physical backup. Never work on the image. Make a second copy. Forensic computer analysis. I use stuff from NTI. New technologies incorporated, but there are others. And that does DOS on Windows 95 and 98. There's some new stuff and it's free from NTI objectives that will help you do stuff with NT, which is pretty good. Never work on the evidence and always assure a chain of custody and a chain of custody means that you can account of your own knowledge for everybody who has handled that evidence since you collected it. Okay. Now what can we do with Forensics? We can identify email addresses, URLs, graphic and zip files. We can search for key words even in erased files. We can recover deleted logs. We can trace chain of events within computers, devices or networks. And we can locate masqueraders. We can't necessarily identify them, but we can locate them. Remember, every action on a network or on a computer leaves a fingerprint. It's there. It may be real hard to find, but it's there. I can try. Come on. Well, we're not... I'm afraid we're... There we go. That's the one you want? Okay.  Happens all the time. May I proceed? Okay. The places we look for are Slack Space, which is the unused portion of a cluster after the end of File Marker. Mostly in BOS, but some Intel-based Unixes like Linux and FreeBSD, we get something similar to this. An allocated space is the space where nothing is or where erased files are. Swap files, window swap files, the whole treasures, and cache files. Especially internet cache files. We can often find passwords and all sorts of other stuff in there. But you have to do it from the physical backup. No other way to do it. Okay, logical backups.  Everybody backs up based upon the file or whatever file system is reporting. It's not the real file space. It's just the file. The file space includes the attached Slack Space. And that's a physical thing. It's clusters. Okay? Physical is bitstream backup or mirror. A mirror is when you take the bitstream backup and you do a restore to a hard disk. And it contains everything that was on the source, even if it's nothing. That means it actually contains the empty sectors. That's critical in the investigation. All those things I said we could get, we got to do this or we don't. So physical backups for files and windows. What we do is we pull a plug. And the reason we pull a plug is because we don't want to collapse any files. We want them to stay there. Also, windows creates and deletes dozens of files during the boot process and during the shutdown process. So we want to ensure that we don't contribute to the evidence. And then we're going to reboot using a DOS disk with a backup on it. Okay, quickly. I'm going to get to you, Nick. It's two slides forward. Okay? Take the backup for evidence. Take the second one as a work copy. Restore it to a lab computer. Now we have a mirror and we can go and analyze that mirror and preserve our evidence. For NT, we do the same thing. But after we build the mirror, we reboot it with DOS and NTFS DOS. Wonderful tool. Let's just look at the NTFS file system as if it were a DOS system. And then we can get the SAM so that we can go take a look at passwords. We can analyze other files in there using DOS tools. And we can use a tool called NTLast, which is one of the tools from NT Objectives. Now, Intel Unixes. Okay, Intel Unixes retreat pretty much like DOS. Okay? Once we have got a mirror of the Unix though, what's going to happen is it isn't going to boot. The mirror usually won't boot. So we mount it as a second volume. So if we've got a Linux disk that we're working on, we stick it as a second volume on a Linux machine, boot the machine, go in and work on that disk. We can also use the DOS tools to analyze the bit stream, the backup, not the mirror, but the backup. Yes. Yes, it does because it's a physical backup. It doesn't care what's on there. It's just copying sector by sector. Good question. Okay, we've got another question here. Oh, yes. Yes. It's a wonder that I'm not bald from tearing my hair out. Cryptographic file systems and certain types of compression just ruin this whole thing. Quickly. Yes. Yes, it does. Saudex's product. It's one of the, he said the sector translation have anything, have any effect. And what he's talking about is the difference in disk geometry between the source and destination. I assume that's what you're talking about. Yep, yep, yep, yep. In general, in general, and this is a general statement. There are exceptions. Some of the remappers, some of the compressors, some of the things that change the interleaves. This doesn't work right with. But generally speaking, the Saudex product, we'll try to adjust the geometry between the source and destination when it does the restore. It also gives you some ways of doing it directly and ignoring all of that and just moving the sectors. It's not perfect, none of it is. Well, as long as we're not doing anything to the evidence and we're only, yeah, that's right, it's read only. All right, so non-intel Unix is like Solaris. We remove the disk, place it in an Intel computer, be careful because we don't want to contribute to the evidence. Boot it with a floppy, a DOS floppy. Remember, we're taking a physical image so it doesn't matter that the file system isn't even slightly compatible. Take your image and we create the mirror. Mount the mirror as a second volume and another Solaris computer and go to work on it. Yes. Oh, yes, raids. Those are eight discs. We pop them and we do them like this and birthdays pass. There's no other, an eight disc raid would be a blessing from heaven. Trust me. When I walk in and I see a cabinet with about 40 in it, generally speaking, I raise my rates. Okay, authenticating electronic evidence. This is a problem because you've got to be able to show at the time that it's used in court that it hasn't changed. We fingerprint it with a product from MTI called CRC MD5 that does a 32-bit CRC check on every file plus an MD5 message digest hash and you're not going to change that file in any way, shape, or form without it showing up one or the other or both of those places. Then we encrypt it and we don't tell the password to anybody. We keep the key to ourselves and we use a nice meaty encryption algorithm. We use 256-bit blowfish. I'm sure there are those of you who can crack that but not within the useful life of the file. And that's the same as signing and sealing physical evidence, maintain a formal chain of custody. Remember I told you we can get stuff out of routers? This is one way. We can use a tool such as ODS's secure detector and it can find all those things. It uses our mind too and queries the cache. I'm moving faster because we're running out of time. CNDS is the Computer Misuse Detection System. What we do is we feed it logs that would be at least 30 days, the more the better. What it does is it looks for changes in usage patterns of users. What that does for us is help us to determine if perhaps somebody is masqueraded and come in as a legitimate user. Their usage patterns will differ markedly or if we have a normal legitimate user who's gone rogue on us and is going into places they've never gone before, it'll show up. This is a neat program. It's available also from ODS Networks. It costs a lot of money. I'm done. Questions? Thank you. I'm also out of breath. I'll hang out and answer questions, yes.


So the here is an 8k this is an 8k camera. So this is the world's first handheld 8k camera So hello, so who are you? My name is Hashimura. So I'm in charge of marketing So these cameras who's gonna buy them? So is it for a broadcast in Japan? Yes, so far in Japan only but maybe we can Expand for the worldwide market So Ikegami has been doing cameras in 4k also how's market for 4k? 4k market is growing So there are lots of lots of countries use the 4k and What's what's the update? How did you update to 8k? What's inside? 4k and 8k is totally different. So 4k is we have a 2,000 inch 3 chip version You have what 2,000 inch 3 chip 4k sensor 3 chip inside. Yeah, this one is not 3 chip. This is 1 chip. 1 chip. Yeah, so this is 3 million 33 megapixel How is it inside? What's inside here? In the process in the CCU And what is this? This is a different product. This is a 5 bar transmission So the signal is going to go over there. So all these So would it be better to have 8k with 3 chip? Is that the future? Probably. Because it's much bigger the 3 chips. I think so. Yeah So what's the price for this camera? Is there a price? Priceless. There's no price. There's only one in the world or there are few So it's very special 8k How many frames per second? 68k and Just goes out through these systems here