Okay, so it's 310 and I've been told by Paul Henning that if I run over time he's going to throw things at me. So I'm going to attempt not to run over time and I'm also going to attempt to start close to on time. My name is George Neville-Neal. I work on the FreeBSD project as well as several other things. In particular my area of interest is networking and this talk is called Network Protocol Testing in FreeBSD and in general. So I'm going to talk about a few things today. I will talk about how one does network testing. What I mean when I say network testing and what kind of things I've been doing with FreeBSD and hopefully this will be interesting and useful. So what is network protocol testing? Why would we want to do it? It turns out that writing network protocols is difficult. For those of you who've ever worked on them you probably know this. This is actually why I like working on them. I once worked in user interfaces which are also difficult but in networking there are no product managers telling you the package should be green. So instead I went to work on network protocols that are deeply technical and very fun. So they're hard to write. People make mistakes. I've made mistakes. We all make mistakes. So how do we find mistakes in complex systems? Well we design various tests and carry them out over time. Hopefully when we find new problems we put them into a regression test suite so when we release the next version of a piece of software we don't have the same problem again. So why is this hard? Well one of the things that makes networking fun is that network systems are non-deterministic which means you may get the message or you may not and things don't generally move forward nicely and synchronously and you have a lot of problems. You're subject to message loss. There are time synchronization problems. Also protocol testing can often require lots of machines or now that we have virtual machines and I do a lot of testing this way. Instances of virtual machines but you need a lot of boxes and there are these nasty what we call knock-on effects. So when everything works it works perfectly but when something doesn't work nothing works and all the machines begin to stop doing things. It turns out and I've this is this is one of these old people's comments now that I'm 40 I can say I'm old. People seem to find systems thinking hard and networking requires the ability to do what I call systems thinking. So there are people who can very or very adept at making a single line a single stream of code work very well or very compactly but there are very few people I've met in the world who can look at an entire large system like you know a transaction processing system or the internet or some subsection of the internet or a set of machines that are interacting and make any sense of what's going on. They might be able to understand one part but they can't understand all of them. So in order to help us you know this is what makes network testing hard because network testing is all about this kind of stuff. It's also what makes multi-period programming hard but that is not the subject of this talk. So what are some measurements that matter if we're going to do network testing? These are the three correctness latency and bandwidth. These are performance tests this is correctness. Most people do these tests first. I don't know why. They don't test for correctness. They test for speed because well you know if the thing runs really fast that's good. Of course if the thing runs really fast and drops all the packets that's bad but you know marketing people tend to want latency and bandwidth. So what are some prototypes some types of protocol testing we could do? There's specification inspection and people actually do this. I was just involved in and dealing with some of this stuff around IPv6. You read what the designers said the protocol is going to do and you think about how it should work in the abstract and then you say well you know if you do this and this and this and I do this I can break your protocol. Your most powerful it's actually one of the most powerful ways of doing testing because you get stuff beforehand and because it's broad based. If I find a broken problem in an implementation that's bad for the implementation. If I find a broken thing in the protocol I break all users of all users who have implemented that protocol to the specification. But we're not going to look at specifications today because it's after lunch and there was only so much coffee. You can do code inspection you can read code which is always fairly important but I'm going to talk about these next two here. So protocol conformance testing which uses real packets to ensure the implementation conforms to the spec and I'll talk a little bit about fuzzing tools. So fuzzing is not a new idea. If you read slash dot they think it's a new idea but it turns out that there was I forget his name there was a professor in the states in the late 70s or early 80s who decided that he would test the abilities of Unix commands command line commands not networking stuff by feeding them garbage from dev random and some large number of them said fault it because no one had ever tested them with random garbage. So now this is suddenly again become a very popular thing to do with network protocols instead of doing something like saying well I know that you know once I've received the sin act from the other side I should send it the act and then we can open the connection instead I go well the packet looks like this and I'm going to send random garbage in this bite until the other side collapses and see if I can do that. So that's protocol fuzzing. We'll talk a little bit about that mostly I'm going to talk about this. So what are some typical network protocol tests. This is my least favorite but it seems to be the most common which is it works. Great that's that's very helpful. You know just because you've been able to download you know www.freebstnetbst open BSD dragonfly BSD or pick your BSD so I don't insult anyone dot org does not mean your protocol stack is conformant and does not mean that if I put it into a production network full of thousands of machines it won't collapse explode cause a major outage or any of these other things. All it means is well yes it's not so broken that it doesn't transmit data. But that doesn't tell you much. So other protocol testing systems that exist. There's anvil and net bits. These are really expensive. I actually once worked with anvil for an embedded systems company. It's not I'm not saying it's bad product but I don't have a hundred thousand dollars to test free BSD. And if anyone does please see me after the talk and I'll I'll share a cut with Paul Henning because you know for the conference and that bits is a hardware device to do this. Next down is a system that the wide project who I spelled wrong called Tahi that they built used for IPv6 and IPsec testing. It has its own issues part of which is I'm going to insult a large group of people now it's written in pearl. Enough said. And so then there's that perfect net pipe and this is the thing that people do where they you know they just pump data through the socket interface and they hope that it all works. So what is I don't know what was just said but it's very funny. So you know what what how do we do this kind of stuff what what kind of system you usually see well the simplest test rig is one that looks a lot like this and you'll notice it's got the little BSD symbol on it. So you've got your lab network which is the network on which you're not putting traffic one of the most important that one of the things you really have to do when testing network protocols especially initially is to isolate them from all other traffic because if you have them mixed with all our other traffic at first you will be completely unable to debug them because you'll get random you know oh look I got this weird ethernet frame what is that you know suddenly your test stops working. So you got a lab network you've got two test lands usually I'm testing things like routing or forwarding the thing I've been working on most recently is Ipsac and free BSD and so we care about you know the tunnel comes in here and then the tunnel goes out here and you have to have this whole setup you want a serial line because you want a way to control the box without using SSH over one of your test networks turns out that SSH generates traffic you wouldn't want to see also when your machine explodes as it you know network protocols are in the kernel when you make a mistake in the network protocol your kernel explodes you want to be able to do serial debugging so this is your typical test rig and I happen to have one right here I won't do the demo but I have attended on the road bring VMware with me and run a couple of VMware's abusing each other on our protocols so let's go over a little terminology because I'm going to use these terms in the rest of the presentation we have what's called a device under test so this is your custom kernel this is the thing with the new protocols in it this is the thing you're going to test now that's may sound simple but I've given presentations where people are like what's the DUT because I do DUT in my in my slides you get the test host that's the thing that's controlling it you have your test land and you have your lab network and I talked about these already again the test land really should be if you're doing conformance testing quiescent it should be quiet there should be no other traffic that that you don't expect because it's going to cause problems but you need the lab network to get things like you know CVS up and to pull things down so what problem was I trying to solve when I built the software I'm going to describe today well writing to network protocol code is hard I said that testing network protocols is also hard most systems that I've seen are incomplete so they only you can find a bunch of these libraries on the net and I'm going to describe yet another one which theoretically will be more complete but they what happens is someone goes oh I really want to test this and they get as far as like TCP you know and they don't do all of ICMP and they don't do anything else and then their stuff isn't in there this is a big problem not extensible this is what makes Tahi really hard to use the system I mentioned earlier is that describing packets in code well you know you can rewrite the protocol stack and then you've got two implementations in C that you try to make talk to each other and that's really hard so you want extensibility you want things not written in what I call right once languages you want to be able to read your code six months from now and then again these proprietary systems are really expensive and actually amusingly enough incomplete the commercial vendors who build these things build the tests that they believe will make money so years ago when we used anvil at a company we wanted a TCP test because we were actually going to we were using the BSD for four stack which tells you how long ago that was an embedded product and we wanted to test that and they said well yeah sure if you pay us four hundred thousand dollars or some huge amount of money which were at least to me is a huge amount of money will do a TCP test for you but we've got this great test for you know some random cellular phone protocol because that's what makes money so these tests are also these systems are also often incomplete so what's a poor open source or to do right your own of course make it open source of course can't be that hard right and you know it's just code and I've got lots of free time so the first way you might do this and actually I had a summer of code student who decided to go this way much to my chagrin you can write package generation tools and see using the socket code and using BPF and directly writing these things to the to the interface you can write all of these tests by hand and I'll talk about what happens when you try to do this in some slides a little later so it turns out that you wind up re-implementing the network stack so our current this is all free BSD quotes just for IPv4 there are a hundred thousand lines of code right and it really would suck to have to rewrite that actually we really sucks just to fix it but you know it would suck to rewrite it so a middle ground is possible so I wrote this thing called the packet construction set which is named after for those of you who are old enough to remember the pinball construction set which is a really cool video game when I was a kid python library for packet construction easy creation of packets gives a more natural syntax for packet manipulation and of course BSD licensed yes yes I've tried using escapee their syntax is interesting so I'll show why escapee is one of the systems that doesn't that has more packets but doesn't go far enough in terms of what it gives you for programming which is what this is all about so let me get to my next slide so what we need to get to hmm that's the TCP header we need to get from this which is a description of a packet to this oh no wait a minute actually this right so this is what happens when you want to calculate a checksum on TCP and C and great you want to build a router you want to do something fast this is how you're going to implement it but if I just want to send a packet in a conformance test I want to say the packets checksum field is equal to the calculation of the checksum and I want this to be automatic so one of the nice things about the way this works in python which I'll talk about is that this attribute here represents a set of bytes encoded underneath the object that you never have to manipulate directly so you set the checksum and it is set and you're done one line of code so what are the advantages of PCS besides the fact that I'm you know giving the talk it's easy to specify new packet formats this was the biggest problem as you look at say a C structure you look at an RFC and you can specify the format but people do really annoying things like they have 13-bit fields that cross a byte boundary right and you've got all these other weird things you've got you know one bit fields and three bit fields and even the people who did IPv6 who claimed they were making things easier by making everything aligned well that's mostly true but only mostly and if it's mostly then you're screwed anyway you're just not as screwed I guess so what you want is a natural way of setting and getting packet fields I wanted to do this written in a well-known language I mean I could originally have done this you know I could have decided to go the full graduate student route which I am not and you know right invent a network line network protocol language and write a lot of papers and pull my hair out oh anyway so instead I wrote it in a well-known language and scripting language is a really easy to play with modular well-documented I actually do write documentation for my code there's now a 20 page manual for this system which describes how to extend it hello there we go so here's something you only want to write once and this is what the TCP packet description translates to in PCS so once you've written this this is inside of an object once you've written this and you instantiate TCP objects you get all of this for free but I will explain this because it's important to understand what's going on underneath the hood so you understand why the power of PCS works so what we're doing here is specifying as you can tell fields we give them a name and we give them a bit width and you can see we can actually have one bit fields right and the system underneath the hood actually underneath the objects does things like bounds checking so if you write a program and you attempt to put more than six bits into the reserve field it gives you an error and it actually gives you an error with like a real error message that says you've tried to put more than six bits into a six bit field it tells you it doesn't say segmentation fault core dumped right please go find the debugger and if it works you'll be able to find out what's going on so you get all of these and what this gives you and I'll put this into the next page no I won't do it in the next page so much for my knowing my own slides what this gives you is the ability I showed you in the earlier slide to set the checksum right you need never twiddle bits so and you can compare packets so this means when you write a conformance test you have a known good packet you read a packet from the network you say does this object equal this object if these objects are not equal print out their difference that's what you want in a conformance test you do not want to have to write you know wander the bits by hand okay so I'm going to show you a test that someone coded and it's this is one of those those happy unhappy accidents so my summer of code student was working on some tests like this and insisted on writing this stuff in C and I said okay fine and then Kip Macy who's been doing a lot of performance analysis work and a lot of TCP work in free BSD said I need a library to test this because we've got this problem with the CINCASH and I gave him the library and he wrote a test so what is it I'll tell you what the test does first so the CINCASH timer the tester the test controller sends a CINPAC to the device CINPACKET to the device under test the device under test is supposed to put the CIN in its CINCASH and then it sends CINAC packets back to the tester to continue the handshake process now what we're testing here is the timer not the actual full setup of TCP so the CINCASH has this timer which controls how often to send the CINAC and the DUT must send four packets back to the tester in under 60 seconds now a slight aside on general network testing issues one of the things people are particularly poor at including in some engineers is understanding what they're trying to do thankfully kernel programmers generally aren't bad at this because if you don't know what you're trying to do your kernel explodes and we don't give you a commit bit but I work with people who are not network programmers I work with people who only code in PHP and the number of times I've had to say what are you trying to do is really scary so what's good about this test or at least about this slide is that this is a really good example of what you want to do in a good network test what makes a good network test is you have an understanding of what you're attempting to do and you have an understanding of the success or failure of the test the problem with the it works test or even with the performance tests is you don't have a very narrow definition of success or failure right if marketing likes the number then the performance test is passed right but it doesn't matter efficiency or anything else like that this is a great example of a test that kip came up with because it's here's what happens here's what must happen for the test to pass and it doesn't happen it doesn't pass this is what you really want to see and this was actually one of the real advantages one of the only advantages actually of working with the anvil system because they were really obsessive compulsive about this they went through the rfcs that they were testing and every test had a page like this and this is the equivalent in testing of good comments in your code right of this happens this happens this happens this happens these conditions must be met pass fell and that's it and then you make a bazillion of these and you run them so that's what this is supposed to do this is what this looks like in terms of a network time diagram we send this in we get synax and xnac and this should happen in 60 seconds very simple what does this look like in code well the font's a little big otherwise I could put this all on one slide but then there'd be no point because you could read it it's about 60 lines of code here we are setting up a IPv4 object right we said it's version it's header length type of service we give it a random ID flags offset time to live check some is initially set to zero because we have to calculate them together set the protocol correctly and then we can pass in a source in destination so here's the interesting part here's TCP we have a port we give it the magic sequence number for those of you who've read Douglas Adams we set the sin bit to one notice there's no funny masking we just set that thing to one we set push to zero of course this being a nice you know high level slow also scripting language I'm not going to win any races with this particular piece of code everything's initialized to zero when you start so you don't have to initialize things to zero we do that for you so you set the sin bit to one I don't know why you set this is your checksum window we can set the ethernet so now you know the ethernet addresses of kips machines because I didn't anonymize them that oh well we get the length and then we calculate the checksum right so this is all handled for us underneath we create a chain of these we set up a connector to the p-cap system and we transmit the packet and this is how we test it right so while you know here's the test coded exactly the way we discussed it in the slide we set an alarm for 60 seconds while the sin counts less than four we read a packet read packet will unpack packets correctly into objects for you so if you have to interrogate a packet that you've read from the network you can interrogate it exactly the way I just set it and you can see the the interrogation here deport the destination port right deport nesport act number sequence sin count it's all taken care of this oh packet that data data so the way packets are changed together is through their data members so there's a there are several if I don't want to go into I mean I can go into a little bit but there are several special data members the special members in the object that you can't replace which are in the documentation and data is oh look I've got a whiteboard if I've got an ethernet packet followed by an IP packet followed by a TCP packet they are connected through their data members this points to this metaphorically right this isn't really see and this points to this and so if you want to actually get the TCP packet out of a raw packet from the network you go data that data actually no because it will fail if these aren't there so if the packets are the wrong kind of packet the other thing is he's filtering for that well he's filtering for IP but he can filter for TCP but yes you could also take for check for that you can actually do because it's a type system you can say is it type of TCP he's not doing that without looking for yes so some interesting statistics about this versus the version written in C which I will not put up on the slides and actually the thing that you know it's I'm not I'm not ragging the C code actually this guy wrote some good C code he wrote 600 lines of it right 600 lines of C to implement a 60 line Python script which seems backwards to me obviously similar code in the kernel which handles a lot larger number of cases for TCP is hundreds of lines and you don't want to have to do that you want to basically focus on what you're focusing on the task so what makes a good test I talked about this in the in the first slide you want a focus test you want to test it's easy to read you want to reproducible you want to be understandable and you want to be able to integrate easily with other tests right so small easy to understand tests are what you want that way when you go back and you look at them you can say is this really what we're testing because if you were to look at that even if you look at the the C code that this the student wrote you're like well yeah yeah okay I've paged through it a few times now I understand it whereas this is easy to understand I believe easy to understand so what packets do we currently support this is where a lot of I've had a lot of problems with other systems so I developed this when I started doing Ipsac because the Tahee tests were missing things that I wanted and because I realized there were there were very focused tests that I wanted to run that they wouldn't do so ethernet loopback interfaces are supported there is actually a loopback packet format it's known as putting the address family into the packet if you don't do that your stuff doesn't work so ethernet loopback ARP IPv4 and v6 ICMP v4 v6 neighbor discovery which is part of v6 incomplete support for a H and ESP because I've been fixing the C code UDP TCP DNS HTTP and then I've got other people who are using this for protocols that are much higher layer so there is a group that works on an instant messaging client that's using this for instant messaging not to do the client but to test the client some future work well so here's the problem more packets lots and lots more packets and if you look at something like wire shark which used to be called ethereal you know you've got a if you really want to be complete you've got to support a large number of packet formats so I've actually had a lot of requests for 802 11 not just from Sam Loeffler by the way who that's when he saw this talk his first comment was does it do 802 11 no it doesn't Sam sorry a more complete DNS maybe Apple talk better support of HTTP turns out to be important because people somehow write applications with it more tests so I've conned that's the wrong word I've sold KIP on using this system to do more of the TCP testing so there's a whole bunch of TCP changes that went into free BSD that need to be tested and need to be checked so KIP's now starting to do a TCP conformance test suite with this stuff so you can get a hold of him is K Macie at free BSD org I want to do a Tahi replacement because the Tahi guys aren't going to keep doing their stuff forever and because I want something that's I can understand and then fuzzer so last year I not just last year but the year before there was a student summer of code student who wrote a whole bunch of IPv6 protocol fuzzers with this actually that's been one of the few criticisms I've gotten from people at work well not in my team but other people who've seen this they say but you can use this to attack network code yeah yes yes you can it's exactly the point actually but that's evil no no writing bad code is evil allowing it to read and allowing it to remain on the internet is really evil because I'm not the only evil guy right you know if I were the only evil guy you could get rid of me if you're right but there's more evil people besides me so protocol fuzzers definitely something we want to look at they're very useful for causing kernel panics and other things to show up a more comprehensive test framework so one of the problems with all of this stuff is that and this is the part that a lot of people don't want to write test frameworks it turns out are hard and test frameworks for networking are particularly hard because the configuration problem is a pain how do I configure the device under test how do I make it so it's generic so that you know all of if I could get all of you to do testing with me if I get all of you to do testing with me then you'd need to be able to configure your systems on your own and sort of a text description of that stuff people have tried it and it's it needs to be done but it's ugly and it's not something anyone wants to do but it needs to be there because there's no otherwise you know once we have more than 10 tests you know once we have hopefully a hundred or a few hundred tests you're going to need something that will run them reproducibly in a regression system and we don't have that yet then integrate that into the free BSD regression tests and then more tools based on PCS so I have until when 50 okay let me talk about one thing first and then I may demo the packet to buy you have a question yes sir you can sit with making a true way to do a packet capture generate PCS code yes yes it's very hard but I was reading a paper about that just recently why would that be well this is this is a packet sequence we're looking for write a skeleton that generates these packets from your checks these packets oh yeah you that could be done I thought you were talking about so what one someone pointed me to good paper recently like in the last few weeks where there's a group at Microsoft Research that built a system to infer what a protocol was with no knowledge so that's that's hard it could be done but that would be harder to write this code from captured cases yeah so that wouldn't be so hard because you can just basically invert the system and you know the thing does do reading correctly so I can I can share that later but yes I'm in my copious free time other questions while I'm being photographed I just notice the guy with the camera okay so here's yet another project because I need so much more to do in my copious free time so the other thing and those of you who read the free BST network mailing list probably saw my mail here's something we do not have this by the way is a standard kilogram of platinum no you can't have it it's in I believe it's at the Alte Metier in Paris and this is supposed to be the kilogram that you when you want to when you want to kilogram this is it well as the web says the old one this was once the kilogram using Google this is as close as I could get to the canonical kilogram so if you ask someone like a networking person which I see many people I know from the networking world in here you say to someone so what is a TCP setup look like well you know first we send an act and then we get a SINAC and then when you know when we send a SIN we get an act SINAC and then we send an act yada yada show me one show me the packet trace that actually has the canonical TCP connection setup nobody's ever as far as I can tell I went looking for this nobody's done it people have huge packet traces of here's all the traffic at you know this particular entry point to this country that's not very interesting to me what I want to be able to do and this is what I'm starting to what I'm going to do next or as part of this is to start collecting canonical traces now of course this is going to be a problem because a canonical trace between free BSD and free BSD is not the same as free BSD and Linux is not the same as Linux and Windows but that way you can have these nice short PCAP captures that can be used in these tests so that's the next thing I'm working on with this stuff and that's going to I'll talk about that on the next page I've gotten to my question slide and the last bit here is where that is so there's a whole bunch of code and involved in this whole bunch there's a whole bunch of projects because I want to keep things separate so I can give different bits to different people in particular my employer has been kind enough to allow me to spend all my time hacking on this even though it's only tangentially related to what they do so I want to make it so that the package they use is separable it's all BSD licensed so you've got PCS itself I'll talk a bit about the packet debugger since I do have some time then there's the net test framework which isn't anywhere near done yet this has a page but no data and I set up the page yesterday so it's a very new project but this is PCAP lab and this is where I intend to start collecting these canonical traces indexing them having them available to people so that when someone says you know I've got two boxes and they do this you can compare them and you can test them right now someone says I've got two boxes and they do this you go okay well you know put a printf into the kernel and tell me what the printf shows you right but you can't really replay a test at them all of it is readable on mercurial I have a couple of mercurial repos sitting on my my server in the US there's a version in ports and this is this is the thing I need to get people to do more of which is contribute I mean I would because I'm an idiot spend all day sitting around reading RFCs and setting up packets but even if I do that I'm only one person so I need people to send me mail send me stuff they're working on look at the system and try to play with it and you can always reach me here so let me take a couple of questions and I'll give the quick packet demo yes for instance if people are testing HTTP they have to care about all bidfields and TTP no right one of the things that PCS provides is connectors at all layers of the socket so you can have a PCAP connector you can also have a UDP a TCP an IP connector which is the raw sockets and HTTP all the way up yeah right but if you're doing protocols that are way up in this what I consider the stratosphere from down here at IP then yeah that's definitely there because this is exactly what these I am people are down right they want to test it over TCP and HTTP because there's stuff over that if you're playing late you may want to control packet boundaries and stuff like that yeah oh and I'm supposed to do an sctp thing for Randall but I haven't done that yet other you are yes you were waving at me I'm not sure I corrected the student he said that when you generate a new packet it will fill out set to zero okay I know SCAPI I agree that SCAPI is not designed for programming but more for interactive to packet generation it has a very interesting feature that that everything has default values which will be computed unless we set a specific one yes for example tp checksum we get that most of the time we want the good value so just before sending the packet unless you set a manual value on I think it could be could generate shorter programs yes so there are two things that we do and I actually the TCP case doesn't give a good example of that but other packets have defaults so an IP object when you instantiate it even though kept set it to four four is the default value so one of the things you can set in a field is its default value and so we do that for a lot of stuff TCP doesn't have a lot of defaults in terms of calculating the checksums automatically I've looked at that there's a little bit of introspection going on in the libraries that I have to pull apart to make it so the thing doesn't do an infinite loop anytime you change any field right so that will happen but it's one of those oh when do I stop check calculating the checksum so but yes there are default values just not in that example there will generate shorter programs other questions all right let me do the quick demo because Paul and I hasn't thrown anything at me yet let's let's hope the quick demo works can you read that maybe not oh no you can't yet must be orange on black always sorry where's the info there we go the black on black that'd be good I like that idea that's good tell me you can't read that the color is kind of easy isn't it okay info color do you like flora that means that more readable second floor is good okay let's do flora yes yeah it doesn't make a nice chunk a chunk of sound when I'm typing so source oh good it does work okay let's try that again I guess I'll send these out over this link so this is the packet debugger and this is a tool built with pcs so the idea behind the packet debugger is you can work with packet capture files the way you work with source code so in a source debugger you can say things like list and break and continue and you can interrogate things in the in the source code well that's the same idea here so if I say list I see oh there are these three art packets in this ethernet trace and it does a mildly not so exciting job of printing these out these are all questions and you can see that the header length is six the protocol length is four here's the destination address here's the source address here's the type right and just like a debugger I could say break two run let me bring up one more of these it doesn't bring up a different one what if I do this oh that's a little better it's not grabbing well it doesn't like it on the end so anyway what what this what this should have done was to transmit these packets on the wireless interface but I believe that by running TCP dump I reset my wireless interface I just saw it do for some reason TCP dumps been doing that lately anyway so what this allows you to do is to set break points you can stop the break point you cannot set packets yet you have help another feature I added recently is graphing so the network time diagram I showed earlier I can actually print out a graph from that using dot so if you've got a TCP session that goes like this it will trace it and label the edges with the IP addresses so this is going to be built up into more of a debugger for this kind of stuff so you know break you can delete packets help list you can do next print run send show you can send a particular packet continue you can graph you can get info on the file and this is reading raw pcap file so this isn't something I the packets aren't anything that I coded up in PCS they were captured from an actual session so any pcap file you can load it in you can read it you can play with it you can have multiple streams and then in the next versions there'll be other things like conditionals yes that's that's 0.2 and I'm at the conference so it's not done yet but I will so what will be nice is you'll be able to interactively modify the packet so you can say tab and it'll give you all the fields one of the nice things that the CLI am using is it has nice tab completion it's the one built it's basically what's the new one read line read line is integrated into this library that's in python this is all in python so so this is why I can do things like when I hit enter it does the same thing again or I can say tab list quit so that's just all in there but it's pretty nice alright I'm out of time any quick last questions alright well thank you very much


One, two, three. Dance is just such a wonderful communicator of expression and ideas and life. And one thing I love about dance is it doesn't discriminate what background you come from, what colour your skin is, what language you speak, because dance is a universal language. Culture Break is about bringing everybody together in harmony. It's about celebrating everybody's uniqueness, regardless of their background or their situation. It's about celebrating them. And we have this statement, you don't become somebody, you are somebody. Five, six, seven. There's freedom in dance, allows people to, there's no right or wrong, allows people with freedom to express what they feel inside. And I just guess, you know, I wish everybody in the world could dance and so have a lot of heartache and problems if we could communicate through dance as dance is a universal language that brings everybody together to celebrate life. Good. You're mingling with Africans, African brothers, Australians. And I dance with all kinds of races and just when we come together, yeah, it's like you forget and it is like a culture, like you end up becoming your own culture and that's the powerful thing about dance for me is that it's brought a lot of different cultures together. Dance to me is meeting new friends, being part of a brotherhood and expressing myself. My life without dance would be boring. I would like, I would be hanging out in shopping centers, getting up to know good, trying to be the person I'm not. We've got one person that's from Kenya, we've got another from Thailand, the Philippines, we've got four of us that have a Māori New Zealand background. We're another from Lebanon. A lot of us boys, that's where we came from, from like a background where we were like some of us were in gang, some of us were bullies. But then when we started getting into dancing, it just sort of put the humble side of us out of it. To download free products to celebrate Harmony Day or to register your event, go to the website. Harmony Day, everyone belongs.


Great. Thank you, Brian. So today I'm going to talk about how research is cumulative. New investigations build on, challenge, or qualify claims made on prior evidence. Explanations for observed findings are often wrong, and the process is made by a constant dialogue between explanations and evidence. It is normal that explanations are wrong and constantly being interrogated for how they can be refined and reconsidered. This is a self-corrective process, and it's a hallmark of the research. And a key part of a healthy self-corrective process is that the evidence used for generating and refinding explanations is credible and trustworthy. But there's concern that there is little opportunity for self-correction to occur, because there's a weak foundation of evidence upon which progress can be built and allowing explanations to be generated that explain phenomena that either don't exist or we have inaccurate explanations to explain phenomena that do exist. And during the last 10 years, a lot of progress has been made by evaluating the credibility foundation in research. And across disciplines, teams have evaluated the credibility of evidence by trying to repeat it. So today I'm going to talk about four different approaches to repeating evidence. I'm going to give examples across disciplines at each stage. And the first one is process reproducibility. So this refers to the ability to have access to the underlying information that supports the claims that are being made. And if underlying data, code, and other research materials are not available, then it's difficult or nearly impossible to evaluate whether the findings are reproducible, accurately reported, or credible. So here's a study that looked at the accessibility of data in articles published in BioMed Central Journals. The most common data availability statement was to ask for the data from the authors. However, when asked to make the data available, many did not respond or declined. And that means just seven percent actually provided the requested data. This means the vast majority of papers being published could not be assessed to see if the findings were accurately reported. And in the reproducibility project Cancer Biology that focused on preclinical research, we observed similar things. So we looked at the accessibility of data, code, research materials, and clarity of protocols. And we also assessed authors' healthfulness in the process. And if accessibility and sharing was high, what we'd expect to see on the screen in very little or no yellow or red. But with every aspect, barriers were encountered to accessing the underlying information of published findings. In ecology, this study assessed published papers for data and code accessibility. So the positive side here is that the policies and norms for data sharing have matured so that 79 percent of articles published have available data. However, code sharing has lied behind. Only 27 percent of those papers had code. That means together only 21 percent of articles made both data and code available, which means only a small proportion of the literature that's being published had the needed output that could enable someone to see if they could reproduce directly from the researcher's work. And these reproduction attempts with only data are more challenging because then if you don't have the code, you have to extract the information from the text, which is itself a very challenging or impossible task. And in the SCORE project, this is a project that we assessed the credibility of overall science literature over a 10-year span that included over 60 journals. When we assessed the availability of data and code in these articles from this corpus, we observed that more recent papers had more content available. And that's because accessibility inevitably erodes over time, but also because preservation and sharing practices have improved during the last 10-year period. But if we split the data differently and look at discipline instead of time, we see a different story. This illustrates that those improvements in economics. So this implies that sharing practices can change through technological normative and policy interventions, but those changes can only occur in communities that deliberately change their culture and implement those practices. Okay, those examples illustrate shortcomings across fields and sharing sufficient information to make it possible to assess reproducibility. There's also a lot of research that actually looks at what's the case when data or code are shared and whether those outcomes can be reproduced using the original analysis strategy. So in principle, we'd expect the literature to be 100% true. All we're trying to do is assess whether the paper accurately reported what the outcomes were of the analysis. What I'm going to show you is no studies ever achieved 100%, and many fell very far short from that. For example, here are three studies in economics that identified a sample of articles that are counted based on how many had available data and codes that they could attempt at reproducible findings. So then they counted from those attempts how many were successful at reproductions of the original findings. So you can look at these two ways. The reproduction success rates were low either way, though, whether you looked at the reproducibility rate for those that were attempted, so that success rate for attempts or for all the articles counting the lack of access as a failure as well, so the success rate for articles. So sharing data and code does not itself guarantee the reproducibility of findings. And in cognitive science, this study examined the reproducibility of findings published in the journal of memory and language under a brand new data sharing policy. So while the policy increased the data sharing, as you can see on the slide, reproducibility rates were lower than desired when code was shared and even worse when code was not shared. This replicates what I just showed you in economics and adds that the situation is worse when data are shared alone. So here are the claims that used electronic health research records. So here they purposely did not use the code. They recreated the analytical strategy based on what was written in the paper. And if reproducibility was high, what we'd expect is all the columns to cluster in the middle where that one is. But instead we see that there's wide variation between the original findings and the reproductions and sometimes quite extreme. And finally in the score project we looked at the reproducibility articles that had access to data or data and code. And if code was not available we'd reconstruct the analysis just like I shared before. And in no discipline did we achieve 100%. Even if we relaxed the criterion to only needing to be within a 15% margin of area so that approximate we still did not achieve 100%. So collectively the first takeaway from outcome reproducibility is it's very replicable across disciplines. The second takeaway is it can change and some fields are doing it. All right, robustness. This is a third way of assessing credibility. And robustness refers to reanalyzing the same data from the original finding by considering reasonable alternatives to the strategy, the analytical strategy. So for example there might be different rules for treating outliers or different decisions about inclusion criteria or different variables to include in the model or different ways to combine variables to measure and we would not expect all of these different analytical decisions to give us the exact same decision, the exact same evidence, but we do expect them to be reliable and robust in terms of what we conclude from those original findings. Here's the many analysts study. This had 29 teams that used the same data set to address the same research question whether soccer referees are more likely to give red cards to dark skin tone players versus light skin tone players. Each team here developed independent analytical strategies to answer this exact same hypothesis. No two strategy was the same and there was a variation in terms of what was actually observed for the findings. About two thirds were significant and positive, that's the green dots you see there. And about one third were negative. Those are the gray estimates. And there was substantial variation from the smallest to the largest observed effects. Here's one that looked at COVID spread early on. Nine teams used the exact same data to estimate the reproduction number of the virus. The estimates varied from one's confidence interval overlapping with zero on the left to one that had a high rate of spread on the right. So in a typical paper the variability associated with this these analytical decisions is invisible. Usually we just get one analysis in that paper. And it's heavily dependent upon which one is picked by the authors, which one they report. So from a single data set like this one might conclude that COVID is either spreading fast or that the pandemic is receding. Here's a last example from neuroscience. 70 teams here analyzed the same neuroimaging data set to test nine distinct hypotheses you can see on the bottom. The y-axis here shows what proportion of the teams observed a significant result for each of these hypotheses. Only one H5 on the right did all the teams consistently have a positive significant finding. While three on the left were consistently not non-significant findings across teams. But you can see that there are five that sit in the middle that range between 20 and 35 percent of the teams reporting a significant result. So collectively these robust studies tell us that analytical decisions are an unrecognized source of variability. For findings and the lack of visibility of that variability is a threat to the credibility of any one analytical strategy. The last indicator I'm going to talk about is replicability. So this refers to using independent data from the original study to see if we can obtain similar evidence for that original finding. And there are many reasons why we wouldn't expect these to be the same. A lot of this is in terms of interrogating what's necessary and sufficient to obtain a finding. And if well powered though a good replication, a good faith replication that does not obtain similar evidence suggests that we don't yet understand the conditions necessary. And that should lower our confidence that we have in terms of the relationship between the evidence and our explanation for the evidence. So for example this is from the reproducibility project in psychology that attempted to replicate 100 findings from three prominent journals in psychology. The X axis here shows us the original effect size. The Y shows us the replication effect size. And in an ideal case, that diagonal line is where we'd expect the clustering to occur. However, we observe a lot of variation between the original and the replication and a lot of it hanging out below the diagonal line suggesting that the replication findings are weaker than the original. And on average it's 50% smaller than the original. Using another measure of replication beyond this we can also look at was it statistically significant in the same direction. There we see only 36% of the replications were significant in the same direction compared to 97% in the original findings. We found similar results in the reproducibility project in cancer biology. So for here we actually saw that the average effect size was 85% smaller than the original. So to put that in perspective, this would be like an original finding. Found a potential therapy that increased the average tumor-free survival in mice for 20 days. It's pretty typical. But in the replication it was three days. So this has a huge impact in terms of the confidence someone's going to have as we're pushing this down that drug discovery pipeline. And finally in the score project here we completed 153 replications across these disciplines. And here replications determined like it was in the psychology project that I gave you as an example, statistically significant in the same direction. We can see that across all these disciplines and similar to the other efforts about half of the original findings fail to replicate using this criterion. So just like the reproducibility studies and the robustness studies I shared with you, systematic replication efforts suggest there's room for improvement if we're really going to be able to repeat the results of the published literature. And this is important, right? It's really important because it tells us the implications we're going to have on terms of what we use these findings for to solve real problems. So here's an example from the ALS Therapy Development Institute. So this graph shows us between the original published findings, these green bars and what they found in their replications, the black. And in all these cases the replications fell well short of those exciting published findings. Also, all of these had disappointing results in human trials. So this leads to wasting time, money, animals in this case to try to build evidence that was not as reliable or credible as it was originally thought. And this also means pushing false hopes into having these go into clinical trials and experimenting when we might not have done that if we had known these replicability rates beforehand. So the evidence that's been provided by all of these systematic efforts of reproducibility, robustness, and replication suggests there's a lot of room for improvement. And if we can improve the foundation of the evidence then we might actually be able to reduce the friction and increase the pace of discovery. So let me give you an example here. This is looking at the drug discovery pipeline from target validation on the left to when drugs actually get approved during the market on the right. If we look at the success rate of all trials across all different disciplines it's a 90% success rate from phase one to drugs that are finally approved. And this only takes into account the failure rates that are known which is only in the clinical trial stages, 1, 2, 3. We don't know anything about the pre-clinical stages so the situation is actually much worse than 90%. So why could this be? Why is this the case? Why are we failing so much? Well, one is it's hard. Research is hard. We're trying to translate the animals we use, the models we use, don't translate to human disease, don't translate to humans as readily. So that's why we have to keep doing investigations and keep further investment in that. So what we do is we're pushing things into that pipeline fast when maybe we should be interrogating the credibility especially between the pre-clinical and clinical divide here. So we're not having enough confidence in these things or we shouldn't have enough confidence to push them too fast. And overall this increases an opportunity. This is an opportunity ahead of us which is to optimize the efficiency of the self-corrective process of science. So if we use methods like I just presented failing is necessary to finding what's right but what we need to be open to is making observing failure possible. Thank you.


The People's Democratic Party Governor's Forum has called on the President, Mohamed Buhari, to summon an immediate meeting of the Nigerian Police Council in a bid to arrest the present security challenges facing the country. The Chairman of the Forum and Governor of Sokota State, Wazeri Aminutambual, made this known while reading the Communique of the Forum's One Day Summit held at the International Institute for Tropical Agriculture in Ibadun, Tambual called on the President to send an executive bill to the National Assembly to amend the constitution to devolve more powers with respect to security architecture culminating in state policing. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


Live from Boston, Massachusetts, it's theCUBE. Covering ZertoCon 2018, brought to you by Zerto. This is theCUBE, I'm Paul Gillin, we're on the ground here in Boston for ZertoCon 2018, the third ZertoCon, and with me is a gentleman who has been to all the ZertoCons as well as many years before that, Gil Levin and I, the Chief Marketing Officer at Zerto. Welcome, thanks for joining us. Thanks for having me here. You were one of the first people at Zerto to understand, way back in the day. Yeah, yeah, I'm kind of like ground zero type person here in Zerto, managed back then the product team and the marketing team, then we grew bigger and bigger, and now I own all of the marketing for Zerto. So how do you feel coming to a conference like this? Are all these people here, you've got big name keynote speakers, having joined the company as one of the earliest employees, what does that feel like? It's like when I have three teenagers, so I know what it means to be proud of your child, it's a very proud moment, okay? Because really, I think the caliber of customers we see here in the conference, of course there's the quantity of customers, et cetera, but also the caliber of customers, the caliber of discussion that's happening here is really, it really makes me proud and knowing that this company started a few years ago and from just like any other startup, a very small company, and now we're really making a dent in an industry and I'm changing and we're making customers successful. It's really, really... Your first user conference, your first user conference is a milestone of sorts, isn't it? Yeah, two years ago it was really like having a child, okay, it was like, so now it's like we got into a rhythm, we know what we're doing now, we know what the conference looks like and we know what it means, but just like with children, when the older they get, the bigger the challenges, same thing, okay? The bigger the conference gets, the different challenges and it's always hectic few weeks, but... But very valuable because you come here, you have a chance to talk to customers and non-stop really for three days, what are you hearing from them? What are the trends that are emerging to you from the conversations you're having out there? So, you know, going back to why I was hired, as one of the first few people, is actually kind of like a good example of that because I was hired because I was actually a marketing and product management guy hired in Zorto before we actually wrote our first line of code. Why? Because we want to make sure that we talk to customers and we get their feedback and we get their guidance on what should be doing back in that day, okay? And so, from that point on, we always are doing the same thing. We're listening to our customers, okay? That's kind of like a key DNA for Zorto. So, this conference is an amazing place to do that, to really hear from our customers what are the challenges? You know, we had our customer advisory board with our partner advisory board here, but also every hallway conversation with customers is the same thing, okay? Hey, what do you need to do? What are you trying to do that you know, capable of doing? And that's where we actually understand the trends in the market, you know? We have John and people like that, the analysts are amazing, but there's nothing like an unbiased conversation with the customer and understanding their needs. And what we see is really two major aspects, okay? And kind of like, as you mentioned in its keynote, one, the SLAs are totally different, okay? It's really, really, really unacceptable to be down or to lose data. Done, it's no industry can do that. No type of customer, no size of customer. It doesn't matter if you're retail, if you're airline, if you're banking. Five nines isn't enough anymore. It's like, it's people have, you know, we all have phones, we're all used to always on, and everything needs to be always on. So that drives, you know, the whole narrative of guys. I can't tell you, hey, I'm going to recover from two hours ago. It's like nothing anymore, okay? The second thing is, they are all facing a lot of complexities in kind of like the changing infrastructure. They all want to move to cloud, we hear about containers, we hear about cloud, private cloud, hybrid cloud, and it's all really coming from the right reasons. It's coming from trying to change the business model. It's coming from trying to change the cross-structure, but to lose so much complexity, so between these two, they just like, they need to really briefing the way they're doing what they're doing in the terms of data protection, mobility, et cetera, and that's where we came to this kind of like, I-to-resilience platform concept, from the needs of the market. So we see customers, I mean, they want more flexibility, they want to use multiple clouds, they want hybrid clouds, they want to have shift workloads around seamlessly, all of this has risk, and resilience in an environment like that is challenging. What is Zerto going to do to make that more seamless for them? I think resilience is double-charging. It's challenging because really to be successful, you can only be kind of like resilient to, hey, how am I addressing bad things that happen? You need to be resilient to the fact that you changing your business is part of your business now, okay? And how am I kind of like effectively change my business, run forward, run fast, while I'm not living behind any gaps or anything that, hey, I might get like, you know, struck by some bad luck or intentional cyber thing, okay? And lose my business. And that happens also. You see major companies that have big impacts on their business because of events like that. So the key to doing that is really to, hey, simplify the way you protect everything and really move to what we call continuous protection. And, you know, that's from a product perspective, but from a thing about from a methodology perspective, you need to have the ability to always recover anytime, anywhere, whatever you need, from realistically, whenever you need it, okay? It's like, it's the only way possible. And only technologies that are, you know, we started from, we're coming from the high end of the market into the, into this kind of like, now more much wider market because we're coming from protecting huge Oracle databases with huge change rates with seconds of RPOs, okay? That's our DNA. That's why we know how to now take that into an entire kind of like, I can say, you need that IT to be available wherever you need it. And you need to be able to protect it at any point in time and move everything around between clouds, as you say. And that's where, that's where our kind of like is our message to the market. You need to be able to do that. Our platform is the way to do that. But that's the only way you can actually not only, as we say, not only survive what's going on, but also thrive in this environment. And that's because you have a converged platform and the time is right. I hear Zerto exactly saying the time is right for convergence. Why is that? It's, you know, when you see the market, kind of like, look today at any player in the back of the RSpace website, all say the same thing, okay? Why are all everybody saying the same thing? Everybody's trying to sell the same thing. Everybody's trying to sell all of these business cases, okay? And some don't call it converged or they do, it doesn't matter. The fact of the matter is from a customer perspective, if you look at any vendor today in this space, all of them are trying to provide you all of these services. So that's where we see, hey, this is what's happening. And the customers are also telling us, guys, we are using, we just heard it on stage now, we are using Zerto instead of our backup, okay? Why? Because in the short term, kind of like retention, you're giving us a much better solution. So we see from the customers that they're saying, hey, I want less systems, I want one kind of like place where I go and I can mobilize, I can protect, I can recover, I have compliance, I have ransomware protection, all of that in one place. So, you know, the market is really telling us convergence isn't happening right there. And that's where I kind of like, you know, we believe we have the best DNA and the best core technology to provide that converged solution, because eventually it's about the atomic engine of how are you doing your protection and I think we have the best avenue there. But if you look at everybody, everybody's talking the convergence game. So where is backup going long term? Does eventually this backup just disappear or everything becomes continuous? Not in a million years, literally, okay? No, pun intended, okay? Because, you know, everybody, you know, the legal holds, the compliance. The attention schedules, yeah. The act of backing up your data, your application is not going away. That's going to stay. We believe that there is a, the shift that needs to take place in the market that we're leading is what does short-term backup mean, okay? Because short-term backup is really kind of like the same thing today as recovering from ransomware, it's the same thing as mobilizing an application. And that's where we think that needs to be continuous and then you need one platform that can also take care of your long-term, you know, months, years, depending on what industry you're in, regulated backups. I just talked to a large customer of ours last night in our great party. He said, hey, I'm doing 14 days and I don't care about anything after that, okay? I'm in an unregulated industry, okay? And I do what is good for my business and for my business, 14 days is good. I don't really need anything. I sometimes have some more copies later, but that's it. So the actual use case of backup depends on the industry, but it's not going away. There's no question that the use case of backup is stayed. We think that the way the technology that drives that should change. As we're talking today, we're two days out from the implementation of GDPR in the European Union. Does this have an impact on your business? Oh, a major impact, you know, first of all, we're a company. Number one, I don't know if you know, International Company, sure, of course. And we work, we have, you know, we work globally with, you know, offices and employees in Europe and customers in Europe and operation in Europe. So all of that. We're marketing, you know, our marketing guy, we're marketing in Europe. We actually had a session here in the conference, joint session by our appropriate council. And some customers talking about how to implement GDPR because it's actually a joint project in Zoto from all sides to say, okay, what are we doing for GDPR? So as a company, that's number one, okay? Second thing is for our customers. There's a number of things that, you know, for example, take locally, knowing where your data is, which is part of GDPR, we help by identifying exactly where is your data, where is your backup data, where is your application data, et cetera. There's lots of kind of like, we believe there's going to be lots of more and less compliance kind of like related things that will emerge for GDPR. And but also, you know, there's like the rise to be forgotten, what does that mean about backups? Do I need to now go open all my backups? So if you can see, and again, not me, but we have people that dove down, for example, the right to be forgotten is one of the less kind of like, the less stronger rights because I think even the regulator understood that, hey, it's a bit hard to forget someone from, you know, do I need to open my back, back up from like five years and delete something? That's going to be a huge cost. So there's definitely going to be implications and I think time will tell you a bit where this is going in terms of like, what are the bigger implications? No shortage of agendas for Zerto going forward. Oh, no shortage, no. Gil Levin, I thank you very much for joining us, Chief Marketing Officer at Zerto. Thank you very much. We'll be right back. I'm Paul Gillum, this is theCUBE. Thank you very much.