Good morning again. My name is Iftah Ian Amit and we're gonna do the down-the-rabbit whole talk today We're gonna talk about Investigating for lack of better terms a criminal server Some technical aspects about it some social aspects about it Some economical aspects about what's behind it even some legal stuff. I'll try to keep it fun and entertaining I'll try to keep it technical enough.  So Bear with me and if it's not fun, just throw something at me. I'll crack a joke or something like that Just a bit about me Iftah Ian Amit in Hebrew it makes much more sense So don't even try to pronounce it Ian is is fine here in the US and everywhere else outside of Israel I'm kind of on my own right now doing some consulting to Three major companies in the past. I used to be director of security research at Aladdin. She's a web security company Same thing at Fingin, which is a web security company various security consulting gigs stuff like that both IT as well as R&D and My spare time and helping hand with the Israeli Air Force during my reserve duty So without further ado, let's dig in first of all, how do you start anyway? How do you even find a criminal server to start looking into and dig up information and do some research to figure out? How do these bad guys work? Well? obviously Over skills you got to be elite you got to be cool and you got to have a lot of luck Anyone familiar with a site called trick cool tips Nothing come on. It's that's the site. It's a forum. It's a forum a for How do they say it tips for better programming? We just happen to stumble upon it seriously scanning the web, you know combing the desert from from space balls looking for sites with some kind of malicious content and You know during some some research we stumble up across this site well, I wasn't familiar with it either until we just you know saw some malicious stuff on it and And then we started correlating The badness that was on that site with other instances of interesting badness that we've seen before Anyone from Italy here? Okay, Federer consumer Sumatori Thank God there are no talents because I'm probably butchering this it's the consumer reports for Italy a Major site a lot of a lot of hits per day was hit with the exact same Badness that we saw on a on trickle chips or whatever it is and we started correlating it'll more and more of those sites looking for that same Badness so to speak and for the same malicious.  Let's dig in. Let's see where everything points to where everything starts And that's where we started seeing interesting stuff first encounter was a site called gwtsdj Eni.com Very random doesn't look really meaningful and we'll see why in a second and Now the thing is it was part of Torpy Anyone not familiar with Torpy? Okay, Torpy is is like a cool Trojan and That spread itself it spreads itself and reports back to domain names generated with some Some form of of a domain generation schema Now it looked real familiar with the Torpy schema, but it had an extra letter You know other researchers Smart and myself figured out what's the domain generation schema for Torpy and it was like four letters J e and I for that specific month or time of the year that we found this but this one had an extra D so Then Bell star ringing, you know light bulb starting appearing above people's heads are like alright So this is a modified Torpy or a new gang Running this scheme and that's where things got really at least got us really interested So we took a closer look Alright now, this is where the fun part starts Working for a legitimate company in most Western countries Don't really let you you know turn your ass vigilante and start attacking those servers to figure out what's running there and and who's behind it especially not if you work for a Public company that's listed on NASDAQ and and stuff like that So the only thing you can do is kind of passively scan look for stuff. That's public Not really like stick your finger too hard and figure out what's up there so with a very basic scan of Predictable names of files and directories that we try to look out on the server. We found a file called in dot PHP This turned out To provide us with this Yeah, I know I know I recognize this Yeah, it's a PHP shell and and Then we're like aha This is where it gets really interesting now. We didn't really do any hacking so far, right? We're just you know kind of lamers looking around But you stumble across It's PHP shell and if you're familiar with this. This is part of the r57 and Shell suite this is just the The command interface.  All right. This is this might be a problem even at that stage Why well, it's not ours.  You just got a shell on a criminal server That's attacking people and spreading bad code So as for the first dilemma, well, we just followed an injector script on a legitimate site so we found out about the domain name about the server legitimately and We kind of ran across a service that was offered by the server wasn't protected no passwords or anything like that That led us, you know using our knowledge to figure out. What's the what's the full shell? R57 new dot PHP Good so far second dilemma How far can we go? That's where lawyers really get messy And thank God for general or console back at the company and the New York lawyers that we worked with and We figured out what can we do and what can't we do? We can't crack things. We can't guess things That aren't public Knowledge start a question. Yeah Well, if it's a hacked server what you're probably gonna do is talk to or contact the legitimate owner of that server These questions are in the context of researching a criminal server and even with that and we'll get to why we're being so careful with With the research itself Because it's got a lot of implication not only on the company, but on on the actual research and its results So basically we can do whatever is being provided to us Whatever is being allowed to us through that shell with the permissions that it has means you can't do privilege Discalation even if it's just glaring obvious you can't do that and you can use whatever tools are provided that are not password protected But if you find additional information that is not protected That can be used to access other parts of the site You can kind of do that and the kind of is is has got some legal fluff around it. So I'm not sure about it. Yeah You you kind of could yeah That's That's as official as I can get yeah, I can't again I can't it's kind of similar to the question that was raised before If it applies to just a criminal servers or to any kind of server that you can investigate or look at Again, this was in the context of researching a criminal operation or a bad operation So again talk to your lawyers my lawyers out on care and figure out the right answer But this is again in the context of this specific investigation So we can brute force we can't guess but if they're out there and provided to us we can kind of use them So thanks to our lawyers and legal department and all those nice guys that that kept me up reading tons of material We actually went down the hole and what we found there Was really interesting Because this goes beyond just some malicious JavaScript or another executable Trojan or something like that We actually uncovered a lot of the behind the scenes a lot of the back-end utilities and communications That the criminal groups and there are multiple criminal groups using that server it's kind of a lease a server or lease an operation and We found it just tons of stuff that that Provided us and the law enforcement around the world a lot of information to pursue those criminal groups Starting with newsploit and there are a couple of talks at least that I went to that mentioned newsploit I'm gonna mention it again because I just you know I spent a lot of time with it and I'm not gonna you know just let it go and Newsploit is a is a an attack framework used to and to spread malicious JavaScript attack clients and infect them with the with Trojans or whatever you want automated tools to Inject legitimate sites with malicious code. All right, because you want to get you want to spread the word You want to get to legitimate sites that are gonna attack their customers their visitors instead of just providing all that Banness from the criminal server and we found a php by admin, which is again very useful For figuring out the back end truck full of Trojans tons of them unbelievable We even found The server was that was also used by the snow a group Which is the why I mentioned before the modified torpig domain generation schema tons of Trojans and actually some some of the Trojan generation Tools back there a double stats logs always fun to look at someone else's logs and figure out You know how active is that server? How many people have visited it? And get some some statistical information from that set up instructions I kid you not set up instructions You get a manual that tells you how to install several parts or different parts of Of the software that is designed to attack visitors and manage those attacks You just get the manual and a mail back end for tracking infections a directory filled with open VPN certificates and Because criminals are secure as well and you need to connect to that server securely All right, so you want to open an open VPN tunnel to it? Instead of just connecting plain text and a huge list of cPanel credentials. I'll talk about this later This was kind of a whiteboard for the several people that were logged. We're logging into the server So one was right down. Oh, look at that cool credential and the other one would say, oh, yeah I got it before it's it belongs to some hosting provider and so on so really interesting and and Some more utilities and exploits and and a funny 15 most wanted Kind of ripoff page hosted on that site again, I'll show it later Start with a tool. I mentioned that at the beginning One of the two interesting tools that we found was called FTP iframer Basically, it's a PHP script Which you feed with a lot of FTP credentials that were stolen using the Trojans that were sitting on people's computers That got infected by and so on and so forth and And just feed it with username password domain username password domain and all it does it is it automatically goes and tries to access that FTP server tries to log in verifies that the credentials are correct once they are it Looks at what is accessible on that server if it's web content great look for JavaScript HTML PHP whatever files and Inject an iframe on to them that points to the malicious JavaScript if it's not look for other interesting stuff All right and set it aside Supports several users so each user has his own login and credentials that he uploads So you can actually run multiple campaigns at the same time. They're separate from each other To cater for several criminal groups, which means that this server again was leased to At least three different distinct criminal groups that we could point out it And the logs themselves were accessible. We found more than 200,000 credentials on That server that's more than 200,000 FTP servers that may or may not not contain Web information web content or even non-web content documents files Applications wherever it is with credentials that vary from read-only to full access The second tool that we found there and and I promised I'm gonna linger on to that a little bit is neosploit Well, Neosploit is my favorite. It's my favorite because it really shows how in the criminal community or criminal ecosystem And there is a supply and demand. There are the software developers and there are the clients and Neosploit has got a long documented history of development from from a very simple exploit framework Through its iterations and and additions of multiple user support to support several criminal groups Enhanced reporting multiple loader configuration so you can say, you know what if you're coming from this area Geographically speaking of the world, I'm going to provide you with that Trojan, but if you're coming from that area There's another Trojan that specializes in you know banking activities In that area of the world and database is improved from fried files to a full-on Sequential database and in version three enhanced licensing again licensing in a criminal tool I can't remember who talked about it yesterday But they mentioned it as well. There is a full-on licensing scheme on this on this thing And it's locked to an IP and IP address and the user password Enhanced installation through a SOX proxy that can only be resolved statically Enhanced reporting on x-plus ri and database management So this is kind of the the 10,000 feet view digging deeper into the tool We were actually able to Gain access obviously all to all all the components and First of all the installation script Went through that before the socks Cool install script by the way great bash programming takes care of all the you know, it's kind of stupid proof You just run it it downloads the The actual CGI the compiled L files and from a socks proc through a socks proxy Sorry That's all the version checking all the permission checking creates the the RC files and logs everything We'll talk about the log in a second This is a quick view of the Install script again using in password or required to even start the installation You may the guy these guys are making sure that your route because Wow Because you need to get there And this is a download URL which again can't be accessed just like this you have to go through a socks proxy This domain doesn't resolve to anything Some error checking the socks and the socks retrieval as I said some some cleanups on the on the Unix system on the Linux system in its scripts everything is ready and Once this is installed, that's it. It will update itself regularly and you're just good to go This is what we got from the logs again. Why is Neosploit? My favorite because it's well maintained you can see that there's a demand out there and that the developers are responding to it and We basically what we did is is we logged all the Major and minor updates basically every time there was a new version or a patch or an update to Neosploit In a specific day we logged the number of times that an update was applied And this is number of times per day. Okay, so you can see that it's it was pretty active during the time where we investigated it There's some rumors that Neosploit was gone when they came back with version 3 1 and after that they kept maintaining it For a short while until they died again But that's that's a pretty interesting graph that indicates on the level of support and the level of updates that you're getting from a tool that you actually bought from And these guys I'm not saying even bad guys. They're just developers. Yeah That's 2008. Yeah. Yeah, the rest is not that much fun I mean it it is if you're like in the zone and Doing reverse engineering stuff like that. It's basically composed of three different parts the demon which takes care of the Database back end interface index CGI is the exploitation front end. This is what the customers or the user see We're all the logic in terms of what kind of operating system you run what kind of browser you run Where are you coming from geographically? And although decision that that decide what kind of exploit am I going to send you and what kind of Trojan is going to be sent and the admin CGI and which is the admin interface where Users can manage all the infections that and report on all the infections that they ran through And some more digging into the the actual tool. This is just the fun stuff again from the the Neosploit key valid validation Going on that they're loading the license verifying the license against a server that's available that the Developers are running on the internet and verifying that the installation is running from a legitimate from an IP that was actually licensed to run from and This this is some more logic in terms of referring the this is the from the customer or the victim end point Getting the hash of the IP and and the browser string Validating the referrer Because they came from a legitimate site and I need to know or document which site actually sent them over here Picking an exploit and encrypting it and sending it back to to the victim This is a little more fun because you can get more data out of it And this is the admin interface again There was a question here about using credentials that are written plain text stuff like that. So, yeah, that's the kind of and This is this was done using the the admin user so you can see all of the users There are configured on the system and using it to spread Badness and all of their statistics and specific like detailed vulnerability a performance Reports and that was back at the time where PDF was like really out there and you know No one patches Adobe stuff and stuff like that So you can see the statistics the success statistics of different exploits and Through this interface if I'm a user of the system I can actually say oh Wait, but that SB active X is no longer like really killing people. How about you update it? Hence the frequent updates that came on later and detailed statistics in terms of per exploit operating system version language and so on so forth Off from the useful any questions so far by the way No, we're good All right, and our script that we found that was again really useful and interesting I mentioned before the domain generation schema. All right and a lot of people were working on kind of reverse engineering the logic of how those malicious domains keep changing Well, that was a That was nice, but What we did is we looked at the source code for the domain generation I mean it was just there on on the server What actually happened was that the the senile group took the Torpy domain generation Script and modified it and adapted it to their needs So it was great for just keeping track of that server Because it was it was just it just kept moving in terms of its domain and actual physical location These are also available and should be available. I think on the DEF CON side. If not, I'll upload them later on And our script that's a pretty interesting for for understanding. How do the bad guys work proxy judge? That's just a little, you know nugget kind of a script It's a CGI were to test where the the victim is behind the proxy or not Sometimes not a lot not always we you know, this logic was only applied to part of the attacks This was applied so that the attack wouldn't get twice to its location that was known to run a proxy Okay for some reason or not Other goodies I mentioned, you know, this is like the killer one the how-to in word It's just a word file sitting on the server with instructions on how to install a specific package So you install run install php make sure the director is writable accessible from the web blah blah blah packer Verifies the integrity change setting check results additional description logging interpreting logs And more fun stuff Downside it's all in Russian I don't know why but uh Yeah, always keep keep a Russian speaking person in your research group.  I think I'm gonna talk about is or finding from the server is the cPanel goodies I mentioned it before again. This is kind of the whiteboard for the criminals to to use for exchanging ideas information stuff like that Hundreds of domains and cPanel, you know what cPanel is. I'm not gonna explain it to you Just you know access access access in line comments in Russian on some of the sections This is like a really really rough translation So but but it's it's still funny like clearly has not been able to look after and that was on on a Credential that They kept like infecting and the guy kept changing it But for some reason they kept getting that password back So probably the guy that was changing the password was infected and it just kept fending back to the criminals and Previously worked as something in Russian clearly no longer works for entries that are dead and shouldn't be Worked on anymore and need to pop in and remove the soap base There's probably like a legitimate word in Russian that that Google translates to soap base that relates to I Don't know maybe an installation or iframe I don't know what too many sites and is not small and my favorite is master admin cPanel That was found right next to a credential that Related to a hosting site address like the whole hosting Provider a lot of fun again. Obviously everyone that was on that list was notified and I don't know if they took care of it or not, but At least we tried Some humor Remember the the top fifth top wanted 15 most wanted thing. Well This was kind of a we couldn't figure out if it was a joke or a teaser to law enforcement or whatever But it was just there and this is how it looked like they took the US Marshals 15 most wanted fugitives fugitives website They ripped it off and modified it to contain their Nicks and aliases and and stuff like that. It's all in Russian and I don't know it's it's kind of funny, but you know now While we're doing this obviously we're not just sitting on this data and Once we got like clearance from from legal and stuff like that we started working with Search CC the reason why we work with search CC and not like specific search in specific locations Is that this thing was just so spread out that it was impossible for a team? I had like five people at the time and working for me and Myself to just you know keep emailing and phoning and and and contacting all those different location We had like 86 countries that were affected by this So search CC came to the rescue a lot of I don't know if there are any search CC guys here You're not fads. It's okay. It's you just work with them and did a lot of you know great work in terms of coordination and And we're very responsive to us They actually created a small task force to handle all and all of that data that was passed along to them And they passed it along to law enforcement as well The analyzed logs and I mentioned before 86 different countries did all the notification process and helped us Again work with the FBI secret and secret service in the US and because there are a lot.  I talked about before we mapped them To the approximate geographical location of the associated servers. They were they were given access to So this is what we got and that's the reason why again we worked with search CC and not specific individual countries this basically covers most of Western Europe East Coast West Coast and and some spots in the Forest Asia Japan and stuff like that and even these guys I mean and you can't just say you know what I'm just going to focus on the US and a few countries here Once you lay your hands on something like that in terms of a researcher. You got to notify everyone I mean everyone's on the same class. So I mean South Africa, Brazil Peru, whatever is there And you can just you know try to phone them all So did we get any closure on this? We did a lot of work in terms of analyzing the tools analyzing the Trojans analyzing the the techniques that Were used to distribute all this all this badness. We worked with the cert To to coordinate the the notification to all the affected sites so they can clean them up and stop You know exposing their their exposing their users to that kind of badness and It took a few good days to get the notification process and after a few good days most of the sites were fixing their code removing the iframes removing the injections, whatever it is and We actually tracked this through the admin interface for new exploit because that's just what we had in place and I'll clarify this this graph Tracks down the top five users that were set up on the new exploit attack framework. We just mapped out the number of Exploits per day that were served and Gilded an installation of the Trojan Throughout the period of I don't know what three weeks or something like that This is the aggregated one with annotations this is the time where we notified search cc and Broke the news and kind of start calling people Shaking down, you know, whatever it is fix your site. This is vulnerable. This is a problem. You're infecting your users and and all that Stuff and you actually see a decline and we're kind of great We're doing some good job here and we're working with with cert and people are fixing their websites and you know We're finding the crime and Yeah, we're kind of vigilante, but we did it the right way And then this happened What we didn't do is that we did not break the business model These guys get credentials all the time these guys just keep you know Promoting and marketing and getting new sources of infections every day And they figured out that if their stats are down They just need to hit refresh on some of their sources and that's what they did So you see a quick peek in terms of site infection that yields installations and then a dip after A lot of the sites fix it and you just get back to the levels of you have a solid install base so to speak of sites that distribute your malware and afterwards even a Greater rise after someone got pissed or something like that and and start working harder So we didn't really get full closure on this We did get a lot of information on how stuff works We did make a lot of ties between the different providers of the software that we found on the server and We got a lot of insight on how criminals actually work in terms of leasing or providing access to Sorry running a single server that caters for several criminal groups with with the segmentation between them and stuff like that But we weren't really happy with this Now I did promise a McCollough connection anyone familiar heard of McCollough Yeah, all right, we call it was shut down happily and It's an ISP here in in the in the US and a lot of bad stuff happened there and this was before my color was was shut down and You probably recognize this Joker by the way is the user that managed this server and This is us looking at the the PHP shell running W And seeing that someone else is logged in to the server at the same time that we're looking at that PHP shell and This is as close as I'll get to like a criminal in real time and right after that Where's a phone call that I've made? To tell someone to dude Work something out. It's out there. This is you know after all the notification went out and The law enforcement was actually in play in terms of tracking down the server and stuff like that. So But the interesting stuff about this is that the IP address from which the user is logged in Actually belongs to McCollough. So that was just another layer or another evidence In the heap of evidence that that gathered up that helped close down McCollough. So Joker was Helping McCollough get get shut down Now if we are getting close and personal as we did before and We did find something interesting some of the some of the applications weren't just publicly accessible to everyone Okay, and some of the more sensitive applications that manage infections and stuff like that stuff like that We're not just open to the public And they were protected with an HD axis on the web server now HD axis was configured with specific IP addresses that Helped again. This is information that was passed on to law enforcement help them Point a finger in terms of oh, so it's that group That's working out of Denmark and that group working out of DC and Newark and Russia and whatever it is and Which is pretty helpful if you keep that on your server and limit access to To specific users you're basically outing them if that server get calm gets compromised now everything that I Mentioned so far and covered so far talks about cyber crime. All right. This is the intent of getting You know John dough and Jane dough infected on their home PC a running some kind of Trojan That's gonna key lock them and sell all send older credentials and and and perform financial fraud and banking fraud and stuff like that but What's the link between these kind of activities and cyber warfare well Yesterday I was at the talk and the cyber warfare talk by Jason And I promised him some some interesting findings from from my talk. So here goes now What I'm going to show you is the only thing that that was kind of allowed to be shown publicly And this was found on the server itself remember when I said that the FTP processing tool was going through credentials Validating them looking for web content looking for other content. Well, this is other content other content is PDFs Word documents Excel files executables whatever one of the sites that were infected or breached and Who's sorry PDA FTP credentials were breached was actually sites that belong to some company that manufactures interesting stuff And this is what we found on the server in a directory associated with the content that was downloaded from that FTP site Now if you don't see that really clearly, this is like a map There are a few items here. There are descriptions over there on the selected items that that you're looking at And if you don't see it really clear, I've kind of magnified a few and this marker, which is placed here is an F16d with a position in 12 digits long to long lat This is it and if you're familiar with this little triangle then shut up and And some some log data report data 864 anyone knows what's 860 patchy F16 they're doing something here. All right. Now. This is just a screenshot Okay, that was sitting along with the application That's running this thing with documentation with data with stuff that I was appalled When I saw there and But that's there and guess who's gonna be the customer of that kind of data. All right It's not Raytheon or any and willing like that. It's a bigger organization or a government that would like access to this kind of information so some final words and Why we should what should be we we should be looking at in terms of Advancement in Trojan technology. We all know that the classic Trojans that communicate over at CTP Send their data Perform a man in the browser attack lurk around look for interesting stuff and send it over well mostly communication because most of those Activities can be monitored and Signatured so to speak to be identified later on and alerted upon What if we apply the You know kind of the communication Mechanisms of web 2.0 to Trojans Well, this is an idea that has been run around in the industry for a while. Everyone's been trying to get their Heads around this in terms of how to cope with this problem. Fortunately, I haven't seen anyone use this yet but again the classic communication was Trojan command control center and HTTP back and forth and the most sophisticated method of evasion was Moving that oops moving that command control center around so it will be hard to catch now What if I send the commands from my command control center through? Legitimate channels blogs, right? I can't blog a block access to legitimate blogs like blogger and wordpad and stuff like that And I can even split the command to different parts so They would be reconstructed using Web applications Google gear Yahoo pipes Google mashups Microsoft spaces whatever it is they all provide you a very easy API a very easy programming environment to process this kind of data RSS web data and modify it adapt it and basically send back or provide a converted or a Programming you know combined view of of the three different parts of the command to one actual command now again This is a legitimate site Hosted at Yahoo Google Microsoft whatever and the Trojan communicates with that not with a command control center same goes for Reporting back to the command control center just posted onto a blog. You can even encrypt it By the way, you have if you have if you want any homework try to look for blogs That make absolutely no sense Not like language-wise, but just gibberish. All right. Guess what? This is a covert channel used by whoever To communicate between two parties over the legitimate parts of the internet and just posted there It's you know, it's got the most uptime ever. So It's just easy final final words and one thing we did think about in terms of Tracking and and taking this a step further, but we didn't really have the time or the patience to do mostly legal is What if we can plant a bug? All right, this server will keep moving around it started when we started looking at it It was hosted in Argentina and then it moved on to Florida It stayed there for a while moved over to DC. It's really easy and really cheap to move this kind of server around now What if we had the equivalent of low jack for a crime? We could plan something on the server like a web bug or whatever it is and just google it or look for it and Figure out where where is it hosted next if we lose track of of the server? and we actually thought about it, but as I mentioned before had a little problem with legal because axis has to be granted and And We didn't get like right access to to the server and the second problem was working You know as security researchers that work for the lighter side Of things is keeping chain of evidence if you tamper with the data if you make any changes or modifications You basically break the chain of evidence and it can be used by law enforcement To prosecute and can't be used as evidence to to look at these groups So that was kind of the last part of us thinking about tracking these guys That's it for me for today. We're right on time. I think and if you have any questions, I'm gonna be at room 103 Q&A room right outside


Christmas season is just around the corner. Have you started thinking about your gifts for your clients? We're going to talk about some ideas that might spark your creativity today. Hey everyone, it's Leanne from leancolderwood.com and you've probably noticed the Christmas music now playing in every store and coffee shop that you walk into, which means we need to start thinking about ways of recognizing our clients at Christmas time. It's a huge, huge task. More so because of the volume. There's a ton of stuff going out at once and you want to recognize as many people as possible and make sure that you don't miss anyone. So I'm going to go through some things that might help you with your Christmas gift giving this here. And these are simple, inexpensive things that you can provide to clients to let them know that you're thinking about them during this holiday season so that when they're ready to start planning their programs again in the new year, you are top of mind. So here we go. Idea number one, the handwritten note card. I believe this is a lost art and I really wish it would come back. I know that I feel super special when I receive a handwritten card in the mail. And if I forget to say thank you for the thank you cards that come my way, I'm saying it now, thank you for the thank you cards. A handwritten card goes a long, long way. And it is one of the easier things to do at Christmas time because it's not that expensive, right? It's a simple card, but it's very personal. So you can put a note in there that's personal to the client or simply sign in your name at the bottom of the card is at least something you can do to let them know that you are thinking about them and you personally touched this card before it landed on their desk. So thank you card, this is still my all time favorite. Idea number two are physical gifts. Now I've done posts in the past about physical gifts. One of my latest posts is up here and you can click on that to view later. But physical gifts can take a lot of different forms. I think that the goal with a physical gift though is to make it as personal as possible. I know a lot of hotels do cookie drives and things like that, but in all honesty, with food allergies and food sensitivities, the cookie drive is really tough to pull off successfully because not all your clients are gonna be able to enjoy those cookies. So really looking at ways to make it personal to the client I think is key when you're looking at physical gifts. Another place where you can find some gifting ideas is on my Pinterest board. And so I'm gonna put the link to my Pinterest board below. You can link on to that Pinterest board and see a ton of small but simple client appreciation ideas that maybe you can work into your plan this year. Idea number three, if their program is around the holiday season, I'm thinking even October to December and a lot of programs are in October, November, December, a welcome amenity in their room when they arrive for their program with potentially a little Christmas gift card is a great idea. It then saves from you having to double up and give them something again in the Christmas season but letting them know that you're thinking about them and you wish them well in the program. It is almost like killing two birds with one stone but it is a very welcome sight when a meeting planner walks in the room and they see that they've been recognized for their hard work. And again, with a little Christmas card, it's perfect, it's idea number four, the one night stay gift certificate. Hotels love doing this and with great reason. A one night stay in the hands of a meeting planner is a great way for that planner to experience the property, experience the amenities, take a look at the meeting space, do all of these great things in your hotel and then take that great experience back to their delegation and to their decision makers. But here's my cautionary tale with the one night states is meeting planners often cannot capitalize on the one night stay. Number one, they might not be local. So getting from Toronto to Vancouver for one night, it's just not feasible. Number two, it's a time consuming task for them to do this. So I'm certainly not implying that you should be giving people flights to and from Toronto for a one night stay in your property but certainly look at some other creative ways to have people capitalize on that one night stay. So a couple ideas that may work is to offer the one night stay as a pre or post night on an event they're already doing in the city. Perhaps there's a meeting planning conference that is in your city or destination that they can come to beforehand and experience your property. Maybe they do business travel in your destination and they can use that as part of their business expenses and stay at your hotel for the night. Or perhaps it is something of a customized fam or at least maybe three or four room nights at different hotels in the destination to make someone coming from coast to coast a little bit more palatable so that they can enjoy an extended period of time and of course see your property at the same time. So those are a few quick ideas. The one night stay is certainly still a great idea and a great plan. Just make sure that it's flexible enough for the planner to use. Idea number five, a letter of commendation that's given to their boss. Now this one's a little bit out of the box, I get it but what an incredible gesture if you were to write a letter of reference or a letter of commendation and give it not only to the planner but make sure that their boss or supervisor receives a copy of that letter. Put it in the mail with a stamp and send it to their boss. It is an incredible way for that person to be recognized in their organization. And let's be honest, meeting planners work hard and there's a chance that they are not recognized for all of the hard work that they do. So this is one way that you can make sure that they are recognized in their organization and that could potentially further their career just from your one simple act and gesture of providing that letter of reference. So give it some thought. It doesn't have to be at Christmas but it certainly is something I think that all meeting planners would appreciate knowing has happened and when it comes from your hands again, that's gonna keep you top of mind. I hope these ideas have sparked some creativity in you this holiday gift giving season. And if you would like more tips on how to attract a meeting planner's attention please download my checklist over at leannculderwood.com. That checklist has everything from gift giving ideas but also on how to leave emails and voicemails that'll hopefully get meeting planners getting back to you more quickly. Thank you for watching this week's video. Have a great holiday season and we will see you next time. Bye for now.


With that introduction, I'm almost obligated to tell you that by the end of this talk, I'm going to have each and every one of you pulling out a checkbook because I'm going to be selling an amazing product that's going to change. So I'm Daniel Fisher. I'm from Microsoft Research. I'd like to thank Domenicus for allowing me to borrow his laptop. I'd like to demonstrate that even when I touch it, neither the laptop nor I catch it on fire. It's really cool. I'd be delighted to talk with any of you about Microsoft Research's role in the world, what we do. But in general, I look something like an academic researcher. My day-to-day work is to get to understand problems out in the field and to think about solutions. And my first encounter with Big Data actually happened something about 2007. And the story begins, as all the best stories do, in a server room on Saturday night at midnight. I had been sitting—oh, was that again? Okay, so they do catch on fire when I touch them. There we go. My story begins, as the best stories do, in a server room on Saturday night at midnight. I had been talking to a colleague of mine who worked on what was then called Microsoft Virtual Earth, but would eventually be called Live Search Maps and eventually Bing Maps. And we were having a conversation about how users use the system. He sort of believed that people would go check out the whole diversity of the world. There's a big role that's full of sky imagery, and gosh, they'd be checking out mountains and forests and oceans and plains, and I, of course, thought this was a really interesting question. I'd love to find out. So he spoke with him and his team, and he introduced me to a program manager who got me in touch with Dev Ops, who put in a change request. And, you know, it seemed like sometime in six to eight weeks we'd be able to put in a spec for the way that the data would be transferred. You've all been through this sort of story.  And a guy from the server room says, I was running the backups and I realized I can take care of this for you. Go check this share over here. And I went to the file share, and there was 20 gigabytes full of zipped compressed files. Internet service logs showing me precisely the logs of what tiles had been downloaded when by whom. So I have, for each of them, a date, a date and time. I have a URL. I have some other stuff. Because it's a tile-based system, you can look at the URL to figure out precisely where it is. So this is actually the address of this tile that is 02123 and so on, is a quad grid that tells you precisely where this location is. So these couple dozen gigs, unzipped to about 100 gigs, kind of overwhelming my hard disk in 2007 anyways. And then a couple days full of pro scripts later and a week of SQL server churning away. And I finally had something that I could actually present, which looked something like this. This is a map of where people are looking at when they look at virtual earth. It's a log scale. The brightest spots are very, very bright with hundreds of thousands of hits. The dim spots are very, very dim with approximately zero. The spot on the corner shows you what the size of the tiles that we're looking at in this particular image are. So these are people who are looking sort of at this, I don't know, 100 mile or so range. And as you can see, people tend to look at, well, English-speaking countries, because that's where the product had been rolled out. And they tend to like to look at cities and they like to look at coastlines and they like to look at islands. And they like to look at this bright spot in the northern Atlantic Ocean, or the southern Atlantic Ocean, right under the Cape of Africa. Anyone? Zero zero. Zero zero, precisely. It turns out that you have just found a bug in the JavaScript control for virtual earth version one, it was correct in version two. If you typed in a location that the system couldn't find, it would fail and drop you into zero zero. And that bright spot is users winding up there. And the tails that you see extending in all directions are the frantically scrolling around, trying to figure out where they are. I know there's a bit of a Seattle contingent here. The bright spot over there on the left side is downtown Seattle, and you can see that people like to look at downtown and they scroll around to the space needle and the university on the other side of the water. You can see Redmond and Kirkland and Bellevue and a bright square. Anyone know what that bright square is? That's Bill Gates' house. And the reason that it's a square is because the slash dot effect. Slash dot had put out an article saying, now you can spy on Bill Gates using his own tools. Click here to see Bill Gates' house. And so people would click over there and their screen would fill up and they'd look at Bill Gates' house. And as you can see, they didn't scroll around. They did not go load the tiles near and see who Bill Gates' neighbors are. They said, okay, I've seen Bill Gates' house and hit the back button. And that's the visible evidence of it. Southern California got the same sort of effect. For those of you who are considering vacations out to Southern California, I'd like you to notice up at the top, we see all sorts of beautiful bright spots next to each other. Hollywood and Santa Monica and Los Angeles and downtown. And locals can, I'm sure, tell me much more about that. And then there's a very, very, very long stretch before you see one other bright spot. It is the brightest spot on the virtual earth map. The single most looked at place on earth is also, of course, the happiest place on earth. That's Disneyland. But when you bring your family there, please be aware that you're in for a one hour, very, very boring drive. And you can see it all right on this map. So we were getting real feedback and real information, both about, as I said, how people were behaving in the system. This bright, again, here's a zoom in on part of Central United States. You can see Chicago on the right side. You can see on the left there's Salt Lake City and Denver, Colorado. And that very bright cross in the middle is the geographic center of the 48 United States. And when you started up virtual earth in the United States, you would get BUS right there. And if you were to pan or search or zoom, you'd look at the various places that you wanted to. And if you were to just grab the zoom knob right over there and shove it all the way to the end, you'd find yourself right in the middle of the, I think, the Snake River Indian Reservation, which has nothing much in particular to look at. The imagery was terrible. And you can see the cross as people, again, try to relocate each other. And I think that little arc on the side was actually them playing with calibration games on precisely where they'd place the screen based on resolution. So it brought this to the team and they started making the revisions. And they started learning about how users were using their system. And everyone is very happy until one of the people said to me, by the way, can you distinguish American users from international? I thought about that. That's actually fairly straightforward. All you have to do is look at the IP address, which had been in my original logs because anonymization had clearly failed badly. Anyways, I could have gotten that, I guess, but I realized, well, the way that I'd architected this thing is, of course, I started with raw data and I'd filtered out all the fields that I didn't, or I'd dropped out all the columns that I didn't need and I'd filtered them. And then I had this process of aggregating it on the server side and building indices. And from there I was generating shapes which I was able to serve up as tiles finally to the client. And the problem is that first step, I was doing pretty much by hand. I had a Perl script that was running it all. To do this distinguishing between American international users, I'd have had to go back to the very source files and those very earliest Perl scripts and do a reprocessing and be able to expose more data. And of course, once they did that, they'd ask me about the next dimension and I certainly didn't have that one available either. So I was going to be making a very difficult world for myself. And I think that this story illustrates in a microcosm some of the challenges I'm going to want to talk about in big data about where you have to make decisions and how this looks a little bit different from some of the exploratory visualization that we otherwise so like to talk about. Now the definition of big varies. In 1975, I was being born, but this guy was giving a talk at the SIGMOD conference where he said, sorry, at the VLDB conference. This was the very first VLDB conference and databases were not nearly as big as they are today, but still this is a substantial size. 200,000 magnetic tapereels for the US census representing 900 billion characters of data. That's 900 gigabytes. That's not bad. But what's most interesting, 200,000 tapereels. Consider what that means. If you've got an algorithm that's, you know, just takes order of n through the data, how many order of n? Four passes, six passes. Really every pass above one is a lot, and even one pass, looking at every single tapereal is a horrifying and terrifying idea when you've got 200,000 physical tapereals that have to be picked up and moved, the robot head going completely nuts on this stuff. Just a couple years ago, Ben Scheiderman gave a talk over at, this one was at SIGMOD, thinking about what he called extreme visualization, squeezing a billion records into a million pixels, and he talked about a number of techniques. Certainly by then, we knew that a billion records wasn't actually a huge deal, but it was certainly, but it gives you some idea of where the visualization community was considering to be big. I want to sort of step away from any particular size because I'm sure someone right here is going, ooh, but we've got this exciting new technology that covers 10 billion, fantastic. At a zero, it falls down. At another zero for if it really becomes a problem. Instead, I think that we want to talk about cases where the size of the data set is actually part of the problem that has to be solved in visualization. We just had a fantastic conversation about performance, but that was all rendering side performance. And on the rendering side, we have certain limitations. We, for example, unless you're sitting inside this room downloading 20 megabytes to your screen isn't actually a huge problem. So I'm going to want to talk about two different parts of what makes big data challenging. One of them is the representation problem itself. What do we put on screen? How do we represent? What visualizations do we choose in the cases where we've got billions and trillions of data points? The other is interaction. What do we need to do to make visualizations useful for interaction when we're dealing with these very large scopes? Of course, as we're talking about this, we're not only talking about user's time and attention and programming effort, we're also beginning to talk about money. A lot of these things are beginning to happen on distributed systems now, and what that actually ends up meaning that if you run a query that runs on a couple hundred cores for a couple of hours, you've spent an awful lot of money just keeping the system spun up. And that's an interesting world. I don't think any of us, at least I certainly have never before encountered a time when I actually was spending money with my computer, but I now definitely am. Speaking of which, I'm not talking in particular about any technologies today. There's going to be no code on any of these screens. I'm talking about concepts and thoughts that I've been working on. But that said, I just came back from the O'Reilly Stratoconference a month or so ago, and everyone there is talking about Hadoop and the Spark stack and various competitors and permutations of them. There's a lot of really exciting things going on in big data and distributed data systems that are really worth paying attention to. And I know a lot of people in this room are very much focused on sort of the front of the system, and there's a lot to be said for what's going on in the back, too. Now, when I'm talking about size and some of the issues going on, I've got order of magnitude a million or so pixels on my screen. Maybe you've got a retina display and it's 10 million pixels.  We're not adding too many more zeroes after that without, like, overwhelming both the human eye and physical capabilities. And that means that if you've got enough data points to fill up memory, 10 to the 9th or so bytes, you're going to have to make some decisions and some sacrifices. And if you're filling up your hard disk, 10 to the 12th or so, you're going to have to make even more. So I think that it's safe to say that no matter what we're doing, we're going to have to have some conversation about more sophisticated visualization techniques then put every data point on screen and call it a day. I mean, we can try. Here I've provided for you a big data scatter plot. And a big data network diagram. And over there, I tried it in parallel coordinates. So there's some limits. We're going to have to aggregate. Fortunately, people have been giving a lot of thought to aggregation techniques. There's this lovely paper a couple years ago from Nicholas Elmkvist and Jean-Denis Fiquette thinking about what they call hierarchical aggregation. But in general, it does a fairly good job of outlining what do we mean to be getting at when we say that we want a scatter plot and therefore what's the aggregate form of a scatter plot? What's the aggregated form of line chart or a bar graph? Some things are pretty obvious. This happens to represent a couple hundred thousand points underneath, but you have no idea. We're showing some averages and some confidence intervals and some values and it's a sum of stuff. It works fine. Sometimes aggregation should take some thought. So for example, we often see stock tickers that look something like this. This is the stock price for a large company which I'm fairly familiar with. You can see some changes in fortune and you can also see some stock splits and some highs and lows and that kind of thing. The critical bit that I really want to bring out about this is that when I pulled this stock trend, I got some sort of numbers. I think that's probably average per pixel maybe or per month or something. So I manually went in and I pulled instead the actual minimum and actual maximum value per month for each pixel so that I can actually and placed one pixel per month here so I can actually say precisely what the values are and you see that this is actually a noticeably different chart than the one above. The highs are higher. The lows are lower. The shape of that curve is a, there's times when you can see a lot of variation. You can see there's times when there's not much at all. So choosing your aggregation as opposed to picking some hidden default can be a very useful way of starting to think about what your visualization is actually trying to show you. I actually pulled this image from Jeff Hare's immense paper and thanks Jeff. Again, he's got one of those big data scatter plots and he's also shown what it looks to bin and bucket that data into a hex grid and you can see much more texture in where the data actually is when you go ahead and bucket it. Lots of techniques however open to aggregation. Martin Wattenberg's pivot graph mechanism was meant as a summarization of network visualizations and I think this is a criminally underused system for a lot of network visualizations I've been seeing. We've got two different attributes. One of them lined up vertically, one of them lined up horizontally and the thickness of the edges represents the number of connections that go between things of those attributes so it's a way of exploring the forms of heterogeneous connection inside a network. We're all fairly familiar with stream graphs. Stream graphs can be fantastic for our fundamental form of aggregation and even some of the more exotic forms. Tree maps turn out to be fairly good for aggregates and I say mostly because we need to have a conversation about precisely how you store your data to represent a tree map and there's when I started thinking too hard about this all sorts of interesting things started coming up but in general a tree map is simply a pile of pie charts stacked on top of each other and so you can in fact build that as an aggregation. Lots of people have independently reinvented this concept of what's turning out to be basically a generalized histogram. Hadley Wickham has a lovely paper called In Summarize and Smooth where he walks through essentially this technique. You're going to choose some sort of bucketing across your data, some sort of range that makes sense. You're going to take points, put them into the buckets and then you're going to create some sort of shapes on screen based on those buckets. Every chart that I just showed you is in that critical sense sort of a histogram-y kind of thing. But here I need to take a moment and distinguish between exploratory and presentation visualization because everything that I showed you so far applies to both exploration and presentation.   And to do that I'm going to tell you the story of Walt. Walt, the hypothetical histogram. Walt wished to exist. Not in a simple world, but in a world with lots of data. He was large, he contained multitudes. He sounded his barbaric yelp, which is why he was named Walt. Walt's world though is a difficult one. Walt's in a world where data sets are very, very big. Where queries against his data set take, let's simplify the story to say precisely one minute. You do one query into his Walt server, it goes across the distributed system, highly optimized, does all the right questions, distributed across Hadoop, spins up the right cores, gets everything done. One minute later we get back a result. You've got another question. You send it off to the core, it goes away, comes back another minute later. And this is actually kind of how a lot of Hadoop and Spark clusters are working today. So when we wish to first bring Walt to existence, that first moment, well what do we have to do? Well we want to do some sort of, we want to do some sort of bucketing. And so to do that, wow, things are coming in out of order. First we need to find out what the minimum and the maximum are. We need, after all, to be able to figure out where those buckets go. It wouldn't do to be placing points into buckets based on the assumption that Walt ranges from 1 to 10, and suddenly see 100 million somewhere in there. So we're going to have to do one pass based to figure out the minimum and the maximum. And that's all that that pass is going to do. I assume that we're all clever enough to at least do that in one pass and not to do one for min, one for max, that would be a pain. Then the second pass based on that minimum and maximum allows us to bucket the points. So our total time to bring Walt to birth, the first time is two minutes. But of course, we like interaction. We like to interact with Walt. And maybe you want to, for example, change the bucket count. We already know Walt's minimum and maximum, that's good news. So pretty much we're still going to have to go back through and re-bucket every single point. We have to look at them and say, ah, you know, this point, you know, this point now goes into this new bucket and this point now goes into this new bucket and this point now goes into this new bucket because you've got new buckets. And so we've got a new Walt. I mean, maybe we were clever. Maybe in advance what we did is we said, you know what, the user might want 20 buckets, 30 buckets, 50 buckets, 80 buckets, 100 buckets, 10 buckets. Go find ourselves a maximum common denominator, you know, say 3,000, and we'll say, well, we're actually going to pre-bucket everything into 3,000. Now when the user sits there with their slide or dragging around the number of buckets, we don't have to do a re-analysis. We can just grab these new values. So that cleverness pays off. The user now has this incredibly rich experience. They, on the front end, are seeing of, you know, the number of buckets they want to. On the back end, we've thought of it as a large number, a couple thousand buckets, and everyone's reasonably happy. And then we realize that we want to add a new sort of interaction, too. We want to cross-filter with another histogram. This is going to be a little bit of a challenge, too, because in this other histogram sitting off to the side, well, you know, when you cross-filter over here, you then want to light up the buckets over here, the parts of vault that apply to that histogram and don't light up them. So you can, again, do another pass. Or, again, we could have been very clever. We could have predicted that users might want to have these two histograms interacting. And so as we are bucketing things in advance, we actually sort of did this multiplicative combination of all possible ways that dimension one and dimension two could have interacted. If we did that, if we were very clever in advance, again, we've got this interactive. We've got this data pre-cached, and we can interact with it directly. But we're getting to a lot of cleverness. If we do both of these techniques, the thousand buckets to make sure that we're ready for different denominations of the slider and the multiple combinations, we're beginning to talk about pre-caching a couple million items. And there's a lot more interactions we wanted, a quick lens. Some of the things that we often want to do with histograms include changing the number of buckets, zooming in on a single bar and breaking it down into more bars, filtering out some data, cross-filtering into other visualizations, cross-filtering from other visualizations. You often want to look at a couple sample rows from the histogram. Grabbing all of these in advance and making sure that they're all ready to go means we're getting pretty close to just actually having collected the original data set and remember our premise is that we can't possibly do that. So we're going to have to make some decisions and some trade-offs. Now, there is a technology out there, I should say, that is actually designed for precisely the scenario. Nothing I'm saying is new. There is nothing new under the sun. OLAP is online analytical processing and then the open-source world, as far as I can tell, the major vendor is like the Pentaho line of products. OLAP is essentially a cubed version of the data in advance, pre-calculating a set of dimensions that you're interested in and putting them together into a form where you can do a series of high-speed queries against those subsets. And so OLAP can pre-contain a lot of this cleverness and can get you much of the way there for these. But you then have to still have made a number of decisions about what are the dimensions that you're going to want to cube and again, if you over-cube, you've broken yourself into so many small pieces you're not actually winning anything. So the moral of Walt's story, if you will, is that we have to decide what operations will support rapidly and which ones we're willing to allow to be slow. We're beginning to outline as we talk about this a bit of a solution space. When we're talking about these big data problems, things are really nice if we can pull a chunk of the data to work with offline, because anything that's offline, anything we've already got in the browser, anything that's, you know, close to us in the server in a cached chunk, we can be very quick with. If we can't do that, we can at least build ourselves really good indices that have the data pre-stored in some useful form, whether that's OLAP or whether that's Jeff's immense, sorry about the typo there, or the nano-cubes project. We can decide that we're only going to deal with the subset of the dimensions, my solution from the hot map project, giving up on certain types of the data. That gets us somewhere. There's another set of solutions, too, the divide and conquer world, as exemplified through Hadoop is trying for this, although, again, we run into timing and speed problems. And there's one other genre of solutions that I want to talk about, which is sampling and streaming. So a couple of years ago, annoyed that the big data queries that I was doing at least were beginning to turn into, well, you know, this technology, the sort of thing where you drop in a bunch of data into a system, come back the next day and find out what your results look like, which actually meant that you were at the point where doing a query might mean that you'd ask, hmm, I have 10 days until my deadline and I have seven queries worth of time to do this between. That's a bad world to be living in. Especially because I knew that a small subset of my data would actually have most of what I wanted to know. I started to think about trading accuracy for latency. This image is adapted from Hauerstein, who did this first and better, but I independently and accidentally reinvented it until one of my database friends said, you should look over there. Joe's doing this stuff. It was great. It really was great. I love finding people who are doing the right thing. So standard databases in some sense have this notion that time passes for a while and then suddenly you have all the data you ever wanted. Your happiness chart can be thought of as like going along and then very suddenly there's a spike. And yet we could be living in a world that looked like this because very often looking at a small chunk of your data can make you very, can make you almost as happy as looking all of it could be. So if we could trade accuracy for latency, we'd be getting somewhere. And well, you know, I'm an impatient kind of person. All of these, we're talking about basically doing samples and the sorts of samples that we're all familiar with and using those to compute confidence bounds. And to give you a demo, I'm going to actually show a project that we put together a couple years ago called Sample Action that tried to explore this idea. We're looking at the FAA Flight Database and we've, let's see, determined date of week versus arrival delay. And we can rapidly see that after looking at like what, .002% of the data, we've shown that flights on Monday are 1.7 million minutes late. That doesn't sound quite right. It may have been stuck on delays, but 1.7 minutes on average seems high. Wait, I did, I did some before. Let's fix that to average and rerun. So having looked at virtually none of the data, I realized I was doing the wrong query and I fixed it. I look at virtually none of the data. Again, I can quickly see them getting, oh, single digit answers. Okay, that's the sort of delay that I expect to be seeing. That seems roughly right. Let's let this tick forward for a few moments. And now having looked at 0.2% of the data, I can already tell you that flights on Friday are going to be much more delayed than flights on Monday or Saturday or Sunday. So, you know, it's worth spending that extra night in town and not spending your night instead sitting on a tarmac. We in fact brought this to real people at Microsoft working with real data, a guy from the Xbox team, someone who was analyzing the Twitter fire hose, a guy who was working with backend server logs, trying to understand how dealing with this incremental data would actually change their lives. Was this helpful? Maybe they needed four digits of precision every time they did things. Here's what we learned. We learned that most queries that most people did most of the time were the wrong question. They were asking the wrong thing. They dropped in the wrong data. They looked at a dimension that wasn't meaningful. They forgot about a significant factor in their data. They forgot to filter out something. But that's okay. They carried out lots of queries. They cut them off early. Our users became fearless, which is what we want to see about it in exploratory data. And they were fairly happy with getting approximate results. It turns out that most numbers that they wanted to were rough. That they wanted to see were rough. As one person explained it to me, if I see one thing all the way up here and one thing all the way down there, I know they're very different. If they're about the same place, I don't want to cut one off or cut the other off. I'm pretty much in the right place. There was one interesting insight for us on the back end, which is that randomness in databases turns out to be a real pain. Our database architecture today really desperately wants everything to be built in terms of indexes and optimized orders. And we just said we want to be the opposite of that. We want to be as random in order as possible, so you can get as good a sample as possible. I thought I was alone in that, but it turns out that the nice folks at Amplab and MIT CSAIL have built BlinkDB, which is beginning to play precisely with this idea of approximate summarization of databases. And Spark Streaming is now beginning to allow some of these same sorts of incremental computations that we've been playing with. Turns out, of course, that you need to start thinking about new technologies to support streams. What we ended up doing for this, for a number of our visualizations, has been dealing with basically digging through old 1980s literature on how you dealt with old systems then when data was big and memory was small, and trying to remember what sort of technology we knew about, like the reservoir sample, where you keep a sample of K elements from the data, such that each element has a K oversized chance of being kept around. A reservoir sample turns out to be a fantastic way of keeping a fair sample of a data stream as it's coming past you. There are, in fact, one pass algorithm approximations for histograms. Unfortunately, they generate equidept histograms, not equi-width histograms. And those are really a pain to visualize. We spent some time fighting with that and got ourselves stuck. But if any of you have ideas on how to visualize an equidept histogram, there's a lot of opportunity there. Incremental visualization really changes the rules on things. In a numerical data, we're talking about changing bounds on everything you can see. Any color map, any scale, any axis might suddenly change on you as you discover that there's new data that was outside of what used to think was the domain. I'm a little short on time right now, so I'm actually going to skip over this conversation about my tentative framework, where I really just go back to my conversation from earlier and say, hey, look, as we're doing all this bucketing and aggregating, somewhere in there you're having to make a decision about where you sample, where you cache, and where you network. All of these trade-offs become an interesting conversation about precisely what degree of flexibility you're going to allow your users for what sort of interactions you can get. So I had just gone through a couple of standard examples pointing out that when you do server-side rendering you're doing all your caching up front, and when you're doing D3 you're trying to send all your data up to the client and then have your client deal with it. What I want to bring out, though, is that there's cross-disciplinarity here. This isn't the way that SQL, or Hadoop, or visualization packages or anything really works today. As this data gets bigger, InfoViz, front-end, and back-end concerns are coming together. This is calling for all of us to learn new skills and to teach new ideas to the people around us to try and explore how these ideas can be integrated. There's going to be a lot of close collaboration across fields. It's not going to be someone shows up with a CSV file or a server or even a SQL database and says, go play with this. And on that, I think this is a call to go build cool stuff with big data as well as small. You can tweet me, you can go find my webpage. I'm around for the rest of the conference. Thank you all so much.


My name is Prawn Noree. I'm a mixed media artist and I worked in Borrome Hill, Brooklyn. So I work on subjects surrounding procreation and bioethics, all the bioethical issues that you have today with the new discovery, scientific discoveries. So all my projects have this kind of reflection about what makes us human and how we can interfere into human selection. So it's a kind of from natural evolution, we go to artificial evolution. I'm born in Paris and I've been raised there. I went to school. So sculpture, especially wood sculpture, craftsman school, there in Paris. And I've moved to New York almost two years ago because I was invited in a residency into the art center, the Invisible Dog. There are no typical day in the studio indeed. That's what is nice. Every day are different. Because sometimes a typical day at the studio, it's in fact in a craftsman's factory in the middle of China and I will live their life for two weeks. Same in India, but here in Brooklyn when I'm here, it depends on what I'm doing. As a mixed media artist, then you can do so many different things. It can be doing working for months, night and days on a sculpture in clay. So I would say that's beautiful as an artist is that every day are different.


Many commanders throughout the history of our species have placed different definitions on what amounts to perfection in the military context. For some, it is simply ensuring that casualties and losses remain as low as possible. Some place primacy upon execution of predetermined strategies, whereas others count tactical flexibility as the highest of virtues. Many soldiers still laud an individual warrior's courage, while their detractors say nay, it is in professional coordination with one's fellow soldiers that true military perfection lies. For the subjects of this record, each and all of these combined was the only acceptable solution to the question. Know then that this is a record of the tactics, chain of command, and operational structure of the 3rd Legion Emperor's Children. At the macro level, the 3rd Legion was organised in a fashion that befitted the ideals of their Primarch Fulgrim, as well as their own ceaselessly perfectionist nature. As with every other aspect of the Emperor's Children, nothing could be left to whim. Every action within the Legion's remit was deliberate and constantly assessed for value, both functional and aesthetic. Fulgrim would often remark that perfection lay in the details, and that no detail was too small to be considered. Similar to his brothers Perturabo and Rubut Gulliman, he was a firm believer that the machine was only as good as the sum total of its parts, and that the quality of the whole is reflected of the quality of its constituents. But, whereas the 4th Primarch was brutish and straightforward, and the 13th, pragmatic and rational, the Phoenician demanded perfection in all aspects of his astartes. Structurally, while remaining steadfast to the strictures of an Astartes Legion as presented in the Principia Bellicosa, the 3rd Legion nevertheless took much from the structures of other legions and added their own terminology and emphasis at every level of Legion organization. At the core of this was an inviolate hierarchy of both authority and ability that permeated the entire Legion. Every Astartes, from the line to the officer core to the Lord Commanders, occupied a place in this hierarchy best suited to their own proven strengths and abilities, as determined by their commanding officers. Whereas other legions, such as the 6th Legion Space Wolves, or the 17th Legion Word Bearers, may have been more fluid in their variations in unit size or nature, the 3rd had no such time for any such whims. Any and all changes, minor or major, were made only after careful consideration of outcome. All ranks and appointments were, likewise, delineated with extreme care. An Astartes' path through the ranks was the result of, and marked by, dozens of accolades and marks of honor, all of which were solely to be presented to him by someone higher than he in authority, with tributes from the Primarch, held, naturally, in greater esteem. For a greater part of the Legion's history, this translated into an unshakable personal and battlefield commitment to excel, sprung from the Astartes' own belief in his inherent superiority and desire to prove this. Lines of authority within the Legion were perhaps the most rigid of the entire breadth of the Legion's Astartes. Beneath the Primarch were ten officers, known as Lord Commanders, who led the first ten Millennials of the Legion, a formation analogous to chapters within other legions, and a good example, besides, of the 3rd's insistence on fashionable terminology to better suit its character and aesthetic concerns. Each of the Lord Commanders formed the inner circle of the Legion, and each was expected to be a paragon example of a leader, warrior, and nobleman. The Primarch Sanguinius of the 9th Legion, Blood Angels, once called these ten the Princes of War, and to them fell the leadership of the largest and greatest campaigns undertaken by the Emperor's children. And in keeping with the Legion's historic importance in the eyes of the Emperor himself, it was common for a Lord Commander to be given overall command of forces consisting of contingents from other Astartes' legions. In addition to millions of soldiers from the Imperial Auxilia, and other forces from the multitude of militant formations within the Imperium, such as Mechanicum Tagmata, or Night Houses. On the rare occasion of a true failure in command, a Lord Commander was apt to take his own life as penance. Command in the 3rd Legion descended through a complex and multi-tiered command structure, which flowed from the Lord Commanders through the Praetorian, which in turn comprised officers known as, in descending order, Commanders, Lieutenant Commanders, Masters, Sub-Commanders, Tribunes, Palatine, and Ordinary, Captains of various divisions, all the way down to Consuls, Equaries, Ancients of various functions, and yet further down through Centurions and Heralds, to the Praefectors and Sergeants who commanded the Legion's individual squads, finally ending in the Legion's line Astartes. As evidenced, the Emperor's children placed far greater importance on rank and station than any other Astartes' Legion, with the subdivision and politicking between them drawing comments both confused and scornful from other legions, who saw the 3rd's chain of command as needlessly bloated and complicated. Order and precision were nonetheless the watchwords for the 3rd's overall structure, with all formation sizes being more regimented and uniform than any other Legion. Variance was always either the result of a temporary aberration, to be corrected as soon as possible, or a deliberate decision employed for a particular stated purpose. Each squad was assigned a particular role on the battlefield, and would train relentlessly in order to excel at its stated role and achieve the pinnacle of efficiency and unit cohesion. Should an Astartes prove himself more adept in any one way of warfare, his reassignment would be swift, as if there was one thing the 3rd Legion truly excelled at, it was in recognizing where the talent of an individual truly lay. It is notable that every single potential Astartes' unit formation from the Principia was present in the Legion from the outset, as there was not one discipline they considered beneath them, nor unworthy of mastery. This being said, the 3rd Legion did display a notable preference for speed of engagement, whether in deployment, maneuver or attack, to the extent that it took primacy over simple martial strength or fortitude, as to them, it was the warrior who struck first, and struck hard, who would be the ultimate victor. Similar in Mien to the 5th Legion white scars, although never to the same levels, this was reflected in the wargear the Legion maintained, and indeed in the formations it fielded, with rapid engagement elements of the Legion being markedly larger than its fellows, save again for the 5th Legion. A higher number of jump-pack-equipped infantry would be included in any ground engagement the Emperor's children participated in, backed up by significant numbers of jet-bike sky-hunter squads and anti-grave attack speeders. By his own admission, these latter formations appealed to Fulgrim's aesthetic considerations, as in aspect, they resembled the swift and noble knights of old Earth, charging the enemy with banners flying and armour shining. In practical terms, it has been theorized, by one's colleagues, that while their Primark would never readily admit to it, preserving his Legion's numbers was a consistent and grave concern. Despite its near extermination in the Gene Seed Crisis, the 3rd would never field active astartes in numbers that would allow it to participate in the attritional warfare, favoured by the 4th Legion Iron Warriors and 14th Legion Death Guard, and besides, they lacked the collective psychology to even consider such a thing, seeing such actions as completely beneath their talents. As such, the Emperor's children increasingly came to value incredibly detailed strategic planning and the perfect execution of said. In much the same way as they approached their own Legion structure, every aspect of a coming battle would be scrutinized, planned around, and accounted for in as diligent a manner as possible. Everything from the weather of the planet to the character of Imperial reinforcements and auxiliaries to the very culture and history of the foe themselves was analyzed and turned where possible to the Legion's advantage. In much the same way Fulgrim prized precision in the detail of his astartes, so too did the Emperor's children leave nothing to chance before engaging with the enemy. Remembrances from the time recorded that, in the closing years of the Great Crusade, the Emperor's children would be so assured of their impending victory that they would hold triumph banquets before the engagement had even occurred. While their planning did often lead to stunning successes, it had its risks. This almost mechanistic approach, closer to the mind of his brother Perto Abo than Fulgrim would ever consciously admit to, had the potential to leave the Legion thoroughly off-guard should some crucial element be unexpectedly removed from the equation or some unforeseen contingency occur. While the Legion was incredibly thorough in its planning, this was known to occur on occasion and would inevitably lead the Emperor's children confused and rudderless, such as during the compliance of the planet 140-20, colloquially dubbed in records as murder. At the outset of the Great Heresy, strength of the Third Legion was approximately in the order of 110,000 astartes. While this placed them in the middle tier of astartes legions by sheer numerical size, the skill with which the Legion comported themselves on an individual basis as well as a military formation meant its functional military strength was markedly higher than its mere numbers suggest, while the number itself is also a markedly impressive feat alone given that barely 2,000 of the Legion remained at the time of its Union with its Primarch. As initial participants in the Istvan atrocity, it is estimated that around a third of the Legion's strength was marked as too loyal to the Emperor to be let live. The loss of this contingent and the military losses involved in their extermination resulted in the Legion having, at most, 50,000 warriors to commit to the Dropside Massacre, a staggering loss in manpower said to have utterly enraged Horus Luprikel. But as with all things involved in the descent into depravity and madness the Third Legion embodied should not be the end to their dark tale. Sinister machinations, biological and sorcerous, would soon work their way through this once-noble Legion. But that, as they say, is a record for another time. Until then, Ave Imperator, Gloria in Excelsis Terror. This video and this channel are made possible through the incredibly kind contributions of my Patreon subscribers. If you'd like to help support the channel, head on over to patreon.com forward slash Oculus imperia. And if you're looking to keep in touch with the channel, get regular updates, you can follow me on twitter, at buttstuffkaiju, or check us out on discord, a link will be in the description and on the channel page.