Next up, I'd like to welcome a veteran of many open-stack keynotes, Mark Shuttleworth. Come on up, Mark. Hello, everybody. Good morning. Yesterday, Troy announced, or described us all as the Star Wars Rebel Alliance, and suddenly a bunch of things came clear to me. I used to think of myself as Han Solo, but recently my girlfriend's been calling me Chewbacca. I can't imagine why. You know, what's amazing about open-stack is this incredible ecosystem.   Well, you may feel that your success depends on open-stack, but the opposite is also true. Open-stack's success depends on you being successful as well. And the thing I hope every vendor, every technology supplier appreciates most about Ubuntu is that we don't compete with our ecosystem. We don't come to the table with our own favorite messaging system or hypervisor or database or PAS. We try to enable an ecosystem where everybody will succeed. And maybe that's why so many people choose Ubuntu when they're moving to the cloud. Even companies that have spent a long time doing things one way in Linux choose Ubuntu as they move to the cloud. If you look at the public cloud, nearly 70% of the Linux workloads on all of the major public clouds are Ubuntu, even from enterprise commercially supported applications. And in open-stack, the substantial majority of the production enterprise deployments of Open-stack, those companies have chosen Ubuntu as well. And what makes me very proud is that of the five out of the six featured users, you'll see here the super users, chose Ubuntu. And in every single case, there are multiple vendors, other vendors, from this audience who are bringing their innovation to bear on those projects and sharing in that success. Six months ago, we announced the Ubuntu Open-Stack Interop Lab oil. And I'm delighted that 14 vendors have now joined that, moved that. There are many more in the queue going through the paperwork. And what's unusual about that is that it's not a single one-time certification of interop. In fact, it's continuous interop testing. Every day we build Open-Stack 100 or more times with different permutations and combinations of components from different vendors. That's extraordinary. We can build a different cloud every 15 minutes across a whole bunch of different clusters of hardware from different vendors. And the way we do that, if we could switch to the demo screen, the way we do that is by using juju, this amazing service orchestration system that's completely open source. And it allows us to build a cloud just like this. So this is a definition of a cloud. We generate these every 15 minutes and throw them at this huge cluster containing hardware from different vendors. And what you should see there is Open-Stack being built essentially from scratch. We do this on bare metal, installing the operating system, and then installing all the different components of Open-Stack. Each box there, it's always a bit of a relief when the boxes start to arrive. Each box there comes from what we call a charm, which is like a package of software for the cloud. And charms are completely reusable. They can be written in anything you like. Chef Puppet doesn't matter. One box there can be in Chef. One can be in Puppet. One can use Docker. Another can use Salt and juju as a service orchestration system glues those all together. So that allows us to build the cloud incredibly quickly. If we can come back to the deck, it also allows us to do continuous interrupt testing, both on the stable version of the code and on the tip on trunk. And it allows us to do continuous integration as a service. For those of you as vendors who are actually shipping Open-Stack itself rather than plugins, and are modifying Nova or Neutron, we can do continuous integration on your branch in that oil environment against the components from other vendors so that you can tell your customers whether or not your version of Open-Stack will work with components from other vendors. And this generates a treasure trove of information for vendors. So here I have a sample report from oil. We share these with vendors every month. And it's an absolute goldmine because it lets you know exactly where you need to focus your engineering resources. It also enables us to do same-day turnaround on validation for a proof of concept if you have a customer who is looking to build a cloud with components from you and from other vendors. So for those of you who are here to essentially figure out how to integrate your products with Open-Stack, that's probably the single, joining oil is probably the single best investment that you could make. What this means for users, though, is complete freedom in your cloud architecture. You know, we can place those services, those different charms, across any combination of any hardware. What that means is that your cloud architects can essentially build in a morning a cloud based on a particular architecture, placing services in particular places in the network. They can study it for performance for your particular workload. Then they can tear it down over lunch and build it again differently in the afternoon. And people have said again and again that speed is what matters. Speed is what accelerates, accelerating your ability to learn, to master Open-Stack is what this is really all about. So people who've adopted this way of thinking say that as they accelerate forward, as they essentially crowdsource all of the core components but maintain their own architecture, they have mastered Open-Stack and move much faster. So when we work with the very largest clouds, their architects essentially model what they want and then use Juju to place the services exactly where they want on their network. But for smaller organizations, you may not have cloud architects or you may want to have a best practice reference architecture. So I'm delighted to reveal for the first time a fully automated reference Open-Stack installer which we've built on top of Juju. Juju's open source and all of the charms are open source. So what we've done is we've built an intelligent installer on top of those which we've made available to our customers. It's part of our landscape systems management product. Again, if we can switch to the, there we go, thank you, you guys are well ahead of me. So over here you'll see a new Open-Stack tab and that's going to let me build a cloud. So I'd like to demo this live on stage. I have over here a cluster and this is in fact a C micro cluster from AMD has 64 fully loaded nodes and we're going to use landscape to install Open-Stack on that cluster. What we've tried to do is to design out, design down to just the key choices that any user would have to make. So what I've done is I've registered all of the machines in that cluster with metal as a service, maz. And I've connected maz to landscape. So landscape sees maz, it sees that I've got at least five machines ready to go and one of them that has multiple necks for neutron gateways and so on.  I need to name it, so let's call it ODS Atlanta. I need to make some core choices. So I'm going to choose a hypervisor. We'll go with KVM. And as vendors come through the oil process, we will add them to this if they want.  We could choose different SDN solutions, lots of interesting options emerging there. I'm going to go with open V-switch. I'll choose Ceph because we love it dearly. It's brilliant. Has been brilliant for three years. And now I need to add some hardware. So in maz, you can allocate hardware to different physical zones. So you can model in maz the physical separation of clusters on your network. So I'm just going to add two zones and then, of course, this will map those up to availability zones in open stack. So you do that. All right. So install the cloud. So all of these machines are switched off. There's one little server in the back which is running maz and landscape. And landscape will interrogate maz, interrogate the hardware profile so it will study every node there. Then based on that, it will build the best practice cloud based on everything that we've learned from building the world's biggest open stack deployments. What's cooler about this is that that architecture will evolve. Landscape's a management system and your update landscape every month. And as you update it, it will evolve your cloud. It will, for the Sony chaps, it will handle version upgrades inside open stack if you wanted to. And it will manage the system from the bare metal all the way up, giving you a consistent cloud experience effectively as a service. But if you wanted to create your own architecture, your own installer, the important thing is that you can do that and many guys are starting to do that now on top of the very same charms. In other words, they're reusing the same charms, the service definitions that telcos, banks, media companies and others are using. As those charms get better, their implementation of open stack gets better just as well. So with crowd sourcing from all the users of Ubuntu, essentially, the expertise to really optimize open stack components. And when you build your own installer on top of that, you're essentially reusing all of that expertise as it gets more secure, as it gets more performant. How good could it get? Well, that C micro chassis is pretty amazing. It has 64 nodes. AMD was kind enough to lend us at first another five. So we had six C micro chassis. And on that, we set what we believe as a new world record for the number of VMs launched on open stack. We managed to launch 75,000 in six and a half hours. So that beat the previous record by an hour and a half. We then kept going. And it got to 100,000 VMs. If you want to know what that looks like, here it is. That's 100,000 VMs on just under 3,000 cores. It was slightly over committed. So AMD got very excited. It gave us another three of those C micro chassis. And we turned them on, registered them in Mars and expanded the cloud. It's all very elastic with service orchestration. And pushed it up to 576 hosts and 168,000 VMs, all in the course of a weekend. But if you wanted to go really big, how about the world's fastest supercomputer? Well, that's in China. And it happens to run Ubuntu. And over the last couple of months, we've been invited to install OpenStack there. The intersection of HPC and OpenStack is going to be amazing. That's all running on Intel Xeons. So the fastest computer in the world at the largest scale. An amazing piece of kit. At the exact opposite end of the spectrum, though, you might think that OpenStack is built to be big and bad. But what if we could show you OpenStack as a super light, super small experience for tiny clusters? For the first time ever in front of a public audience, this is ARM 64-bit hardware running OpenStack on Ubuntu. Those are the GeneX development boards from Applied Micro. At our booth, you can see OpenStack running on those with a whole bunch of really interesting workloads. Log Stash, Kibana, everything up and running. And it's beautiful. And just to complete the silicon picture, last week we announced the completion of the port of Ubuntu to little Indian power. And so that makes Ubuntu the one platform that can give you an OpenStack experience across all of the different architects that are out there in widespread enterprise use. Each of them has something beautiful about them, something interesting. And we're here to enable innovation, enable you to choose the platforms that suit your workloads. So you could have a lot of fun with this. And here's some of the fun that we've been having. You may have spotted these on our booth. Come and have a look at them. But Intel also makes these very cool microservers. And we wondered what it would be like if we had 10 of them in a box. And so here we have the orange boxes. Inside each of those are 10 Intel microservers with Juju and Maz. And so you can do anything with those boxes. They're an amazing way to just master distributed systems. We're using them for jumpstart training. Our jumpstart program now has a training dimension for 10k plus travel and accommodation. We'll come to your offices with an orange box. We'll spend two days training you to master OpenStack, build it in different ways, as many different ways as you like, combining components from those different vendors in oil. And once you've mastered it, we'll leave that box with you for two weeks so that you can figure out exactly what you want to do with it. Most people at that stage are ready to go and build their own OpenStack clouds. They've figured out everything that they need to make OpenStack exactly what they want it to be. And if you do build a cloud, if you do choose Ubuntu on your cloud, you're in really good company. Some of the most amazing companies in the world are building on Ubuntu and OpenStack. Deutsche Telekom, building their next generation network. Time Warner, building a huge media property essentially on Ubuntu OpenStack. NEC, one of the world's leading telco network equipment providers, shipping telco solutions on OpenStack on Ubuntu and offering a public cloud. And NTT, one of the great telcos in the world, building a series of clouds and also bringing some really amazing SDN technology to the OpenStack community. And they've chosen, and many others have chosen, I think five out of six of the featured super users here have chosen Ubuntu. They've chosen them because Canonical and Ubuntu have many firsts in the cloud. We were the first to adopt OpenStack by some margin. We were the first and still have the most KVM hosts and guests in production use. We were the first long way to adopt Ceph and there will always be a vastly more Ceph in production on Ubuntu than other platforms I imagine. We demonstrated live migration on stage here at OpenStack for the very first time. We pioneered cloud in it. So every time you boot Linux on any cloud, you're using technology developed by Canonical and contributed as open source. And right now, the key thing that we think is absolutely fascinating and amazing is Linux containers. In 1404 LTS, we shipped Linux containers 1.0. And this is a cross distro effort led by Canonical. More than 60 companies participate in that initiative. If you haven't played with LXE, just grab an Ubuntu desktop and spend an afternoon with it. It will blow your mind. When you see any Linux distro, you know, CentOS, Debian, Ubuntu, Red Hat, booting in less than a second right there, it's virtualization like you've never imagined it before. So it's really amazing, really fantastic. For Linux on Linux environments, it's changing the world. And it's the basis for a huge amount of innovation, including Docker, which you might have heard of. And tons of other things, essentially in the PaaS space, using containers, not VMs, to accelerate everything that they do. We have been an innovation enabler. You know, innovation comes from small companies like Docker, like Cep. We are delighted for the Cep guys with their acquisition. You know, when you buy a company, you don't necessarily buy the magic and you can't buy the ecosystem. And what is important to us is to be at the center of these thriving innovation ecosystems. We will be fully supporting Cep with all of the management solutions that are available for it, for all of our customers and users on Ubuntu. Now, for those of you who are operating clouds, this is the cold hard truth. You will be benchmarked against the world's biggest and leanest cloud companies, Amazon, Microsoft, Google. They do a great job of what they do. For OpenStack to succeed, it has to deliver innovation. That's what this ecosystem is for. And it has to make it possible for people to build clouds that can compete economically with those companies. And that's why we make this really strong commitment that Ubuntu is the most cost-effective way that you can build a complete, supported, commercially supported, sustainable private cloud. Year ago, we announced availability zone pricing ranging from 75K to 350K per availability zone. Never gets bigger than that no matter how big the zone gets. And that's immediately become popular with all of the people who aspire to scale. If you aspire to scale, if you want to build a big cloud, then this is the model for you. But today we're also announcing another model for folks who are building smaller clouds and who would like essentially to get a best-practice cloud that is entirely managed. So on your choice of hardware in your data center, from the pool of certified on oil and Ubuntu hardware, we will help you design and build a cloud. We will manage that cloud, a complete managed cloud to an SLA for $15 per server per day. And we'll make it fly using everything that we have learned from the world's biggest open stack deployments. If you do that with us, your cloud will be very good. It'll be carrier grade. These are big carriers that are building big open stack infrastructures. It'll be banking grade. Delighted to be working with NEPs like Cisco on the carrier market, the telco market, meeting the specific needs of telcos. It'll be banking grade. You'd be amazed at how much of Wall Street is adopting Ubuntu. And I was there three weeks ago and heard this completely non-attributable quote from someone who I believe is here. I hope that they're continuing in the same theme throughout the week. But most importantly, it's going to be Chuck Norris grade. I love dealing with the media companies. They're kind of crazy. They're pushing the limits. They love Ubuntu. And it's great to work with companies in that space. Comcast, Bloomberg, Time Warner, Disney, OpenStack is incredible what they're doing with OpenStack on Ubuntu. With all of that activity, of course, we're hiring. Now, we don't hire pre-Madonna's. The open source community has plenty of that. What we're looking for is people who are passionate about serious scale and serious quality. If you think you could really enjoy working at Google, because of the automation and the scale, but you want to work in a smaller company and bring that kind of capability, that Google-esque automation to the rest of the world, then come and talk to us with the guys in orange. Right. Now, I promised you a little bit of magic. So here it comes. Juju, which is that magical service orchestration system, enables you to do almost anything in distributed systems really, really easy. People don't believe that you could spin stuff up that quickly. At an IBM conference two weeks ago, we spun up WebSphere and Hadoop and a bunch of other things in about three minutes, and the audience just couldn't believe it. We had to actually go and show them WebSphere running. It was there. It was live. It was real. So I want to show you a couple of things that you can now do with Juju that you couldn't last time we spoke. So if we can, thank you. Oh, here comes the cloud. And I see blinking lights. So far so good. This is Cloud Foundry. This is Cloud Foundry, which will be available as both an open source and a commercially supported product, supported by both canonical and pivotal by the time we meet again. So this is Juju deploying Cloud Foundry. So you could deploy this as a result on any cloud. This is on EC2, I'm afraid to say, but it will also be on OpenStack and on Azure and on bare metal. So however you want to consume Cloud Foundry, you'll be able to get it like this with Juju from Pivotal and Canonical. If you wanted to do Hadoop, for example, this is what Hadoop looks like. I can do this straight from the store. So this is a little cluster, very simplistic cluster of Hadoop. This, of course, doesn't show you anything about scale. This cluster could be thousands of nodes because that's just a logical model of what's connected to what. I could go in here and do slave cluster and just scale that up to 200 nodes and it would ask the underlying cloud for another 199 and we'd be off to the races. What else could we do? Toby Ford is talking about NFV, network function virtualization. So really, this is about creating distributed services on top of clouds that improve, enhance, create software versions of some of the existing appliances that telcos use. Well, here is an open source NFV solution. This is an IMS, it's called, which is a really critical piece of kit for telcos. And so that's Clearwater from Mesa Switch. As a series of charms, you can spin that up in a couple of minutes on any cloud or again on bare metal and just in a couple of minutes, you're good to go. What else? Oh, yes, we spent some time with a company that's quite famous for cars on demand and they were looking at ways to make their developers more agile, their DevOps more agile. So this is what we modeled out for them. They were able to get a complete replication of their full environment that the developers can spin up in memory or throw it a cloud for test or throw it VMware or physical infrastructure for production. So that's a whole bunch of different services reusing these charms. There's Rails in there, Memcache in there, tons of different app servers, tons of different databases. So it really is getting pretty amazing. So coming back to the slides, the ecosystem around Juju has grown fantastically. These are some of the companies that are using Juju to deploy complex systems or charming up their applications so that you can deploy their applications just as fast as that. And this week I'm delighted to announce another significant company stepped up committed a full team to work on Juju and that's IBM. So I'm going to hand over to Daniel Saba, the CTO and GM of Next Generation Platform at IBM to tell you a little bit about that. So if you look at the specific areas of collaboration between IBM and Canonical, what we're talking about is basically leveraging the wide variety of Juju and charm assets in the context of OpenStack, in the context of BlueMix with Bosch, in the context of SoftLayer and the way in which we actually characterize workloads and services so that basically people can take full advantage of the underlying SoftLayer resources to then optimize their individual workloads. The only way we can actually do that is through that kind of a partnership. So we'll be working with IBM on three things. The first is the integration of Juju and Heat. Heat does a fantastic job of OpenStack infrastructure orchestration. Juju does an incredible job of application orchestration and bringing those two together will be amazing.  And you'll be able to use those same charms anywhere, of course. In OpenStack it'll be amazing to use them with heat. We're going to do the same with Juju charms and Tosca, which is another orchestration standard, so Juju in Tosca. And of course we're charming up IBM's portfolio of key software for scale out and distributed applications, as well as software from IBM partners. So people all over the world tell me they're just loving that agility, the acceleration of their DevOps type process that Juju gives them. But there is one common request, one very frequent request, which is for the ability to use Juju with other operating systems. Our focus, of course, has been Ubuntu throughout. But I'm delighted to say that thanks to a customer contribution, over the next couple of weeks we will add full support on Juju, which is orchestration, and MAS, which is physical infrastructure magic, physical infrastructure provisioning, for CentOS. Now if you look at the charts, CentOS and Ubuntu between them are about 80% of the cloud workloads that are out there. And that pretty much anything that you're doing on the cloud you could now orchestrate with Juju. But perhaps that's not enough. Perhaps in some cases it will be useful to use a platform supported by a much larger company. And so I'm also delighted to tell you that thanks to an unexpected contribution from a new member of the Juju community, both Juju and MAS, so orchestration and physical provisioning, will grow full support for Microsoft Windows. And to tell you a little bit about that, here's the CEO of Cloudbase who contributed that work. Cloudbase Solutions leads integration between OpenStack and the Windows ecosystem. We recently added Windows support to Juju and MAS to meet enterprise customer requirements for provisioning and orchestration in mixed scenarios. Now MAS supports governmental provisioning of Windows at the BA Linux deployments at NSK, including OpenStack environments with mixed at the BA KVM units. Juju enables complex Microsoft deployments with just a few clicks, including Active Directory, IIS, SharePoint, Exchange and more. Windows software charms use PowerShell and Python that can have all the same power as Linux charms. The result is amazing. Rich orchestration with both Windows and Linux components, like magic. Really, the best orchestration on the market. And it's entirely based on open source. I would encourage you to check out the Cloudbase booth. They've done some very, very cool work. I was absolutely stunned when they made this proposal. Just to prove that that's not just talk. On those orange boxes over there, one of them, you'll see all the lights are on. That's actually running OpenStack right now. The other one has most of its lights switched off. So why don't we turn those lights on? So what I'm going to do is kick off a deployment of OpenStack with mixed hypervisors. So there'll be physical installation of Ubuntu on that orange box going on. And there'll be physical installation of Windows on that orange box from scratch. And then orchestrated on top of that all of OpenStack, including Nova controller, sorry, Nova compute on Hyper-V. So that that Cloud will essentially be running OpenStack with Hyper-V on that orange box. So here it comes up. It takes a little longer than Ubuntu to install, but it does come up and you'll see it live on the booth. Also, what about applications? Well, here's what's running on the other orange box. Again, that's an OpenStack cloud with Windows images. And we've spun up on there. Windows Active Directory, Microsoft SQL server, and Windows SharePoint, all orchestrated. So imagine the possibilities. Imagine the possibilities of being able to deploy OpenStack with Active Directory backing your Keystone or SQL server as the database controller. So the world is moving very, very quickly. Heterogeneous environments, heterogeneous hypervisors, heterogeneous applications, heterogeneous platforms. That's the ecosystem that we want to enable. I very much hope we'll see you tonight at the Mardi Gras and Magic party in the Murphy Ballroom, where we'll be celebrating some of that juju magic. And it's just one last thing. If we can come to the demo screen again, Maz has support for all kinds of different hardware. It turns hardware on and off and Pixie Boots it and stuff. So if you go into the node, you can actually see how it's powered. There are all drivers for, there's a driver for C micro, there's a driver for the HP moonshot clusters, IPMI, AMT and so on. But there's this robot plastic as a service thing. What could that possibly be? Well, what do you do if you're trying to sort of test a service that turns machines on and off but you don't have a BMC?  So coming back to Troy's theme, this works really much better, Spaceball's style. If you have little figurines and you go, turn on the server, use the force, Luke. Yay! Who would have thought the same infrastructure that can power on supercomputers, power on whole data centers, deploy hundreds of thousands of servers and put OpenStack or Hadoop or anything else on that could also power Lego. Have a great week. Thank you very much and stay well. I just want to give a special thanks to Chewbacca for making time out of his busy schedule to get here. You have no idea how hard it was to get him. So just one more reminder for everybody. Our next summit is going to be in Paris. You may have heard. November 3rd. So just wanted to make sure everybody knew the date. There will be more details coming soon. But in the meantime, enjoy the rest of your time here in Atlanta. And thank you very much. I hope you have a good time with OpenStack.


Live from Houston, Texas. Extracting the signal from the noise. It's theCUBE. Covering Grace Hopper Celebration of Women in Computing. Now your host, Jeff Brick. Hey, welcome back everybody. Jeff Brick here with theCUBE. We are live in Houston, Texas at the Grace Hopper Celebration of Women in Computing 2015. It was 8,000 women last year, 12,000 people this year, probably 12,000 women, 1,000 men. The conference is growing like crazy. It's a, celebration is the right word. It's like no other conference that we cover. And so we're really excited to get the keynote. Hillary Mason, Cube alumni. We haven't seen Hillary since Big Data, New York City, 2013 I think was the last time we had you. It's a welcome back. Yeah, that's right. It's nice to see you again. Absolutely.  You were helping out at Excel. Now you've founded another company, Fast Forward Labs. So congratulations. Thank you. So give us an update. What's Fast Forward Labs all about? So what we do at Fast Forward Labs is look for technologies around data and machine learning that are emerging, that might open up possibilities for new products or new features or new business opportunities, but that are not really well understood or commoditized. And we pack them up in a form where our clients can understand what it is, how it works, who's doing it, what's in the market, and really figure out how to plan their product and business roadmaps around these machine learning technologies. And just to give you an example, we just finished a project on deep learning for image analysis, which is having algorithms look at images and tell you, oh, there's a dog and there's a can of soda in this picture. And that's something that's new. It's a breakthrough over the last couple of years. And now people are starting to really build it into their product roadmaps. It's a really different process because it used to be, if you built a technology and a product without an application, oftentimes you just burned up a bunch of VC money and didn't go anywhere, right? The much better path was to have the problem first and then build the technology to go attack it. But you're kind of flipping it the other way because these algorithms and stuff are moving so fast and the technology's moving so fast that the business people and the people thinking up the applications can't hardly keep up. So it's a pretty, it's an interesting kind of twist on it. Exactly. So our goal is to make our clients as smart as we are about what's possible with their data and machine learning and data science capabilities. So one of the things that's come up around machine learning and kind of algorithmic suggestions, right, is if it's done right, it's magic. If it's done poorly, it's creepy, right? So not right actually. I take issue with the phrase of that. Because machine learning is not magic. It is fairly simple mathematics deployed on data at scale. And we act like it's something that's inscrutable, that it's a black box no one can understand. But we exist to say that's not true. You can understand it. And yes, we do see these bizarre cultural consequences of machine learning applied sometimes in products where people are not aware of what could happen in edge cases where you might classify a photo, say of a concentration camp as a jungle gym, which was a recent issue that came up with a photo product from a major tech company. And that's not algorithmically wrong, but obviously very upsetting in the product. So part of what we do is actually explain how these algorithms work, how the math works, how the code works, how the data works, and then help people think through the potential cultural side effects of deploying one of these things in a product. And it is not magic. It's, but it feels like, but it should feel like magic, from my, I mean, just, it's crazy. Yesterday, what did I search like Southwest Airlines or something, or no, I searched, I think Grace Hopper, I was trying to get to the agenda. And it came up with my registration number. And my whole thing's like, wow. And you know, and of course the classic case is the target example, not to beat a dead horse, but that's the one that everybody knows about. So even if it's not magic in the machine that delivers the result, there still now has to be this kind of next factor, this, you know, is it appropriate, is it ready? Do we have to retune the algorithm? Do we have to run it through a filter? How do those things kind of get addressed? So the best data products are the ones that you can use without any awareness of the technology behind them. And in my keynote, I talked a bit about Google Maps as the best example, in my opinion, of a product that integrates a huge amount of complex technology. Like if you bring up the Google Maps traffic view, you look at it, you make your decision about where to drive and you turn it off. You can use it without any idea about the prediction algorithms going on, the data they're collecting, how they're displaying it to you. It's a great example of a product that is really simple to use, but you don't have to understand that. From the perspective of the product designer, I think we have to grow our engineering and product design processes to take this sort of evaluation into account where it's not just, is the product usable, but is the product providing value to the person using it in a way that is not going to upset them or make them creepy, of course? Also, we tend to ask things like, what are the ethical considerations of building a product around this technology and how could it possibly go wrong? Because you need to think about that stuff in your product development process. Which brings up another question, a great topic. We're at another show and they were talking about the moral issues around writing the algorithm. What are you creating this algorithm for? What is your expected outcome? But it seems like kind of a fallacy to even attempt that beyond kind of the first order, because now its algorithms are built on algorithms are built on algorithms, right? And the data keeps increasing and changing. So even if I have all the best intentions today, and we bring you guys in and you make a great little product for us, who knows what either the data or the input or the social mores are going to be two years, three years, four years. I don't even want to think beyond that. So how do you help customers kind of think those issues through or do, is it more let's do it, see what comes back and then we can start to make some of those judgment calls. There's an element of both. So you really have to be thoughtful about this from the beginning of the product design process to think about, if I build this algorithm, what are the potential edge cases and how might this go in a way that I don't want it to go in my product? But then of course there are things you can't predict. And in the keynote I shared this example of Sony created a product called the Ibo, which was this robot dog, it's really cute, right? And they've debuted it in 1999 and they finally ended technical support for it and repair service in 2013, which means that now these dogs are dying and people are holding funerals for them. They've become incredibly emotionally attached to their little robot dogs, which I can understand, dogs are great. And you don't have to scoop the poop on the Ibo too, I don't think maybe it does a little plastic one. So this is the kind of cultural consequence you could never have predicted when you're an engineer building this little robot dog, and yet it had some real emotional impact for some people who really love this device. And so you can never predict everything, but if you don't even think about it, you're not trying to be thoughtful about it in your product design process, you're never going to do a good job to begin with. Right, so you are such a pioneer in this space and obviously back at Bit.ly, where we all just thought we were using URL shorteners, we never really thought through the fact that this is providing all kinds of good information about who's sharing what. The world has really changed since then. People are realizing there's so many data sources, your own data, other data, outside data, public data, the speed of the data creation, now the speed of the tools, Hadoop, and Spark, share with us your perspective on how those types of technologies are really advancing, you know, kind of the machine algorithm world, and we're just going to go next. That's a great question, and it's something that I'm pretty enthusiastic and optimistic about because if you think back, even just five or six years ago, getting Hadoop to work on a large data set in a reliable way with a good response time was itself a technical challenge, and now I can spin up an elastic MapReduce cluster on Amazon Web Services with one line of code, and it works most of the time, and so the reduction in cost, in time, and in friction that we've seen over the last five years has been remarkable, and then when you add in things like Spark and other in-memory computation technologies, we get to a point where we can now do online machine learning and online computation in a really cheap way, from an infrastructure perspective, so what we've seen is that people have been able to now play with the technology to start to build it into their products, to see a lot of success with that, and when you look at what's going next, you can sort of think about it coming up the stack, so first we had to get the data in one place, then we had to build a way to analyze it, but we didn't really care how long it took, then we cared how long it took, and now it's real time and in memory, and the next thing, you know, I have a lot of theories on this, but I think we'll start to see some algorithms commoditized in that process, so you'll no longer have to write your own code for a recommendation algorithm, or for, you know, sort of an online averaging or variance model, and a lot of that is happening in the open source community. We're starting to see the same thing around analyzing data that was previously too difficult to analyze, either because it was too dirty, so we have products emerging to clean data, or because it was too complex, as in the case of image data, and I'll stop there, because I could talk about this whole thing. No, I love it, I'm just, as you saw, we have to have like Hillary Mason Day sometime, we'll come out and do like eight segments with Hillary Mason. No, I mean, just concepts are just coming in my head as you're talking, the whole concept of sampling where before everything had to be sampled, right, you didn't have the opportunity to take the whole data set, that's no longer the case anymore. And not in the technical age that we are in right now. And the other thing is just the opening of innovation, especially here with, you bring in some young engineers, they don't have to set up the cluster, they can actually start to execute their theories, their ideas and kind of get out of setting up the infrastructure, which I think is the big power of AWS and how it is really fundamentally changing industry. Absolutely, and I first realized this back when I joined Bitly in 2009, which is a social media analytics company gathering a massive amount of data for the size of the team and for what the product was, which is a, it was a URL shortener, right, short links on the social web. And I realized that this tiny little company with relatively modest funding was able to do analytics at a scale that previously would have been so expensive, it was just out of reach. And that that data was such a rich, interesting resource. Right, and it's often spoke about, we're certainly in a little bit of a bubble right now, compared to the old bubble, but the old bubble you had to take all that VC funding and buy sun boxes and build out data centers and Oracle databases and all this other stuff where now, like you say, you can spin up an instance on AWS, get started, get your app out there with a much less significant investment upfront. Absolutely, and we're seeing that for not just building a web product or a mobile product, but for building a data product, which is pretty exciting. So let's talk about one of the other things going on here to need a board is the variety of companies that are here, right? Googles here, Yahoo's here, LinkedIn, Uber. Those two aren't that different. Right, the usual suspects, right? But then you look around and you see nationwide insurance, we just had Bank of America on, Target is here, Best Buy, Johnson & Johnson. So clearly, and we talk about it all the time in theCUBE, every company's a software company now. It's just what they happen to wrap their software around. So it really opens up opportunities from a computer science point of view to go into lots of different places, not just kind of the traditional things where you might suspect. No, that's absolutely true. There are finance companies here, insurance companies here, retailers. I think any company large enough to get a team here might be here. And it's great to see that, and it's great to see the variety of career opportunities that are on offer here for all of the young engineers in the room.  That's great. So what are you working on next? When we talk to you next year, when we have Hillary Mason Day, back at fast forward later this year, what are you working on next? Building the team? Any kind of big things you're working on? Yeah, so we work in generally a quarterly cycle where we pick one idea to look at every quarter. And currently we're looking at language summarization. So this idea that you might have an article and you want to pull out the most interesting sentences, or you might have 10,000 articles, say on the Democratic debate, and you might want to understand what's the distribution of different viewpoints in all of the coverage, and what does that look like? Or you might be, say, an insurance company who has 10,000 customer profiles and want to understand what kinds of customers do we have? And when a customer appears, what products are best for them, how do I really think about that? So we're looking at neural models for language analysis and summarization right now, but we have a long list of ideas. So next year it'll be something else entirely. You must love all the horsepower that you could bring to bear on these problems like you did in that. I love what we're doing. It's getting cheaper and cheaper and cheaper. And we monitor that pretty closely. I bet. The economic constraints on some of these techniques, once it gets below a certain point, it becomes something you can play with, and then you start to see this plowering of creative products around the different techniques. It's amazing. That's all. We had Jason Stowell on from Cycle Computing, and they basically get excess capacity from Amazon that people have purchased but aren't using and spin up massive capacity for heavy, heavy jobs that are single-run, life-testing, et cetera. So it's a really crazy time to be able to have that at your disposal. Yes, and it just gets more interesting every year. So last question. What's your favorite hamburger? Cheeseburger. Can you say it? I'm Hillary Mason, I love cheeseburgers. What do you recommend? Well, so your access to cheeseburgers is somewhat geographically constrained, but if you happen to be in New York City, I recommend a little pub called Corner Bistro in the West Village. The menu's on the wall, and they serve you everything on plastic plates, and it's amazing. Corner of what and what? Corner of what and what? Oh, it's on Greenwich Ave and Jane Street, I think. All right, and what about not in New York City when you travel? So I am always up for trying burgers. I haven't found one in Houston yet, but if I'm on the West Coast, I always swing by In-N-Out, which is a, you know, it's in and out. We know In-N-Out, for sure.  Hillary, thanks for stopping by. Thank you so much. Best of luck to you at Fast Forward Labs, and we really enjoy getting you on theCUBE. Likewise. Absolutely, so I'm Jeff Frick. Next week, or no, two weeks from now, we're going to be at Oracle Open World, one of the biggest tech conferences there are. We are really excited. We'll be out in the middle of Howard Street. So hopefully you can tune in or stop by if you're in San Francisco. I'm Jeff Frick. We're in Houston at the Grace Hopper Celebration of Luminant Computing. We'll be right back with our next guest after this short break.


Move to our next speaker, Sean Eddy of the Howard Hughes Medical Institute in Genilla Farms is here to talk about reading genomes bit by bit. Sean. So it's an honor to be here and it's been a great ride throughout this project. I entered graduate school when Renato Delbeco wrote that article about the human genome project and why we should do it and I've been able to see this from the beginning. I want to convey to you some of the excitement that I feel as we get to look at all of this stuff and immerse ourselves in the AC's, G's and T's and also I want to talk a little bit about some of the biology besides medicine and besides the human genome and in the biology parts of the stuff I'm going to talk about. I'm also going to talk about some geekly things about computers and software but in the biology parts I'm going to try to explain two things.   We know a lot about cracking codes by both experiment and statistical analysis. Figuring out the meaning of apparently impenetrable languages is something that humans have done for a long time and it's a great intellectual thing to do. Puzzles are based on this. It gets very addictive. One of the great examples of this is actually described in a book called The Decipherment of Linear B by John Chadwick writing about his buddy Michael Ventress, a mathematician at Cambridge. This is an apropos tale. It is 1953 in Cambridge. One code was cracked in the pub and then another code was also cracked by Ventress. This is Linear B a language that had been found in Mycenaean caves and was only known in symbols. It's one of the few examples where we didn't know anything about the language it had to be cracked by direct statistical attack. This attack was comparative sequence analysis and the idea was to look for statistical regularities between different tablets and as far as I'm concerned this is one of the early successes of comparative sequence analysis. In this case Linear B turns out to be an alternative script for Greek and so the problem becomes somewhat easier than the problem that we're faced with. But we have a lot of examples and it's fairly mind blowing to me still to go to our disks and what you're seeing on the left unreadably is a listing of the top level of the disks at Janelia Farm where we keep our genomes. And this is like some pre-Victorian phylogenetic tree because you can sort of see algae, amoeba, amphibians, archaea, bacteria in an order that's sort of percolating up and down depending on what my laboratory and I and my wife are working on at any given time. And it's remarkable to me that sort of sitting in these directories is the actual source code to all these different wonderful creatures that we see. There is a lot of talk about how much data we're talking about. It is a fair amount of data but it's not at least at the moment completely mind blowing. For my lab that disk that I just showed you is about 450 gigs. We have most of the genome sequences that are available. If we were in the business of manipulating human data, if we were taking raw images off an Illumina, that's a lot of data per human genome and that's difficult to store, it's difficult to ship down the internet. But once it gets to an assembly, it's not so bad to store it per year. It's not so bad to transmit it down the internet. And as Eric just mentioned, there's proof of principle that we can store human genomes as differences. And so we know sort of intellectually, sort of in principle, we know that we can handle the genome for the next couple years but it's non-trivial. So the 1000 Genomes project has generated five terabytes in its pilot project. That's not a big deal. We have a petabyte of spinning disk at Genelio. We could store that if we needed to. The NCBI short read archive is starting to fall over. It's going to be approaching a petabyte soon. But these are volumes of data right now that are not too different than what's, you know, your iTunes collection or I've got this up. This is sort of like the Vikings used to drink, you know, beer from the skulls of their enemies. My coffee coaster is the Solera genome sequence. When we look at this sequence, this is one of the few genome sequences that fits on a slide. This is FIX174, sequenced by Sandron Coulson back in the 70s. This sequence, this little bacterial virus, is actually quite interpretable because of course there are statistical regularities. There's an ATG start. There's one of the three stop codons. There's an open reading frame. We can walk through a genome like this because of that bacterial genome. It tends to be packed with open reading frames. And we can do a pretty good job of interpreting this genome. There's interesting stuff going on in the gaps. And sometimes there's an interesting little RNA. I'm quite interested in little RNAs that have functions, as Eric was also talking about. Once we get into the human genome and the big vertebrate genomes, we have a bit of a problem. Only one, two, three percent of it is coding, depending on who you're talking to. And one of the most important signals that we can try to use is sequence conservation. We line up a bunch of genomes and we say this set of bases is essentially the same across some clade of evolution. Really key in our ability to do this kind of analysis with the human genome was the development of genome browsers by Jim Kent and David Hausler, the ensemble browser by Ewan Burney, by having the ability to align large quantities of genome, which has been done by Webb Miller's group, among others, and then the ability to take those genome alignments and calculate some simple statistic of how conserved do I think the various bits are, generating these plots that you can sort of see in blue that spike out, of course, on the exons, in this case of a single gene, the p53 gene of the human, and then also spiking out on some other areas of non-coding conservation. Adam Seples' program, Fastcons, underlies a lot of the calculations that people use for getting a quick look at this conservation. Those little spikes outside exons represent unexpected, well, we expected there to be regulatory sequence. We know there's transcriptional regulatory stuff. We know there's enhancers, there's promoters, there's what have you. But now we can use the conservation to really tell us where that stuff is likely to be. And so that's really the answer to the first sort of biological thing that I said. Why did we sequence so many flies? Why did we sequence so many worms? You can sit down and you can do a power calculation and you can say, okay, what I'm really trying to do is I'm trying to count the number, here's a region of DNA. I've got the human, I've got the mouse. I'm expecting about 40% difference between those two sequences by just neutral drift, by evolutionary clock. So if it's 100 nucleotides, I expect 40 differences. I only see 10. Should I be surprised? 10 is a small number, 40 is a small number. You can do a power calculation of for a given amount of conservation, how much do you have to drive it to distributions apart before you can reliably tell that you've got a conserved piece of DNA, given the size of the human genome, which is quite big, so you're gonna have false positives. Greg Cooper, Aaron Sidao, and others, and then followed by me, have done those kind of simulations and power studies. And if you wanna drive the resolution down to single nucleotides or five or 10 nucleotides, you can pretty quickly convince yourself that for typical distances between vertebrate genomes of like 0.2 to 0.4 substitutions per site, you're gonna need tens to even hundreds of genomes lined up. So it's not so much that we're interested in the platypus per se, but it's one of the 100 genomes or the 1000 genomes that we're gonna try to line up against the human to try to figure out what's functional in the human. And not just, so that's a very crude calculation. It's conserved, must be doing something interesting. What you can also do is you can look at the pattern of conservation and now you can do much more interesting things. For instance, if you're dealing with a coding region, obviously, there's a pattern of conservation that tends to respect the triplet periodicity of the genetic code. So you can do something like take a region that seems to be conserved. In this case, this is a poster child for a long intergenic non-coding RNA, a gene called SRI1, a gene that when it was first cloned had a truncated CDNA that went to this point and you'll notice the ATG start code on there. And then you color in where all the mutations are and you color third position changes red and then first position's green, second position's blue. And you can see most of the changes here are in red. They're respecting the frame. There's two insertions in the picture that I pulled out. One of six, one of three, again, respecting frame. And when you do this over the entire aligned regions of the SRI1 non-coding RNA, it's pretty clear that this is a coding gene of 232 amino acids in the mouse and 236 amino acids in the human. Now, when the first paper came out, this is 1999. They didn't have as much data as we've got now. And this, if we went deeper into the SRI1 story, it gets more and more murky because it does look like the RNA has some function independent of the coding region, but that's a different story. This ability to recognize coding regions and discriminate them from other conservation is something that's now at the heart of a lot of computational gene finding methods that are trying to harness the vertebrate sequences that we have available or the drosophila sequences that we have available or the senorebditis sequences, work that was really pioneered in bacteria by Jonathan Badger, a graduate student in Gary Olson's lab, but is now sort of fundamental to the field. Now, not just coding regions impose their own evolutionary constraints on sequence, but now once you have that idea, you can say, oh, I expect transcription factors because of the way they contact DNA to show particular patterns of conserved bases and then the middle would not be so conserved because of the way they sort of reach out across and leave a little gap in their conserved binding sites. Or in the case of my laboratory interested in RNAs, you can say there should also be a constraint imposed by RNA secondary structure, at least for structured RNAs. So it's that I could imagine making a statistical test for you give me a aligned pair of sequences. I can do a statistical test for whether the pattern of changes here showing four changes is randomly distributed. Sort of every column is independent or whether the four positions are respecting frame. They tend to be in the third position or they tend to be respecting Watson-Crick base pairs, preserving Watson-Crick base pairs in a correlated fashion, which is a feature that we see in lots of structure RNAs. You can formalize this and I'll talk in a few minutes about how we formalize this. There's now a sort of Lego box of tools that we use computationally to build this kind of statistical test for sequence analysis. And you can turn this kind of approach into an RNA gene finder, something that will look now for structural RNAs in conserved regions of whatever genome you're looking at. And it's been very successful in bacteria. Signal noise makes it very problematic in the bigger genomes. But there's been a lot of great work from Ival Hoffacker and Jacob Peterson and other people in taking this basic approach, which was developed by my wife Elena and actually get it to scale to the big genomes. And you can, one of the things I love about this field is that you can find little subtle effects sort of the way a hacker will say, well, if I heard you typing on a keyboard, the pattern of spaces between your pauses is informative about what you're typing and so I can figure out your password if I've got a microphone close to you. We can do the same kind of thing for genome sequences. We can look for very subtle things that evolution is putting on the sequence. We can detect those patterns. This is just one of many examples I could have pulled. This trick only really works in bacterial genomes. I wish it worked elsewhere. The graph is I'm sweeping across four million bases of E. coli and I'm gonna do a very simple thing. I'm gonna count the number of Gs I see and I'm gonna count the number of Cs I see. And as I just go across the top strand, the Watson strand and I'm gonna plot the excess Gs and what you get is you get that plot. That's very non-random. It's not much. There's an excess of about 20,000 Gs as you go across and then that excess goes away and then it starts coming back. Turns out that's the terminator and that's the origin of replication. And remember that E. coli replicates like this. And what happens is there, it's not actually understood why this is but one of the models is that the lagging strand replication is more solvent exposed and more prone to deamination of the C because that's a water-driven deamination. And so you get a depletion of C on the lagging strand and that pattern shows up. If you believed that, you'd expect that to also show up for transcription. It happens that E. coli also, its transcription direction respects its replication direction which is probably leading to why the signal is so clean in E. coli. Phil Green, Arian Smith and others try to turn this idea into an RNA gene finder. Doesn't work very well with single genomes or pairs of genomes but it might be that there's enough signal in the mutational biases of transcribed regions that we can use this to find things that have not yet shown up in RNA-seq or epigenome experiments. Things where evolution knows that the thing is being transcribed at some point in some single cell that would be difficult to measure experimentally. Now, I also showed you a little picture of a pond ciliate. So now let me get to that part. Once you're looking for little patterns, you don't have to look for just little patterns in all organisms that you think E. coli shares with humans, shares with the worm. You can now say, you know, I can exploit the fact that here's this weirdo creature in some weirdo evolutionary niche that has a weirdo evolutionary pressure. I'm interested in RNAs. I can take advantage of this little weird creature to find RNAs and then use homology search to get out of the little weird creature and find homologs in other organisms. So here's an example of the kind of thing you can do once you have tons of genome sequences lying on your disk. You can notice that a great sort of observational computational biologist, Gene Lowbury, has published a paper where he says, well, I looked at a bunch of bacterial genomes and I noticed something interesting. If you plot the optimal growth temperature of the organism versus the GC content of the genome, I sort of expected that the higher the temperature, the more GC rich the genome would get. But it doesn't happen that way. Bacteria and archaea that grow in high temperatures have other evolutionary adaptations to high temperature other than just strengthening the hydrogen bonds in their DNA. They hold their DNA together by making a reverse gyraase that overwines the DNA, burns ATP and puts positive supercoil into the DNA and other tricks to stabilize their DNA. But Lowbury looked at the GC content of structural RNAs, structural RNAs go off they make a little single strand, it folds up. Now to stabilize the structural RNA at high temperature, evolution does drive the GC content. And so GC content of the genome is sort of randomly varying but the structural RNAs tend to be tightly correlated.  What I'm gonna do is I'm gonna reach into the database, find me the most AT biased genome that still grows at a super high temperature. The most extreme one you can find is pyrocaucas furiosis. It's isolated from the Macaroon Volcano Island, Italy. I actually know some of the people who have the benefit of being able to go in their scuba gear and collect on the Italian islands. This thing grows at 98 degrees C, 60% AT genome. It is not something you would think of as an experimental organism, dies in trace oxygen. It's a strict anaerobic, grows on elemental sulfur, generates hydrogen sulfide. Nobody likes you in the floor that you're working on. It does have the advantage that because it's normal growth temperatures, 98 C, you don't need minus 70 to store your samples. You just put it on the bench and it thinks that's minus 70. But what you can do with its genome, so the easy part is looking at its genome sequence because that was done by not us. We could just reach into the databases and then say, okay, this is a pearl script sweeping across its genome, just counting GCs in windows and there's two tRNAs. So finding structural RNAs in this organism is completely trivial and when you do this, you find a bunch of little RNAs, 10s of structural RNAs that had not been discovered before, all of which were in known classes. So this was unable to discover novel classes of structural RNAs for bacteria or archaea. Then you say, okay, one of our big problems in interpreting genomes is that we don't know where a gene stops and a gene starts. So when I'm looking at cis-regulatory stuff, I don't know when the enhancer is for this gene or I don't know when it's for that gene or indeed whether it's transvecting over to some other chromosome. And if I'm looking for structural RNAs, I have a problem of just finding them in the first place because statistical signals are pretty subtle. So this is some ongoing work from a student in my lab, Silkyung Jung, working in collaboration with Laura Landweber's lab at Princeton, working with an organism that was pioneered by David Prescott at the University of Colorado Boulder and I was a graduate student there and I've always held this organism in my head as this organism is gonna be useful for something someday. Its adaptation is a bizarre one. It has in its macronucleus, it actually has two different kinds of nuclei and I won't go into all the biology, has a macronucleus that's a somatic transcribed nucleus and in the macronucleus, it has about 20,000 different chromosomes. There's actually several million chromosomes. The average chromosome size is two kilobases and these are individual chromosomes, telomere, gene, telomere, telomere, gene, telomere, telomere, gene, telomere, extra gene. For the most part, not completely, unfortunately, but for the most part this organism has identified all of its genes for us and put telomeres at the ends of them and so now we can just sequence the gene, I'm gonna say that's a gene, that's a gene, that's a gene and since protein genes are relatively obvious, not completely but relatively obvious, we can do a subtractive screen, just throw away all the proteins who cares. Everything else is either an interesting gene that we didn't know about or a structural RNA or something like that and that screen has gone and has identified a small number again of new RNAs. So let me close with a couple words about deeply things, forgive me, but underlying all of this is computer science, statistics, mathematics, it's being used to interpret these genome sequences and at the end of the day, what we're trying to do is we start with a sequence and we draw these cartoons where we say, okay, the sequence is aligned and we're trying to attach labels. I could show you a protein sequence and these could be domains in this case of the Dicer protein which was actually done by sequence analysis, Brenda Bass wrote a review where she said, okay, based on what we know about RNA interference, it's gotta have an RNA helicase activity, it's gotta have this, it's gotta have that and it turns out there's only one protein in C. elegans that has the right combination of domains of known function and that protein turned out to be Dicer and so that was an important computational clue in the early days of RNA interference, but I could be drawing a DNA sequence and enhancers, you know, this is how we draw things, exons, introns, we're trying to attach labels to sequences, that is, you show me a piece of sequence, what I'm really interested in, what label do I attach to that piece of sequence? The label is hidden and I'm trying to infer, should it be this label or that label? And it turns out that there's a big field of mathematics in digital signal processing and speech recognition that does this for whether it's a speech signal or whether it's an encrypted signal or whether it's the telemetry off of your car engine, which actually, I could tell you a story about how those guys are now using software developed in bioinformatics to do prototyping of engine telemetry and astronomical telemetry, which is amazing, the field has sort of gone full circle because we've put so much work now into adopting methods called hidden Markov models and stochastic context free grammars and other methods, probability models of attaching labels to sequences that are appropriate either for linear sequence analysis where you just say, okay, I'm just gonna do a sequence alignment or in the case of RNA, I'm gonna align pairs in a sort of nested sets. We have models to do that, they were introduced in the 90s by Gary Churchill, Gary Stormo, Anders Krogh, David Haussler, SCFG models for RNAs that's introduced by Yasu Sakakibara and myself at about the same time. Yasu was in David's lab at the time. And this has given us a real toolkit to build models, which gives me an opportunity to say that the most important tool in computational biology without doubt is the tool called BLAST and it's probably familiar to all of you. BLAST does a sequence alignment between a query that you give it in every sequence in the database and it looks for things that are significantly related. When we look at the BLAST algorithm as probabilistic modelers, we say, okay, this is an approximation, it's a two-level approximation, well, it's a multi-level approximation. It's doing sequence alignment where what it's trying to do is attach a label saying these two residues are aligned or these two residues are not aligned. This is an insertion, this is a deletion and so it has three states. It says I'm either gonna try to align residues X and Y or I'm gonna throw X out as an insertion or I'm gonna throw Y out as a deletion, so as an insertion on the other strand. Those are three states that I can move between. That's a Markov model. Where I go next is dependent on where I just was and so I have hidden states that I'm trying to infer connected by arrows, that's a hidden Markov model but the arrows in BLAST are associated with mostly zeros implicitly and then there's a gap open penalty every time I open an insertion, either way and there's a gap extend penalty every time I extend an insertion by another residue on either strand and then there's a score for aligning the two residues that's due to the Blossom matrix from the Hanukkahs. The bottom line is we now understand that those zeros really should be probabilities or at least if you're thinking in this sort of probabilistic inference context and we can represent BLAST's internal model as a probability model and now we can do things like, well instead of just giving me the optimal alignment, some overall possible alignments and then tell me what's the probability that you're really confident that this residue aligns to this residue integrated overall possible alignments and other things. The field has been trying to do that for a while and now I wanna make a somewhat sociological point. There are lots of research and a big literature on developing better methods for sequence analysis. Those methods going to journals like BMC Bioinformatics and what have you and none of you read them. Then there's another field which takes very important algorithms and then reduces them down to their bare bones and speeds them up on particular hardware and there's lots you can do on modern hardware. For instance, this paper from Michael Ferrar who was until his untimely death a couple months ago, unfortunately, our chief software engineer. I recruited him from Boston. Notice, unaffiliated, this is also a sociological comment. He was not in a university, he did this particular paper in his spare time and then was recruited by both Bill Pearson and me to actually become a biologist. So you can use SIMD, what's called single instruction multiple data being driven by the graphics industry. All modern chips are capable of vector paralyzed processing. They're being driven to this by all the games everybody plays. You can take advantage of those instruction sets to make Bioinformatics software but the question is, who's gonna do it? The difference between writing a piece of software that works for your BMC Bioinformatics paper and a piece of software that runs fast and it can be used by the rest of the community is an enormous difference and it's to blast credit that one of the things that's underlined that is not only terrific theory from Steven Altshule and Sam Carlin and others and great algorithmics from Gene Myers and Warren Gish and others but terrific software engineering from the NCBI team. It's very rare to get that kind of investment in a piece of software and that's really one of the things that makes blast fast. Frankly, in our lab, we are frustrated that blast was written 20 years ago and it's now difficult to adapt it to the stuff that we think we know with probability modeling now. There's an effort in my lab to now take the hidden markup model methods and speed them up to blast and we've been investing a lot of time in this to get the engineering up to snuff. One of our goals is for you to be able to do an HMM-based search like a blast search on a web server where you get the response no matter what the size of the NR database in 100 milliseconds or less. That is faster than a Google search so that you can do interactive searching rather than waiting for a batch job and actually start exploring what sequence space looks like in all these wonderful organisms and we're within an order of magnitude of being able to do that. Again, reinforcing the point, I don't want to belabor this but we have been working to engineer tools that can do this kind of probability analysis. The point I want to make on this slide is the difference between a tool that runs good enough for a publication. I could probably write hammer in 1,000 lines of C code. The actual code which is not as good as we want is 44,000 lines and similarly so we maintain a big code base trying to make this stuff useful and that requires engineering. But it also means that I have a two-faced view of what we're looking at and it's sort of not great for one's mental health at times but we are very much immersed in two levels of code, one the level of looking at all these wonderful genome sequences but also the level of our C code of trying to interpret all that and both of those are evolutionary artifacts that are difficult to understand. And with that I'll stop and in contrast, and now at sort of counterpoint to Eric's world, my little lab, which is a husband and wife team, we've been working together for longer than we've been together as a couple. Very small laboratory at Janelia Farm where we're pretty much dedicated to building these kind of tools for the community to use and I'll stop and I'd be happy to take questions. We have time for a quick question if someone's gonna race to a microphone. Is that someone racing to a microphone or racing out the door, racing out the door? Okay, so in that case Sean will be available at the break. We're gonna take a break now then. I'm sure he'll talk to you if you have any questions.