So, as James mentioned, my name is Matthew Garrett, I work at CoreOS, I've recently been concentrating primarily on security work and part of that is a focus on trust computing measured boot. So, I'm going to be talking about some of the work that we've done on making this work out of the box and what still needs to be done for this to be generally useful, why we care. So, security of the boot chain is a pretty vital component of any other security solution. If someone is able to tamper with your boot chain, then any other security functionality you have available can be subverted. If it's possible for me to interfere with your kernel, then it doesn't matter how much self-protection the kernel would otherwise have. If I've modified the kernel, I have arbitrary code execution in the kernel. If I'm able to modify your firmware, I'm probably still able to circumvent any security functionality the kernel has, and the boot load is in the kind of intermediate position. The boot load can modify the kernel before it passes control to it, and there's no way for the kernel to then verify itself once it's running. In terms of what we have deployed in the Linux ecosystem so far, the primary protection we have against this is UEFI secure boot in the desktop and server space. UEFI secure boots a firmware level feature where the firmware verifies a signature on the boot loader before it executes it. The boot loader in turn then verifies a signature on the next step of the boot process, and so on. You can then build an arbitrary level of signature validation before you give up and just let arbitrary code run. Microsoft have extended this on the Windows RT platforms to the point where the secure boot root of trust ends up locking down the individual execs you can run in user space. You can't run any user space binaries unless they're signed by Microsoft. Outside that world, in the embedded world, we also have a large number of different solutions, many of which are rooted in the SOC itself, the firmware in the SOC. The SOC will not execute additional firmware code unless it matches either a hash or signature that is actually flashed into the SOC itself at provisioning time. So in that world, again, you have a chain of trust that goes to an arbitrary level and it's rooted in the SOC. The problem with these is that they rely on the security of the firmware or of the SOC. Whichever component is performing the initial validation, if that can be subverted, then you have no actual trust. So if someone's able to modify the firmware, then there's not necessarily any actual validation of the firmware before you hand off control to it. Beyond the signature validation, it's conceivable that you can modify the lowest level and then own the system. But the other thing, which that allows attacks on the system, but the other thing is that there's no way to prove that the verification happens. The only thing that is doing this validation is the prior stage of the boot process. Once you're running a kernel, you can't ask that kernel whether it's booted in a verified manner because if someone was able to boot an unverified kernel, they could have just modified the kernel to lie to you about that. And then suddenly you're happily sitting there running something that says, yes, I booted securely, and it's actually not secure. It's changed behavior behind your back, you're in a bad position. It's easy to think that, well, this doesn't matter if the system's been signed appropriately. Then who goes? The chances are I can just assert that this route of trust is sufficient. If materials I expected to be rejected is rejected and materials I expected to be accepted is accepted, that's probably okay. We're seeing people starting to move towards providing bare metal server hosting. So a lot of people want the agility, the convenience of cloud hosting, but they also want the performance of bare metal. They don't want necessarily to use your virtualization infrastructure, they just want to use your cloud management infrastructure. And we're seeing vendors start to offer bare metal. The problem with offering bare metal is that often if you have route on a system, if you have the ability to access hardware resources, many of these systems have not been designed to be resilient against that. The trust model has basically been if you're an admin, then access to the hardware is outside the trust model. This has resulted in it being possible in various cases to basically escalate from route into the system firmware, potentially in a persistent way. Such that the next time somebody installs an operating system on there, there's now an agent running in the firmware that can compromise that system. So say you wanted to, if you know that a company you want to compromise is using a particular bare metal hosting provider, you could just get a large number of short term leases on hardware in their facility, install this persistent firmware agent, and then get out, and then later use that firmware agent to exfiltrate data about the target company. And secure boot will not help you there. Verified boot is not going to help anything. There's no point at which the initial system firmware on X86 class hardware will necessarily be completely verified. This is slightly untrue. Intel's boot guard allows some degree of control around this, but there's a lot of systems that don't actually chain all the way up through boot guard terribly well. There have also been cases where people are perhaps legitimately concerned that if their laptop is taken away from them when passing through borders, that the firmware may be modified in a way that results in system compromise. The degree of, well, how much of this is paranoia and how much of this is legitimate concern, I'm not going to make any firm statements about, but it's the kind of thing that you do perhaps want to think about. Obviously, we can't protect against hardware attacks in a particularly straightforward way. If someone adds a hardware key logger, there's pretty much nothing we can do in software, or even using some degree of hardware protection to protect against that. Something people have talked about is the publishing X-ray images of known goods laptops such that you can then X-ray your laptop and check whether anything appears to have been added to it. And that, yeah, well. But there are a bunch of other cases that we can protect against, and measured boot is a component of this. So measured boot is based around trusty computing, which got a fairly bad reputation in the early 2000s as it was very easy to portray it as a basically DRM mechanism. In practice, that was not necessarily a particularly plausible outcome even then, but what we've certainly seen is nobody's ended up trying to use trusty computing for DRM. There are lots of reasons why it's very difficult to actually deploy it in a particularly user-hostile way. But the hardware components of trusty computing is the trusted platform module, which is a small chip. The idea is that the spec defines the pinout for any given bus. So they're manufactured by several different manufacturers. You should be able to buy one that has the functionality that you want and drop it onto a board, and it will work regardless of which vendor you got it from. They mostly differentiate based on the amount of NVRM storage they have and the performance. Otherwise, they're pretty much all the same. In practice, they obviously being basically embedded devices have a bewildering array of different bugs. But most of those you don't end up hitting. It could be a lot worse. TPMs do a lot of different things, but I'm going to focus on some functionality provided through what are called platform configuration registers. Platform configuration registers are basically between 16 and 24 typically registers on the TPM. Those are not directly accessible. You cannot directly write them. You have to ask the TPM to store some data in there, and then something magic happens, so I'll get to it in a moment. Platform configuration registers are commonly referred to as PCR, so I'll probably end up doing that for the rest of the presentation. On TPM 1.2 devices, for PCRs, there are a static set, and then there's 20 bytes each, which is the length of a SHR1 hash. On TPM 2 devices, you can have several different banks of PCRs, which vary in size depending on the hash algorithm that's used for that bank. The idea is that rather than limiting the spec to a specific hashing algorithm, TPM 2 allows you to use a range of different hashing algorithms based on your feelings about the security of those hashes. So when we boot, we measure things and we store those measurements in the PCRs. Measurement is the idea there is to store something that represents the next part of the boot process. So the firmware measures the boot sector of the hard drive. The boot sector measures the rest of the boot loader. The boot loader measures the kernel. Each of these measurements gets stored in the PCRs. And through convention, different PCRs are used for different purposes. Zero through seven are used as part of the boot process and are defined by the spec. Once you get past seven, then that's up to the operating system for the most parts. PCRs, as I said, are not directly addressable. If I want to store a measurement in the PCR, I need to call a TPM function that says, here is a measurement and here is the PCR, so I want it to be measured into. Now, if I were able to just say, set this PCR to this value, if at any point you booted some untrusted code, I'd be able to go back and say, OK, set PCR 0 to this known good value, set PCR 1 to this known good value, and just go through and overwrite the history of what was actually booted and replace it with known good values. And that would defeat the object entirely. Instead, when you perform a measurement, the existing value of the PCR is taken, the new measurement is taken and they're concatenated. So you now have, for TPM 1.2 devices, a 40-byte value and then the hash of that is taken and that hash is stored in the TPM. So this means that you can't set a value unless you perform exactly the same sequence of writes or, alternatively, if you can somehow calculate the value that you would need to measure in order to get from value A to value B. And doing that involves basically breaking SHA-1, which is not yet practical. So we're not too worried about that. But TPM 2.2 devices allow you to use more complex, secure hashing algorithms, so we have even more confidence. Right now, SHA-1 is basically sufficient. So this has that property. The property that it is very difficult to guess a value unless you perform the same sequence of writes. But that also means that if you change the order of the writes, you'll end up with a different value. If I measure A, B, I'll have a different value to if I measure B, A. And that'll become relevant later. Now, outside the TPM stores in system memory, there is also a log and each measurement event usually results in the log being extended. So you add a new entry to the log describing the value that you wrote into, that you measured. So where it came from and what the hash of it was. As I said, that's just stored in RAM, so there's no direct protection of this. Any untrusted code can overwrite the log and make it have arbitrary contents. That's not as much of a problem as it sounds like and again, I'll get to that later. So in order to have a measured boot, you need to be able to measure each component of the boot chain. Now, first of all, that requires you to have firmware that implements this. Pretty much all systems, all x86 systems shipped these days have firmware that does this. So, and we're also starting to see this appear on some ARM devices. Assuming you have firmware that does this measurement and assuming that the firmware actually performs the measurement in all cases, I've seen systems that are fine at measuring stuff booted off disk and which forget to measure anything that you pixie, which is kind of a problem. But I think the one of those that I saw, I reported and they seem to have fixed it. So that's great. But you need support in the later parts of the boot chain as well. And the big issue here is that no standard Linux bootloader does this out of the box. In the past, we had Trusted Grub. Trusted Grub was a set of patches on top of the old legacy version of Grub 0.97. It was not particularly good. The first thing is that that old version of Grub is unusable in a lot of modern contexts. It doesn't have a lot of functionality that people want. The quality of the Trusted TPM support codes on top of it was also not great. It's never really a good sign when patches have completely different code formatting to the original code. It usually ends up indicating that maybe people weren't paying that much attention to what was there in the first place. So this worked, but it wasn't particularly nice. A German company called Rosentrotz have a Trusted Grub 2 project. This was originally being developed by a group called Cyrix, who now seem the GitHub repo now, as I discovered today, redirects to Rosentrotz. So I assume that something interesting has happened there. And this was a version of Grub 2, which has a TPM support, but it had no UEFI support. It only had support for using the legacy BIOS interfaces to the TPM, and it only supported TPM 1.2 devices. It had no support for TPM 2 devices. So when I was starting to think about this at CoreOS, I basically decided to just... Well, I actually started on this before seeing the series code and ended up just writing my own support code, in some cases based on the code from the original Trusted Grub patches and, in some cases, just written from scratch. So you think this is okay. I now have a bootloader that supports measuring the kernel, and arguably that's a complete job. Things are a little more complicated than that. There's a question of what more should I measure? And the reason that you have to ask that is, is the kernel the only thing that influences the booted state of the system? If I boot a kernel and I measure the kernel and I know that the correct kernel was booted, was there anything else that could be done that would alter the security of the system? And the answer is yes. The configuration that's passed to the kernel is very relevant. The command line that's passed to the kernel is very relevant. Grub 2 has a significant scripting language that allows you to basically, if you want to, do anything you want, including poking values into RAM or port IO.  You could disable the IOMU before passing controls to the kernel, or possibly even overwrite the tables that cause the kernel to believe that there is an IOMU and therefore disable that protection entirely. And the kernel measurements would still be the same. Measuring what happens during the grub phase is pretty important in terms of being able to say the system is where it wants it to be. So the traditional approach here was to measure each component into a separate PCR. This is a little less than ideal, firstly because there's not a unlimited number of PCRs and secondly because some of those PCRs are being used by other things. PCR 10 I believe is used by the Linux IMA project to measure user space binaries and have the kernel provide measurements of those. If you've already started putting stuff into PCR 10, that interacts somewhat poorly. So having each individual component put into a separate PCR, it doesn't scale. And if you want to measure, say, so if you wanted to measure grub into one PCR, the grub configuration into another PCR, the kernel into a PCR, the initRMFS into a PCR, then you've already used five PCRs. And then if grub supports loading modules, are you putting each module grub loads into a separate PCR or all into one PCR? And then if you're loading, putting multiple things into one PCR, then the ordering starts mattering. So reusing PCRs is important here, the order of loading masses and you probably want to avoid those constraints. So if you're reusing the same PCR, if you're trained to actually measure everything into a small number of PCRs, which is basically a requirement, you also have to deal with the problem that unimportant configuration changes will alter values. If you measure the entire grub configuration and then someone changes a comment, does that result in a change in measurement? If someone changes a particular constant in the grub configuration that doesn't have any security impact, that will again alter the measurements. And it's not really straightforward to teach grub everything it needs to know about whether something matters or not because there is no single correct answer. You have to deal with this differently depending on your security policy, depending on your threat models. So just sticking everything in PCRs and then looking at the static values of the PCRs is suboptimal for multiple reasons. This is where we go back to the log file that's not actually a file, using the TPM log. Now, as I mentioned, the log is just in memory. So an untrusted component can overwrite the log. And if you're just looking at the log, that means that a hostile component can fool you. If you trust the log, then a hostile component can cause you to believe that everything's okay. But there is a way around this. As I mentioned earlier, the mechanism that the TPM uses to change the values in the PCRs is very straightforward. It's easy for you to look at the log and replay the log in the same way that the TPM extended the measurements, extended the PCRs during the boot process. And you should end up with the same values that are in the TPM. If the values in the log file end up giving you the same answers as the values that's currently in the TPM, you know the log file has not been tampered with. And at that point, you can look at the individual log entries. So there's two ways of doing this. And the traditional one has been to just have the log contain a description of the binary and the hash of that binary. So you might have kernel and the hash of the kernel. You might have a graphics module and the hash of that module. That's great for looking, that's great for being able to measure a large number of small binaries. You're now able to look at the log, verify that it hasn't been tampered with and then say, okay, I don't care about the order that these things were loaded in. I merely care that each of these was appropriately, was measured and gave me the measurements I expected. That doesn't work hugely well for configuration. So again, as I mentioned, there are certain changes that you could make that are irrelevant. And there are certain changes you can make that are relevant depending on your threat model. So instead, you can have a log entry that contains some text, like the commands that grab executed and then the hash be the hash of that text. So in this case, when looking at the log file, you can verify that the text hashes to the hash that's in the log file and then you verify that the log file gives you the correct PCR values and now you know that the text in the log file was the text that was executed by grab. So you can have a policy that describes each binary and says these are the set of permitted binaries and you can also then have a policy that provides a set of regular expressions that you can match against the text entries and then you can have a policy that describes something like this grab command followed by any argument is acceptable. This grab command can only be followed by these three arguments in order to be valid for my particular use case. But if you're writing policy, where does it come from? Ideally, you don't want to have to write a brand new policy yourself for every release of an operating system and for every operating system you need to support. So CoreOS, as of three months ago, every release that we build ships with a file describing the known good hashes for every component of the operating system and a policy that describes a valid grab configuration from our perspective. And that's not complete because unfortunately you still need some way of validating the firmware values. The firmware measurements are things that are provided by the manufacturer, they're not things that are provided by us. And at the moment, I'm not aware of any vendor that provides known good values for their firmware. Until that happens, there's still an amount of manual policy development work involved. But that's a case where you can have a known good system, you can extract the firmware values from the known good system and as long as you never then upgrade the firmware, those values will remain static and everything will be fine. But we're still faced with a few problems. In the CoreOS case, the init ramfs is something that's statically generated by us and is actually attached to the kernel image. So it's easy for us to provide a static value here. Most distributions generate the init ramfs at install time. And what's even worse is that even if you don't change the contents, when you build the init ramfs, you will typically get different output because timestamps end up being embedded in it. So something that we need is a reproducible init ramfs. And ideally we also could do with known good generic init ramfs is being distributed by vendors, operating system vendors. There will be users for whom these will not be suitable but this at least gives you a case that would work for the majority of people and sidesteps that problem. Another thing is where do we store boot data? So you may need some amount of data that is normally stored in the init ramfs and which is relevant to the TPM. But if you put it in the init ramfs, then the init ramfs measurements will vary and so suddenly you're outside your known good state. So one answer here is on UEFI systems, some of this information can just be stashed in UEFI variables which are portions of NV ram that are available in the firmware but also in the running operating system. The TPM itself also has a small amount of NV ram so you can do things like put data inside the NV around the TPM and have it stored there. And why, what sort of boot data am I talking about? There are some things that you can use the TPM for that are particularly entertaining. One of these, one problem with using a TPM for boot measurement is that I've been talking about this as it needs to improve local security and that's still very difficult because even if you have value stores in the TPM, even if it's impossible for a compromised kernel to cause the TPM to give you good values, you still need to get the TPM values by asking the kernel and it would be possible for the kernel to then lie and give you back fake values that didn't actually come from the TPM. What you can do is ask the TPM to encrypt some information and only decrypt that information if the PCR values match a defined policy that you imposed when you encrypted that information. So that means you can encrypt a secret with the TPM and the TPM will only be able to decrypt that secret if the boot process has not been tampered with and then it can display that secret. Problem with a static secret is that it's possible for someone to just shoulder surf you, steal that static secret and then print the static secret even if your system has been tampered with. So the way I got around this was rather than using a static secret, I'm sealing a seed for a TOTP, one-time password generator.  So this is familiar to anyone who's used 2FA on Gmail, for instance. And then when the system boots, it shows you a six-digit number. And then you check on your phone and you check whether the six-digit number that your phone is displaying is the same as the one that your laptop's displaying. And if so, then the TPM decrypted this information and you know that your boot process has not been tampered with. If the number's different or if there is no number, then you have to be suspicious. Another traditional thing that's used here is disk encryption keys. The system will refuse to decrypt your disk unless you booted in a good state and then the TPM handed over the AES secret for your disk encryption. We can also store SSH keys on the TPM such that if the system's been tampered with, you won't be able to SSH into another system. The basis of this is, a problem that we haven't solved here yet is if you upgrade the operating system or if you upgrade the firmware, then the PCR values will change and suddenly the TPM will refuse to decrypt your disk encryption key or your TOTP key.  So we need a mechanism that permits automatically unsealing and resealing the information when we perform system updates and that's a problem that we really haven't started solving yet. There's a lot of details that make this quite difficult to do. So even once we've solved the rest of measure boots, in order to make it useful, we still need to solve this one. So there's an argument that this is all made much easier using Intel's TXT technology. TXT is what's called a dynamic Roos of Trust rather than a static Roos of Trust. What I've been describing so far is a static Roos of Trust. You have the firmware as this Roos of Trust and it measures each thing beyond the firmware measures the next component. You have to get to the desired outcome by performing exactly the same sequence of operations. A dynamic Roos of Trust allows you to remeasure at basically arbitrary times and only really cares about the stage you're in, not how you got there. And this simplifies a lot of stuff. It doesn't really matter if someone modified the boot process as long as the boot process still resulted in the same kernel being loads in the same place and had not tampered with any other components. TXT allows you to, just before exiting the boot loader and passing over to the kernel, do that measurement, write those values into the TPM and then go from there which makes writing policy much more straightforward. So in theory, TXT lets us sidestep a bunch of these problems. In reality, not so much. TXT does not currently solve many of these problems because well, TXT does not have meaningful support for secure boot. TXT does not allow any signature verification of components. In fact, the T-boot loader that is used in order to perform TXT validation, TXT measurement when booting Linux does not, it uses the multi-boot protocol which doesn't have any real concept of signature validation and so there's no way to verify that T-boot itself has not been tampered with which is not ideal. It actually gets a little worse than that because many systems including servers are now booting with UEFI by default run in legacy BIOS. And TXT nominally supports UEFI boots as long as you entirely disable UEFI functionality in the kernel which is a problem because then you can't tell the firmware where your boot loader is after install. There's various other things that we need to do with UEFI. So it's basically at the moment incompatible with the way we boot operating systems. So until that's fixed, TXT while theoretically nice is just not particularly helpful. So quick summary, in order to get to a point where we have measured boot out of the box we need to ship boot loader support. Now the code for that's written. The big problem is that CoreOS is mostly deployed on servers and so I have full expectation that as we push this out to desktops and laptops we'll find a large number of fascinating firmware bugs that will break in all kinds of entertaining ways and when your boot loader breaks users tend to get upset. We need to ship known good measurements. We need to have integration with firmware updates so that we know when we expect the firmware values to change and ideally we need to convince vendors to start shipping the known good values with firmware updates so we can do this automatically rather than just having to assume that because we perform the firmware update the new value is good. And ideally we also need to terministic in a trimFS generation so that we know in advance what legitimate measurements are going to be there. So that's everything I want to cover and we have five minutes until the break. So questions? Are there plans to upstream the trusted grub codes to upstream grub? So yes, there are plans to. After a certain amount of discussion with Richard Solman I finally managed to convince him that we could use TPMs for purposes of good rather than being intrinsically evil. Which I really hope is something that I can put on my performance review. In addition to that, so in addition to that grub2 recently gained new maintainers who are much more active than the previous maintainers so I'm going to start working with them. The big problem here is that as a GNU project grub requires copyright assignment and some of my code is based on the original trusted grub code and I can't assign the copyright of code that wasn't written by me. So I'm going to need to figure something out about that. It's somewhat annoying because this is code that which you basically can only write one way. Especially within the constraints of having to fit this in a 512 byte MBR. So ideally I'm going to be able to track down whoever did this originally, hope that they're still employed by the same people and get them to also sign a copyright assignment because otherwise we're going to need to jump through a few hoops on that front. It is theoretically possible that the grub maintainers could be convinced to not do this to not require a copyright assignment. I should say that I'm a member of the Board of Directors of the FSF and therefore I have something of a conflict of interests in this situation. So I'm not going to say anything on that topic right now but yeah, ideally this will be upstream. I do not want a situation where people have to carry a bunch of out-of-tree patches just to get this kind of functionality. It's sub-optimal for everyone. Anyone else? Have I implemented anything to hand over the event log from the bootloader to the kernel? Right, so in TPM 1 space there's an ACPI table which just provides a pointer to the TPM event log and that's very nice and straightforward because that is still there after you've got into the kernel. You can just read it out of RAM. In TPM 2 land, that's not the case. When you exit from the firmware, that memory is reclaimed and so the kernel cannot access the event log anymore. I've got a written but as yet untested implementation where the EFI stub loader within the kernel finds the event log and then passes that to the kernel as loader data and then that gets pulled out and then handled by the TPM 2 code. As I said, this is not as yet meaningfully tested and with luck I'll get the opportunity to do that in the near future and then we can submit that for review and inclusion. OK, so the question was why is there a difference between TPM 1 and TPM 2 in this respect? My understanding is that the reason was people didn't want to have to have this lump of RAM that they couldn't touch sitting around if they weren't using the functionality. So this way it's basically opt in. There's no requirements that the operating system maintain this RAM. Instead, it can be free to use by the operating system if it doesn't care about it. So if the operating system does care, then it has to explicitly do something about that. If it doesn't, it's fine. So basically that is my understanding. Right, so the point that was mentioned there was that UEFI does not necessarily imply ACPI and therefore requiring that an operating system have ACPI in order to be able to obtain these logs was unhelpful. So I think we're out of time. I'm around for the rest of the week. If anybody does want to chat to me about any more of this, let me know. There is a further TPM discussion session tomorrow morning, which I'm 10 something, sometime around 10. So please also come along to that if you're interested. Thank you.


Welcome, I'm Mark up to Grove, the president and CEO of the LBJ Foundation. What a pleasure it is to be back home among so many friends. I was telling members of the LBJ library staff and the LBJ Foundation staff, I'm like a bad cold, you can't get rid of me. Before we get started tonight, I have a few things by way of housekeeping. The first is I want to thank our generous friends, sponsors, St. David's Healthcare, the Moody Foundation, and Tito's Handmade Vodka. We very much appreciate your support of the Friends of the LBJ Library. Thank you. Later this week on March 1st and 2nd, the LBJ Library is participating in a citywide giving campaign, Amplify Austin. Our goal here at the library is to raise $5,000 for bus scholarships to provide LBJ library field trips to students from underserved Texas schools. This is a wonderful cause and we would ask that you consider making an online donation, however small, to help. Tonight we'll be treated to the premiere of Hulu's upcoming 10 episode series, The Looming Tower, based on Lawrence Wright's Pulitzer Prize-winning book of the same name, which will be sold in the lobby after the program. In a moment, Larry will come out to cue the episode and I will just warn you quickly, there is some very, very harsh language. You can all take it, I assure you, but I just wanted to give you a fair warning. Also, the credits will roll afterward and then you're going to see a preview of upcoming issues. After that, the screening will last 52 minutes and afterwards I'll moderate a conversation with Larry. Ali Soufan, the former FBI supervisory agent who was depicted throughout the series and was a pivotal player in this story, as you'll see. And actor Michael Stulbarg, who portrays Richard Clark, White House counter-terrorism officer. Michael is on a bit of a hot streak right now. He's had roles in three films that have been nominated for the Academy Award Best Picture, including The Shape of Water, The Post, and Call Me By Your Name. It's my pleasure to introduce our fellow Austinite Larry Wright, a renowned author, screenwriter, playwright, and staff writer for The New Yorker Magazine. His book The Looming Tower, Alcada and the Road to 9-11 was published over a decade ago to international acclaim. In addition to winning the Pulitzer Prize, the book spent eight weeks on the New York Times Best Seller's List and has been translated into 25 languages. Larry is an old friend of this institution. In 2006, he was on this stage to preview, rather, his one-man play, My Trip to Al-Qaeda, which he later took to Manhattan for the New Yorker Festival. And he returned here in 2015 to discuss his best-selling book Going Clear, which he also later co-produced as a documentary for HBO. Ladies and gentlemen, please welcome our friend Larry Wright. Well, thank you so much. It's very meaningful for me to be here in Austin where I live and have been coming to this library. We moved here in 1980. We've been coming here and been members 38 years. So this institution and this city means a lot to us. I want to talk a little bit about, you know, The Looming Tower, the book really began on 9-11. You know, I was in our house trying to decide how to get into this story. If you remember back then, the planes were grounded and I couldn't get to New York. So I was reading obituaries that were streaming online and looking for some way to find a story to tell that would bring this vast human tragedy down to human level. And on the Washington Post site, I found an obituary for John O'Neill. And he was the head of counterterrorism for the FBI in New York at the time. And the obituary made him seem like something of a disgrace because he had taken classified information out of the office and he had been let go from the FBI. And he became the head of security at the World Trade Center. And so my reaction on reading that was, so he didn't get bin Laden, bin Laden got him. And how ironic. And the more I learned about John O'Neill who was played in this episode by Jeff Daniels, the more I realized that it wasn't ironic. It was Greek in its tragedy. His friends told him, John, you'll be safe now. They already tried to attack them. They did attack the World Trade Center in 1993. And he said, no, they'll come back to finish the job. So he instinctively placed himself at ground zero. Ali Safan came into my life indirectly when I was researching a movie I co-wrote called The Siege with Denzel Washington who actually held the job in the movie that John O'Neill held in real life. But Ali Safan was undercover and I heard about this young Lebanese-American FBI agent so I wrote a part that Tony Shalu plays in the film. And then just three years later, 9-11 happened and I began researching the book. And I got to meet Ali Safan. And come to realize what a great debt of gratitude the American people owe to him for being the one who uncovered the names of the hijackers, the one who unmasked Khaled Sheikh Mohammed as the mastermind of 9-11. There's so much more about Ali Safan that we can talk about later. But I wanted to give you a little bit of that background and hope you can appreciate this young French-Algerian writer or actor named Tahar Raheem plays Ali and I think he's just going to be a breakout role for him. All of these actors are terrific. I love them. And so I hope that you will come to appreciate them somewhat as much as I do. And I'll talk to you after this show but thank you so much for coming. Well, welcome and congratulations. Thank you. This is an extremely compelling premiere to the series and I can't wait to see the rest. I'm sure my audience agrees. Larry, let me start with you. You talked a little bit about the genesis of this book as you were queuing up the episode. How do you tackle a subject this big? When you're watching 9-11 play out, you get this idea to do this book, but where do you begin? Well, when I tell a story that's a complicated story, I always look for individuals who they don't have to be famous or even very attractive, but they have to be able to convey information to the reader or the viewer.  I know it sounds disparaging, but a donkey is a very useful beast of burden. He can carry a lot of information on his back and also he can take the reader or the viewer into a world that he's never been in before. And, for instance, John O'Neill, when I found that obituary, I didn't know if he was a hero or a goat, but I thought he's a hell of a donkey. He can take us into the world of counterterrorism and show us why it failed. And so he was the very first person I seized on. And then, you know, the other characters, the book itself uses O'Neill and Iman El-Zawahri, now the head of al-Qaeda, and Ben Laden and Prince Turkey Al-Faisal, who was the head of Saudi intelligence and worked with Ben Laden. And these interweaving biographies became the spine of the looming tower. So, as you were doing the research, what revelation most took you aback? Well, it involves Ali. You know, when I met Ali, I had a lot of questions about what did the intelligence community know about the plot. And Ali was a case agent for the USS Cole in Bombing in 2000. And while he was interrogating these figures in Yemen, he began to uncover information about a meeting in Malaysia and money that had flowed to Malaysia during a time when normally money comes to an operation. Why was it going out of the country? And, you know, there's a meeting in Kuala Lumpur and he suspected maybe this is where the operation was planned. And indeed, that was true. What he didn't know, couldn't have known then, is that at that meeting there were two future hijackers. And so he queried the CIA on three occasions, asking for information. Do you know anything about a meeting that was in Malaysia? And they said, no. You know, they spurned it. Not only did they know about it, they got the Malaysian authorities to surveil it. They got photographs of all the participants. And two of those people flew to LA in January of 2000, 19 months before 9-11. And the CIA found out about it in March of 2000. All of that time, they knew that Al Qaeda was in America. And they failed to tell the FBI, which was legally entitled to have that information, already had an indictment on bin Laden. Could legally follow them, clone their computers, whatever. You know, they could have disrupted the plot, but they were in the dark until it was too late. Ali, first of all, how does it feel to be a donkey? It's very hard. This dovetails nicely into something you said last night. Larry threw a party last night in and we had a chance to hear from Ali. And you said you were trepidatious about this story coming to the screen. But ultimately, you wanted to see it made because of accountability. Talk about that. What did you mean by that? One Larry and Alex talked to me about this. It was very awkward for me to be involved in a drama about 9-11. 9-11 is extremely important sacred event for me. I lost friends. And it's also more than that, it's also an indication of how we collectively failed the American people and this tragedy happened. So I prefer something like this to be a documentary rather than a drama. And I have to think about it. But also in the same time, I know that this story has been written, this story has been told. Everything that Larry said is part of not only his book but many documents that have been declassified over the years by the intelligence community and the folks who did this stated true to the story. So that was extremely important. That's number one. Number two, even though the story has been told, but it hasn't been shown to the American people and I think there's a big difference. And I felt that 16 years, 17 years after 9-11, we still don't have closure. And in order to do closure, we need to have understanding and we need to have clarity and we need to have characters like Michael and Jeff Daniels and people bring it to life so the American people can finally understand all the tragic events, the personalities, the institutional pickering that end up, we call it that tragic Tuesday of 9-11. Unfortunately, after all these years also, we never had an accountability on 9-11. The closest we get to an accountability is the CIA's own inspector general report where they pointed fingers at their own director and a few other people in the agency and required them to be held accountable for the 3,000 American death that took place on 9-11. The summary of this report has been declassified and has been published by the intelligence community and by the agency. So if we never had accountability because of that in a legal sense, we might have an accountability in a cultural sense, in a public sense. And I think that's what I hope this show is going to do because accountability is extremely important. You know, we thought if 3,000 Americans are murdered, we're going to have an accountability. But the people who are supposed to be held accountable gave the administration back then what they wanted. They gave them a false link about WMD and about war in Iraq and we end up going to Iraq. And if we never had an accountability because of the Iraq war, because of all the thousands of people who were killed from our side and actually, frankly, from the Iraqi side, the Iraq war messed up the whole Middle East. You just look at the map today. If we never had an accountability for that, we thought definitely we might have an accountability because of, you know, what torture? Because of the so-called in haste interrogation techniques. We are, after all, the United States of America. We fight for justice. We fight for human rights. We fight for the right thing. We are the shining city on the hill. But guess what? We never had an accountability for that either. So legally, we're not going to have an accountability. And I think a lot of countries are basically getting used to the fact that there is no accountability in America to include President Putin himself and look about the actions, their interference in our election, in our political system. So maybe if politicians don't want to do accountability, let us try to do an accountability. Let us show what happened to the American people. Let us change the world we live in. Let's change our mind about our system because if we don't change our mind, guess what? We're not changing anything about our future. Michael, congratulations. Not only on the Looming Tower, but on what has to be described as a breakout year. It's an astounding year for an actor. So congratulations on all that you have achieved this year. So clearly you picked the right projects and enhanced them. What led you to this one? Danny Futterman, our showrunner. I had known Danny in New York City as actors. We are contemporaries. We're the same age. And I got an email from him saying, there's a project I'm working on. We're thinking of you for playing this role. Mr. Richard Clark. And I had seen Mr. Clark interviewed on John Stewart's debut show. And he made a really, a really strong impression for me. He was a man who was coming out and telling the truth. And he was taking responsibility for our government's failure at a time when we all really needed to hear that. And I was kind of flabbergasted, because I didn't think of myself being right for the part. But I weighed it. And then I got another email from one of our writers, but Sheila Dorn, who had written for Lord Walk Empire, a show that I had been on before. And she also said, this is going to be something very special. Please consider this. She's throwing her two cents in as well. And then learning about the Looming Tower, the novel, learning about Larry, learning about Alex Gibney, learning about the cast that they were assembling, and the challenge that it was going to be. But Danny also said to me that, most likely you will get to say some of the things that Richard said in front of the 9-11 commission as well. And that kind of pushed me over the edge to think, wow, an opportunity to get to actually speak the things that our government, that people needed to hear. So I took a leap of faith. And I'm so, so glad that I did. We are too. Thank you. Clearly benefited the project. When you are choosing a role, any role, what goes through your mind? What are the factors that lead you to, you mentioned this role specifically, but generally speaking, what are the factors that go through your mind when you say, yeah, I've got to do this? You know, beggars can't be choosers, to be completely honest with you. And in a lot of ways, it's more along the lines of what comes towards me, what somebody else thinks I might be able to do. I had nothing to do with fighting for the role that I ended up playing and call me by your name, or in the shape of water, or in the post. It all just kind of fell in my lap. So I took these opportunities and I ran with them. Generally, if I have a choice in the matter, it's something that engages my mind, something that is obviously smart, more often with a sense of humor, because that's kind of how I see the world. And just something that sort of says, either you can do this, or you should do this, or I'm terrified to do this. All of those things kind of combined. Ideally, it would be a wonderful project to throw myself into. So I've just had a really, I've been really lucky this year. Larry, you work in many mediums. Unlike a lot of historians, you work easily on the stage. You have done many things relating to the screen, you've written books. What was the challenge in bringing this book to the small screen? Well, one thing is the small screen isn't as small as it used to be. There's a lot more expanse and the ability to tell a story in such a broad canvas wasn't present when I published the book. The first problem we faced was where to start. The book actually begins in Saeed Kuttev's journey to America in 1948. He was a man who went back to Egypt and wrote Malham Phil Tariq, this book called Milestones, that was the book that Ben Laden and everybody read. How often it always goes back to a book. Well, this was the mind count of the Islamist movement. And it was smuggled out of prison and it electrified a generation of young Muslims who were so alienated and disoriented in their own cultures. The story of the rise of Ben Laden and his boyhood and Zawahri, all of that is a prehistory and occupies about 300 pages that we don't deal with. One day I'd love to tell that story in this same form. I think it would be quite fascinating to talk about how these cultures diverged and came to collision in the way that they are now. So we decided that we should begin with the embassy bombings where you see this. And the reason was, for one thing, we knew that Ali was going to be a big part of it and this really introduces him to us. Also, Americans weren't aware of al-Qaeda until 9-11. And we wanted to show that we'd been under attack for three years before 9-11, beginning with the embassy bombings, which killed 224 people. Cold bombing where 17 sailors were killed. And that was in October of 2000. Right in the middle of the presidential campaign and not a single question about terrorism was raised in the presidential debates which shows you how distant that concern was from any American. It was happening somewhere else. And so when 9-11 happened on our shores, that's why it was so unsettling because where did this come from? Well, you know, a few people like Ali Soufan knew what was going on and were trying to protect us. And that's the story we decided we'd have to begin there and show the conflict of people inside these agencies with each other and also within their own agencies to be taken seriously because really in both the bureau and the agency, people were seen as kind of fanatics or, you know, off the deep end. The Alex station, you know, one of my CIA friends used to call it the island of lost toys. You know, they were discards from other departments that had been thrown in there. So nobody took it seriously except for the people that you see depicted in this and the war that they were fighting. Unfortunately, they were also fighting against each other. What did you make about Qaeda before 9-11? Well, you know, I was researching the siege. And so I had some unhappy experience. The movie came out in August of 1998. Well, it came out in November of 1998 but the trailers came out right after the embassy bombs. And there was another bombing that took place that same month that people don't hear very much about in Cape Town, South Africa. And it was of a planet Hollywood. Two people were killed, a little girl lost her leg and the radical Islamist group that claimed credit for it said they were protesting the siege which had, you know, only had heard rumors of the trailer in America which had not been, but two people were dead and the movie hadn't even come out yet. So I had been writing about, you know, an Islamist terror group and the movie when it came out in November was, there were protesters, there were pickets around the theaters. It was a box office failure. People were afraid to go to the movie and so I was very aware of the conflict internationally and also in my own country about radical Islam and where it was going. Ali, we see that harrowing scene of you arriving in Albania.  Awkward.  You know, there's dramatization. I won't be jumping off roof. You know, if a guy running like this, I'll be like, okay, I'll wait for him on the other side of the block, right? But that's usual. But yeah, it's, you know, as you've seen in the first episodes, the very first hard drive with the Mabrok, the Albania operation, the East Africa Embassy bombings, all these things were basically, you can Google them and you'll see that these events actually took place and happened and yes, it was put to the public in a dramatized version, but the story is true. Is it a temptation to look at it and say, no, it didn't quite happen that way or you're getting this wrong? How do you suspend that impulse? Well, it's very difficult.  Oh, they didn't do much drama in it. You know, people are anticipating that. But one of the things that I really loved about this story is that it's stated true to the facts and that was extremely important.  I mean, you have an episode where you have to take years and years of operations, intelligence collections, pickering, you know, human beings fighting with each other and using their own institutions and the culture of their own institution as a tool in order to win their own personal battles. You have a lot of these things happening, but in the same time, you have to convey it all to a person who's going to watch everything in about 50 minutes. And I think the show did a phenomenal job in bringing the emotions out that existed at the time, a lot of it. While you all were watching the film, I was giving a tour of the library to our guests and showed a letter that Jacqueline Kennedy had written in LBJ, which informed Brian Cranston's portrayal of LBJ, but Michael, you have portrayed two real-life characters in the last year, one of whom is alive, Richard Clark, and you spent some time with him. How did the time you spent with Clark inform your portrayal of him? Larry hooked us up together. We connected, and we went out to dinner together. He was kind enough to let me record our conversation, so I would have his voice to listen to in the future, but there's so much out there on the Internet of him giving speeches in a number of different situations and testimony as well, so it's readily accessible. He has become even more of a public figure since that time, so there's a lot of information out there for him. Generally, his generosity made me feel relaxed a little bit more than I was going into the situation. I was in the middle of another job at the time, so I hadn't had the time to do the research I needed to do to ask him intelligent questions.  So basically, it was an opportunity to sit across from him, to listen, to ask questions if he brought up a subject that I hadn't thought about, to hear about his inspirations as a young man to get into government, what 30 years in government was like working under four presidents. Why he does what he does? All of that was stuff that I wouldn't have found necessarily in his book or on the Internet. It primarily took away a great sense of humor of someone who has had their hands in our foreign policy and our world for the last 30 years, and I'm very grateful. I had one thing big or small that said, yeah, I got it. I understand him now. No. I mean, he's a lot to take in. To be completely honest with you, he's lived lives, you know, the things he's done are things I could only imagine. And it did spark my imagination though to try to put myself in his shoes when he's out there having a hand in basically who lives and dies. That's an experience I've never had and it's amazing to sit across the table from somebody who has had such influence. So it's been a great challenge and he was very generous. Allie, Larry, front and center in this story is the dysfunction between the FBI and the CIA, much to the detriment of the nation, if not the world. We now live in an era where our Commander-in-Chief has put those agencies under attack. He's called our former DNI James Clapper, former CIA Director John Brennan and former FBI Director Jim Comey, political hacks. What does that do to our standing abroad? When our intelligence apparatus is under siege in that manner, what is the threat to this nation? My feeling is the greater threat is to ourselves, not to our standing abroad. If there's a lesson that you might want to take in this series, it is divided we fail. You see the division, you know what happened. You can see that it might not have happened if we had been united, if we had been at one purpose. Now the intelligence community is reorganized and it's doing a far better job of cooperating with each other than on 9-11 and our relationship with foreign services is far much better than it was back then. But the division inside our country is profound. Maybe as great as it's ever been in modern history and the demonization of our intelligence community for partisan reasons is a way of sowing division which is aided and abetted by foreign agencies like the Russians and widening this chasm. Well, if there's a lesson that we should take from 9-11 in this series is that it's dangerous. It's dangerous to create that kind of division and when we do that we place ourselves at peril and so I fault the political people in our country who are attacking our intelligence agencies simply for partisan reasons because they're placing us in danger in doing that. Ali was 9-11, this might be a rhetorical question but was 9-11 a catalyst in seeing better cooperation between the FBI and the CIA? One of the things that I always say that this is not an issue and even what happened with the events that led to 9-11 an issue between the CIA and the FBI the people in the field were working closely together. There was something as you see happening on a headquarters on a headquarter level and people decided not to share information. When the CIA IG came out or the 9-11 commission came up they pointed fingers at specific people, not at an entire agency. I worked more with the CIA than probably with the FBI when I was in the field and these people are really ready at any second to lay their lives for the sake of this country. We never differentiate who's FBI and who's CIA when we're fighting the bad guys in Albania or in Nairobi or in Afghanistan or in Yemen, you call it. So we have to keep that in mind and I support what Larry said about all of us being attacked. After 9-11 I think the relationship is a lot better. I think there are a lot of things that have been implemented that kind of prevent individuals like Martin Schmidt for example to utilize institutional culture in order to create their own fiefdom inside any agency in the FBI or in the CIA. So we're a lot better, but also in the same time we see today all the heads of the intelligence agencies I don't know if you saw the hearing a few weeks ago sitting and each one of them saying yes, Russia interfered in our election in 2016. Yes, Russia is planning to interfere in the election of 2018. This is a DNI that's appointed by the current president a CIA director that's appointed by a current president the head of NSA, the head of FBI that's also appointed by the current president and each and every one of them when they said are you getting an instruction to basically retaliate against the Russians and each one of them said no. See we don't have deep state regardless of what the president tweets. What we have is intelligence agencies that gets their orders from the commander in chief and we are under attack today Al-Qaeda attacked us to divide us and they failed but the Russians attacking us to divide us and they are winning and we're not retaliating and we're not responding and we're not holding them accountable and they are not feeling the pain to stop and our intelligence agencies I believe can inflict a lot of pain on the Russians and others who are interfering or trying to interfere in our system but the commander in chief need to acknowledge that it's not a hoax and if they are not getting the orders they're not going to respond I know I'm pessimistic but I hope shows like this will make more people involved more people going back and figuring out what happened and how we can hold people accountable and I think we're in the LBJ library so maybe we should say we shall overcome I don't want this question to come off as though I'm asking if we have a bigger button but we clearly have a bigger military we spend more money on defense than other countries are our intelligence agencies discernably better than those of our enemies and our allies? I'll let you be a better... Absolutely man, we can smoke anyone we just need the orders I'll take a little bit of issue with it the capabilities that we have in one division in the CIA or the FBI is better bigger and more efficient than the capability a whole country has we're the United States of America we're the richest country in the world if you put all the militaries of the world together they are still smaller than our military we are the United States of America we can do it but we need the political leaders to give us instructions to do it and unfortunately in the last 15, 20 years we always get the wrong instructions you know instead of going after al-Qaeda let's go after Saddam you know instead of doing it the right way let's do Abu Ghraib and that's why we are in a debacle and we're going in a vicious cycle we will be inviting you back when we need a pep talk I am the wrong person for them you have a guy for a pep out there I'm just a donkey remember a donkey but not an ass I am independent by the way I'll take it as a Michael you are clearly a master film actor but it didn't come easy you started on stage and as I understand it it took you a while to adapt to screen acting talk about that process oh gosh yeah I couldn't get a TV job to save my life took me 13 years to get a small part on law and order in the meantime I was doing plays in New York that was my life blood it's still in my blood and yeah it was hard I guess I just had some growing up to do honestly and I didn't understand the craft and I had an opportunity to audition for the Coen brothers in 2008 for their film A Serious Man a small part in their film and it didn't cast me in that small part but they came back to me several months later and asked me to audition for the lead and for some reason they cast me in it and it changed my life and it gave me an opportunity to spend two months on a film set for the first time in my life because those two months that really just sort of gave me a kind of confidence they put confidence in me so I could just do the work in a way that I hadn't had the opportunity to do that before and really because they put their faith in me other people have put their faith in me and I've had opportunities now that I would never have had before so I just got lucky well deserved Larry, before we leave I have to ask, 9-Eleven was the most momentous event of this century what do you worry most about now? as you look around the world there are many we have diversity of threats maybe more threats than we can possibly put our mind to but if you just think about terrorism itself when 9-Eleven happened al-Qaeda was like 400 guys there are thousands now and we don't even think about al-Qaeda but they're in multiple countries there are many chapters of al-Qaeda there are progeny like ISIS and Boko Haram we're fighting against them but they're still proliferating and spreading and so many foreign fighters going back into Europe Russia it's a very dangerous world and the distraction just one tweet can capture the attention of the whole country and take our mind off of the fact that we're in a dangerous world and that we need to support the people that are trying to protect us and I worry oftentimes that we've forgotten who we are all the things you cited about torture and overgrave and we had two wars that we're still involved in and I know I'm dating myself when I think about this but I often reflect on what America used to be and the episode that comes to my mind is when I was in high school I took my girlfriend to the airport on a date in Dallas it's called Love Field but it was a place you could go when you didn't have any money to go to the movie and so we went to the airport and we walked out on the tarmac and then climbed up into a jetliner that we had decided had just come from Paris and we sat in the first class section and stewardesses as we called them then brought us a snack and then we walked up in the FAA tower high kids come on in and so we sat there and watched them landing the planes and that America's dead and terrorism killed it but if it's forgotten then we will have lost something precious and one of the reasons I wanted to do this series I felt so many young people 9-11 is not a part of their lived experience they don't know why we're in the fix we're in now they don't know about why we created the security state that we have why we're engaged in these prolonged wars why we have so much conflict they all sprang from the fear and the paranoia that gripped the country after 9-11 so our goal is to try to represent that progression but my secret goal is to try to remind people that you know who we are and where we came from and try to navigate back to that because if we don't find that America again then terrorists will have won the looming tower debuts on Hulu tomorrow night they will roll out the first three episodes and then the seven subsequent episodes will be rolled out week by week my congratulations to Larry Wright and to Alistair Barg and to Ali Soufan and please come back for Apostle Season 2 thank you very much


Welcome to the snap. This is the segment where we answer frequently asked questions in a very rapid fire format. So let's get right into it. On this week's episode of the snap we are going to be discussing getting to know your cap. So what does that mean exactly? Will most Dinovap caps click at the same temperature range? There may be slight differences from cap to cap. Think of your brand new cap like a brand new pair of denim jeans. Both serve the same function but they feel different due to your personal relationship with it. The primary differences are the volume of the click. Some caps may click louder than others. The number of clicks. There will always be two clicks but they may occur simultaneously so it may sound like there was one big click. The range of clicks. Where does the click occur within the temperature range? The best way to learn the uniqueness of your new cap is to be cognizant of the potential differences and create benchmarks. Whenever I get a new cap I perform the following tasks to develop a better understanding of my cap and create a more intimate relationship with it. Put your cap on an empty device and heat with your preferred torch and technique. Here I am using a triple torch with an infinite spin. I go more in depth on the infinite spin techniques and the video found in the corner. Pay attention to how long it took for you to hear the click. How many clicks did you hear?  If you heat two seconds past the first click stop. More than likely that second click already occurred. Repeat these steps as many times as it takes for you to be accustomed to the unique qualities of your cap. After you've gotten familiar with your empty device now you can move on to the live trials. Load it up with your preferred material and heat like you did before and then check the doneness of your material. Is the material darker or lighter compared to your other cap? If it's lighter your cap is clicking at a lower temperature. If it's a darker color it's going to be clicking at a higher temperature. Based on the results you can now dial in your performance by adjusting your heating technique and you can find more about adjusting your heating technique here. Being aware of the unique qualities of your cap will improve the desired experience and most importantly prevent combustion. I hope you found these steps helpful and creating a better understanding of your cap. Let us know in the comments below if you like this snap and what you would like discussed on future episodes. And Matt has been the snap. Thanks for watching and be sure to check out our other videos here and here and make sure to subscribe and ring the bell to be notified for when we release more great content.