0
SPEAKER_01
The following is a conversation with Chris Latner.
0
SPEAKER_01
Currently, he s a senior director at Google, working on several projects, including CPU, GPU, TPU accelerators for TensorFlow, Swift for TensorFlow, and all kinds of machine learning compiler magic going on behind the scenes.
0
SPEAKER_01
He s one of the top experts in the world on compiler technologies, which means he deeply understands the intricacies of how hardware and software come together to create efficient code.
0
SPEAKER_01
He created the LLVM compiler infrastructure project and the Clang compiler.
0
SPEAKER_01
He led major engineering efforts at Apple, including the creation of the Swift programming language.
0
SPEAKER_01
He also briefly spent time at Tesla as Vice President of Autopilot Software during the transition from Autopilot Hardware 1 to Hardware 2, when Tesla essentially started from scratch to build an in-house software infrastructure for Autopilot.
0
SPEAKER_01
I could have easily talked to Chris for many more hours.
0
SPEAKER_01
Compiling code down across the levels of abstraction is one of the most fundamental and fascinating aspects of what computers do, and he is one of the world experts in this process.
0
SPEAKER_01
It s rigorous science, and it s messy, beautiful art.
0
SPEAKER_01
This conversation is part of the Artificial Intelligence Podcast.
0
SPEAKER_01
If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.
0
SPEAKER_01
And now, here s my conversation with Chris,
0
SPEAKER_01
Ladner.
0
SPEAKER_01
What was the first program you ve ever written?
0
SPEAKER_00
My first program back.
0
SPEAKER_00
And when was it?
0
SPEAKER_00
I think I started as a kid and my parents got a basic programming book.
0
SPEAKER_00
And so when I started, it was typing out programs from a book and seeing how they worked and then typing them in wrong and trying to figure out why they were not working right.
0
SPEAKER_00
That kind of stuff.
0
SPEAKER_01
So basic, what was the first language that you remember yourself maybe falling in love with, like really connecting with?
0
SPEAKER_00
I don t know, I mean, I feel like I ve learned a lot along the way and each of them have a different special thing about them.
0
SPEAKER_00
So I started in BASIC and then went like GWBASIC, which was the thing back in the DOS days, and then upgraded to QBASIC and eventually QuickBASIC, which are all slightly more fancy versions of Microsoft BASIC.
0
SPEAKER_00
Made the jump to PASCAL and started doing machine language programming and assembly in PASCAL, which was really cool.
0
SPEAKER_00
Turbo PASCAL was amazing for its day.
0
SPEAKER_00
eventually got into C, C++, and then kind of did lots of other weird things.
0
SPEAKER_01
I feel like you took the dark path, which is the... you could have gone lisp.
0
SPEAKER_00
Yeah.
0
SPEAKER_01
You could have gone a higher level sort of functional, philosophical, hippie route.
0
SPEAKER_00
Yeah.
0
SPEAKER_01
Instead you went into like the dark arts of the
0
SPEAKER_00
It was straight into the machine.
0
SPEAKER_00
So it started with basic Pascal and then assembly, and then wrote a lot of assembly.
0
SPEAKER_00
And I eventually did small talk and other things like that, but that was not the starting point.
0
SPEAKER_01
But so what is this journey into C?
0
SPEAKER_01
Is that in high school?
0
SPEAKER_01
Is that in college?
0
SPEAKER_00
That was in high school, yeah.
0
SPEAKER_00
So and then that was really about trying to be able to do more powerful things than what Pascal could do, and also to learn a different world.
0
SPEAKER_00
So he was really confusing me with pointers and the syntax and everything, and it took a while.
0
SPEAKER_00
But Pascal s much more principled in various ways, sees more
0
SPEAKER_00
I mean, it has its historical roots, but it s not as easy to learn.
0
SPEAKER_01
With pointers, there s this memory management thing that you have to become conscious of.
0
SPEAKER_01
Is that the first time you start to understand that there s resources that you re supposed to manage?
0
SPEAKER_00
Well, so you have that in Pascal as well, but in Pascal, they use the caret instead of the star, and there s some small differences like that, but it s not about pointer arithmetic.
0
SPEAKER_00
In C, you end up thinking about how things get laid out in memory a lot more.
0
SPEAKER_00
And so in Pascal, you have allocating and deallocating and owning the memory, but just the programs are simpler and you don t have to, well, for example, Pascal has a string type.
0
SPEAKER_00
And so you can think about a string instead of an array of characters, which are consecutive in memory.
0
SPEAKER_00
So it s a little bit of a higher level abstraction.
0
SPEAKER_01
So let s get into it.
0
SPEAKER_01
Let s talk about LLVM, CLang, and compilers.
0
SPEAKER_01
So can you tell me first what LLVM and CLang are, and how is it that you find yourself the creator and lead developer of one of the most powerful compiler optimization systems in use today?
0
SPEAKER_00
Sure, so I guess they re different things.
0
SPEAKER_00
Let s start with what is a compiler?
0
SPEAKER_00
Is that a good place to start?
0
SPEAKER_00
What are the phases of a compiler?
0
SPEAKER_00
What are the parts?
0
SPEAKER_00
Yeah, what is it?
0
SPEAKER_00
So what is even a compiler used for?
0
SPEAKER_00
So the way I look at this is you have a two-sided problem of you have humans that need to write code, and then you have machines that need to run the program that the human wrote.
0
SPEAKER_00
And
0
SPEAKER_00
for lots of reasons, the humans don t want to be writing in binary and want to think about every piece of hardware.
0
SPEAKER_00
And so at the same time that you have lots of humans, you also have lots of kinds of hardware.
0
SPEAKER_00
And so compilers are the art of allowing humans to think at a level of abstraction that they want to think about.
0
SPEAKER_00
and then get that program, get the thing that they wrote, to run on a specific piece of hardware.
0
SPEAKER_00
And the interesting and exciting part of all this is that there s now lots of different kinds of hardware, chips like x86 and PowerPC and ARM and things like that, but also high-performance accelerators for machine learning and other things like that, or also just different kinds of hardware, GPUs.
0
SPEAKER_00
These are new kinds of hardware.
0
SPEAKER_00
And at the same time, on the programming side of it, you have your basic UFC, you have JavaScript, you have Python, you have Swift, you have like lots of other languages that are all trying to talk to the human in a different way to make them more expressive and capable and powerful.
0
SPEAKER_00
And so.
0
SPEAKER_00
Compilers are the thing that goes from one to the other, no?
0
SPEAKER_00
End to end, from the very beginning to the very end.
0
SPEAKER_00
End to end, and so you go from what the human wrote, and programming languages end up being about expressing intent, not just for the compiler and the hardware, but the programming language s job is really to capture an expression of what the programmer wanted that then can be maintained and adapted and evolved by other humans, as well as interpreted by the compiler.
0
SPEAKER_00
So when you look at this problem, you have, on the one hand, humans, which are complicated, and you have hardware, which is complicated.
0
SPEAKER_00
And so compilers typically work in multiple phases.
0
SPEAKER_00
And so the software engineering challenge that you have here is try to get maximum reuse out of the amount of code that you write, because these compilers are very complicated.
0
SPEAKER_00
And so the way it typically works out is that you have something called a front end or a parser that is language specific.
0
SPEAKER_00
And so you ll have a C parser, and that s what Clang is, or C++ or JavaScript or Python or whatever.
0
SPEAKER_00
That s the front end.
0
SPEAKER_00
Then you ll have a middle part, which is often the optimizer.
0
SPEAKER_00
And then you ll have a late part, which is hardware specific.
0
SPEAKER_00
And so compilers end up, there s many different layers often, but these three big groups are very common in compilers.
0
SPEAKER_00
And what LLVM is trying to do is trying to standardize that middle and last part.
0
SPEAKER_00
And so one of the cool things about LLVM is that there are a lot of different languages that compile through to it.
0
SPEAKER_00
And so things like Swift, but also Julia, Rust.
0
SPEAKER_00
Clang for C, C++, Subjective-C.
0
SPEAKER_00
These are all very different languages, and they can all use the same optimization infrastructure, which gets better performance, and the same code generation infrastructure for hardware support.
0
SPEAKER_00
And so LLVM is really that layer that is common that all these different specific compilers can use.
0
SPEAKER_01
And is it a standard, like a specification, or is it literally an implementation?
0
SPEAKER_00
It s an implementation.
0
SPEAKER_00
And so it s, I think there s a couple of different ways of looking at it, right?
0
SPEAKER_00
Because it depends on which angle you re looking at it from.
0
SPEAKER_00
LLVM ends up being a bunch of code, okay?
0
SPEAKER_00
So it s a bunch of code that people reuse and they build compilers with.
0
SPEAKER_00
We call it a compiler infrastructure because it s kind of the underlying platform that you build a concrete compiler on top of.
0
SPEAKER_00
But it s also a community, and the LLVM community is hundreds of people that all collaborate, and one of the most fascinating things about LLVM over the course of time is that we ve managed somehow to successfully get harsh competitors in the commercial space to collaborate on shared infrastructure.
0
SPEAKER_00
And so you have Google and Apple, you have AMD and Intel, you have Nvidia and AMD on the graphics side, you have
0
SPEAKER_00
Cray and everybody else doing these things and like all these companies are collaborating together to make that shared infrastructure really really great and They do this not of the goodness of their heart but they do it because it s in their commercial interest of having really great infrastructure that they can build on top of and
0
SPEAKER_00
and facing the reality that it s so expensive that no one company, even the big companies, no one company really wants to implement it all themselves.
0
SPEAKER_00
Expensive or difficult?
0
SPEAKER_00
Both.
0
SPEAKER_00
That s a great point because it s also about the skill sets.
0
SPEAKER_00
And the skill sets are very hard to find.
0
SPEAKER_01
How big is the LLVM?
0
SPEAKER_01
It always seems like with open source projects, the kind, and LLVM is open source?
0
SPEAKER_00
Yes, it s open source.
0
SPEAKER_00
It s about, it s 19 years old now, so it s fairly old.
0
SPEAKER_01
It seems like the magic often happens in a very small circle of people.
0
SPEAKER_01
Yes.
0
SPEAKER_00
At least at early birth and whatever.
0
SPEAKER_00
Yes.
0
SPEAKER_00
So LLVM came from a university project, and so I was at the University of Illinois, and there it was myself, my advisor, and then a team of two or three research students in the research group, and we built many of the core pieces initially.
0
SPEAKER_00
I then graduated and went to Apple, and at Apple brought it to the products, first in the OpenGL graphics stack, but eventually to the C compiler realm, and eventually built Clang, and eventually built Swift and these things.
0
SPEAKER_00
along the way building a team of people that are really amazing compiler engineers that helped build a lot of that.
0
SPEAKER_00
And so as it was gaining momentum and as Apple was using it, being open source and public and encouraging contribution, many others, for example, at Google, came in and started contributing.
0
SPEAKER_00
And in some cases, Google effectively owns Clang now because it cares so much about C++ and the evolution of that ecosystem.
0
SPEAKER_00
And so it s investing a lot in the C++ world and the tooling and things like that.
0
SPEAKER_00
And so likewise, NVIDIA cares a lot about CUDA.
0
SPEAKER_00
And so CUDA uses Clang and uses LLVM for graphics and GPGPU.
0
SPEAKER_01
And so when you first started as a master s project, I guess, did you think it was gonna go as far as it went?
0
SPEAKER_01
Were you crazy ambitious about it?
0
SPEAKER_01
It seems like a really difficult undertaking, a brave one.
0
SPEAKER_00
Yeah, no, it was nothing like that.
0
SPEAKER_00
So, I mean, my, my goal when I went to the University of Illinois was to get in and out with a non-thesis masters in a year and get back to work.
0
SPEAKER_00
So I was not, I was not planning to stay for five years and, and build this massive infrastructure.
0
SPEAKER_00
I got, um, nerd sniped into staying.
0
SPEAKER_00
And a lot of it was because LLVM was fun and I was building cool stuff and learning really interesting things.
0
SPEAKER_00
And, uh,
0
SPEAKER_00
facing both software engineering challenges, but also learning how to work in a team and things like that.
0
SPEAKER_00
I had worked at many companies as interns before that, but it was really a different thing to have a team of people that were working together and trying to collaborate in version control, and it was just a little bit different.
0
SPEAKER_01
Like I said, I just talked to Don Knuth, and he believes that 2% of the world population have something weird with their brain, that they re geeks, they understand computers, they re connected with computers.
0
SPEAKER_01
He put it at exactly 2%.
0
SPEAKER_01
He s a specific guy.
0
SPEAKER_01
Very specific.
0
SPEAKER_01
He says, I can t prove it, but it s very empirically there.
0
SPEAKER_01
Is there something that attracts you to the idea of optimizing code?
0
SPEAKER_01
I mean, it seems like that s one of the biggest, coolest things about LLVM.
0
SPEAKER_00
Yeah, that s one of the major things it does.
0
SPEAKER_00
Um, I got into that because of a person actually.
0
SPEAKER_00
So when I was in my undergraduate, I had an advisor or a professor named Steve Vegdal and he, I went to this little tiny private school.
0
SPEAKER_00
We, there were like seven or nine people in my computer science department or students in my, in my class.
0
SPEAKER_00
So it was a very tiny, uh, very, very small school.
0
SPEAKER_00
It was a kind of a work on the side of the math department kind of a thing.
0
SPEAKER_00
at the time.
0
SPEAKER_00
I think it s evolved a lot in the many years since then.
0
SPEAKER_00
But Steve Vegdal was a compiler guy, and he was super passionate.
0
SPEAKER_00
And his passion rubbed off on me.
0
SPEAKER_00
And one of the things I like about compilers is that they re large, complicated software
0
SPEAKER_00
pieces.
0
SPEAKER_00
And so one of the culminating classes that many computer science departments, at least at the time, did was to say that you take algorithms and data structures and all these core classes, but then the compilers class was one of the last classes you take because it
0
SPEAKER_00
pulls everything together.
0
SPEAKER_00
And then you work on one piece of code over the entire semester.
0
SPEAKER_00
And, um, and so you keep building on your own work, which is really interesting.
0
SPEAKER_00
And it s also very challenging because in many classes, if you don t get a project done, you just forget about it and move on to the next one and get your, you know, your B or whatever it is.
0
SPEAKER_00
But here you have to live with the decisions you make and continue to reinvest in it.
0
SPEAKER_00
And I really liked that.
0
SPEAKER_00
And so I did an extra study project with him the following semester, and he was just really great.
0
SPEAKER_00
And he was also a great mentor in a lot of ways.
0
SPEAKER_00
And so from him and from his advice, he encouraged me to go to graduate school.
0
SPEAKER_00
I wasn t super excited about going to grad school.
0
SPEAKER_00
I wanted the master s degree, but I didn t want to be an academic.
0
SPEAKER_00
But like I said, I kinda got tricked into saying I was having a lot of fun, and I definitely do not regret it.
0
SPEAKER_01
What aspects of compilers were the things you connected with?
0
SPEAKER_01
So, LLVM, there s also the other part that s just really interesting, if you re interested in languages, is parsing and just analyzing.
0
SPEAKER_01
Yeah, analyzing the language, breaking it down, parsing, and so on.
0
SPEAKER_01
Was that interesting to you, or were you more interested in optimization?
0
SPEAKER_00
For me, it was more so.
0
SPEAKER_00
I m not really a math person.
0
SPEAKER_00
I can do math.
0
SPEAKER_00
I understand some bits of it when I get into it.
0
SPEAKER_00
But math is never the thing that attracted me.
0
SPEAKER_00
And so a lot of the parser part of the compiler has a lot of good formal theories that Don, for example, knows quite well.
0
SPEAKER_00
Still waiting for his book on that.
0
SPEAKER_00
But I just like building a thing and seeing what it could do and exploring and getting it to do more things and then setting new goals and reaching for them.
0
SPEAKER_00
And in the case of LLVM, when I started working on that, my research advisor that I was working for was a compiler guy.
0
SPEAKER_00
And so he and I specifically found each other because we were both interested in compilers and so I started working with him.
0
SPEAKER_00
and taking his class, and a lot of LLVM initially was, it s fun implementing all the standard algorithms and all the things that people had been talking about and were well known and they were in the curricula for advanced studies and compilers.
0
SPEAKER_00
And so just being able to build that was really fun.
0
SPEAKER_00
And I was learning a lot by, instead of reading about it, just building.
0
SPEAKER_00
And so I enjoyed that.
0
SPEAKER_01
So you said compilers are these complicated systems.
0
SPEAKER_01
Can you even just, with language, try to describe how you turn a C++ program into code?
0
SPEAKER_00
What are the hard parts?
0
SPEAKER_01
Why is it so hard?
0
SPEAKER_00
So I ll give you examples of the hard parts along the way.
0
SPEAKER_00
So C++ is a very complicated programming language.
0
SPEAKER_00
It s something like 1,400 pages in the spec.
0
SPEAKER_00
So C++ by itself is crazy complicated.
0
SPEAKER_00
Can we just, sorry, pause.
0
SPEAKER_01
What makes the language complicated in terms of what s syntactically?
0
SPEAKER_00
So it s what they call syntax.
0
SPEAKER_00
So the actual how the characters are arranged, yes.
0
SPEAKER_00
It s also semantics, how it behaves.
0
SPEAKER_00
It s also, in the case of C++, there s a huge amount of history.
0
SPEAKER_00
C++ is built on top of C. You play that forward, and then a bunch of suboptimal, in some cases, decisions were made, and they compound.
0
SPEAKER_00
And then more and more and more things keep getting added to C++, and it will probably never stop.
0
SPEAKER_00
But the language is very complicated from that perspective.
0
SPEAKER_00
And so the interactions between subsystems is very complicated.
0
SPEAKER_00
There s just a lot there.
0
SPEAKER_00
And when you talk about the front end, one of the major challenges, which Clang as a project, the C, C++ compiler that I built, I and many people built,
0
SPEAKER_00
One of the challenges we took on was we looked at GCC.
0
SPEAKER_00
GCC at the time was like a really good industry standardized compiler that had really consolidated a lot of the other compilers in the world and was a standard.
0
SPEAKER_00
But it wasn t really great for research.
0
SPEAKER_00
The design was very difficult to work with.
0
SPEAKER_00
And it was full of global variables and other things that made it very difficult to reuse in ways that it wasn t originally designed for.
0
SPEAKER_00
And so with Clang, one of the things that we want to do is push forward on better user interface.
0
SPEAKER_00
So make error messages that are just better than GCCs.
0
SPEAKER_00
And that s actually hard because you have to do a lot of bookkeeping in an efficient way to be able to do that.
0
SPEAKER_00
We want to make compile time better.
0
SPEAKER_00
And so compile time is about making it efficient, which is also really hard when you re keeping track of extra information.
0
SPEAKER_00
we wanted to make new tools available.
0
SPEAKER_00
So refactoring tools and other analysis tools that the GCC never supported, also leveraging the extra information we kept, but enabling those new classes of tools that then get built into IDEs.
0
SPEAKER_00
And so that s been one of the areas that Clang has really helped push the world forward in, is in the tooling for C and C++ and things like that.
0
SPEAKER_00
But C++ and the front end piece is complicated and you have to build syntax trees and you have to check every rule in the spec and you have to turn that back into an error message to the human that the human can understand when they do something wrong.
0
SPEAKER_00
But then you start doing the, uh, what s called lowering.
0
SPEAKER_00
So going from C++ and the way that it represents code down to the machine.
0
SPEAKER_00
And when you do that, there s many different phases you go through.
0
SPEAKER_00
Often there are, I think LLVM has something like 150 different what are called passes in the compiler that the code
1
SPEAKER_01
passes through, and these get organized in very complicated ways, which affect the generated code, and the performance, and compile time, and many other things.
1
SPEAKER_01
What are they passing through?
1
SPEAKER_00
So after you do the Clang parsing, what s the graph?
1
SPEAKER_00
What does it look like?
1
SPEAKER_00
What s the data structure here?
1
SPEAKER_01
Yeah, so in the parser, it s usually a tree, and it s called an abstract syntax tree.
1
SPEAKER_01
And so the idea is you have a node for the plus that the human wrote in their code, or the function call, you ll have a node for call with the function that they call and the arguments they pass, things like that.
1
SPEAKER_01
this then gets lowered into what s called an intermediate representation and intermediate representations are like LLVM has one and there it s a it s what s called a control flow graph and so you represent each operation in the program as a very simple
1
SPEAKER_01
like this is gonna add two numbers, this is gonna multiply two things, maybe we ll do a call, but then they get put in what are called blocks.
1
SPEAKER_01
And so you get blocks of these straight-line operations, where instead of being nested like in a tree, it s straight-line operations, and so there s a sequence and an ordering to these operations.
1
SPEAKER_00
And then you have- So within the block or outside the block?
1
SPEAKER_01
That s within the block, and so it s a straight-line sequence of operations within the block, and then you have branches, like conditional branches between blocks.
1
SPEAKER_01
And so when you write a loop, for example,
1
SPEAKER_01
In a syntax tree, you would have a for node, like for a for statement in a C-like language, you d have a for node, and you have a pointer to the expression for the initializer, a pointer to the expression for the increment, a pointer to the expression for the comparison, a pointer to the body.
1
SPEAKER_01
And these are all nested underneath it.
1
SPEAKER_01
In a control flow graph, you get a block for the code that runs before the loop, so the initializer code.
1
SPEAKER_01
Then you have a block for the body of the loop.
1
SPEAKER_01
And so the body of the loop code goes in there, but also the increment and other things like that.
1
SPEAKER_01
And then you have a branch that goes back to the top, and a comparison and a branch that goes out.
1
SPEAKER_01
And so it s more of a assembly level kind of representation.
1
SPEAKER_01
But the nice thing about this level of representation is it s much more language independent.
1
SPEAKER_01
And so there s lots of different kinds of languages with different kinds of, you know, JavaScript has a lot of different ideas of what is false, for example, and all that can stay in the front end, but then that middle part can be shared across all of those.
1
SPEAKER_00
How close is that intermediate representation to neural networks, for example?
1
SPEAKER_00
Are they, because everything you describe is a kind of echoes of a neural network graph.
1
SPEAKER_00
Are they neighbors or what?
1
SPEAKER_01
They re quite different in details, but they re very similar in idea.
1
SPEAKER_01
So one of the things that neural networks do is they learn representations for data at different levels of abstraction, right?
1
SPEAKER_01
And then they transform those through layers, right?
1
SPEAKER_01
So the compiler does very similar things, but one of the things the compiler does is it has relatively few different representations, where a neural network often, as you get deeper, for example, you get many different representations, and each layer or set of ops is transforming between these different representations.
1
SPEAKER_01
In a compiler, often you get one representation, and they do many transformations to it.
1
SPEAKER_01
And these transformations are often applied iteratively.
1
SPEAKER_01
And for programmers, there s familiar types of things.
1
SPEAKER_01
For example, trying to find expressions inside of a loop and pulling them out of a loop so that they execute for your times, or find redundant computation, or find
1
SPEAKER_01
Constant folding or other simplifications turning, you know, two times X into X shift left by one and things like this are all the examples of the things that happen.
1
SPEAKER_01
But compilers end up getting a lot of theorem proving and other kinds of algorithms that try to find higher level properties of the program that then can be used by the optimizer.
1
SPEAKER_00
Cool, so what s like the biggest bang for the buck with optimization?
1
SPEAKER_00
What s a... Today?
1
SPEAKER_00
Yeah.
1
SPEAKER_00
Well, no, not even today.
1
SPEAKER_00
At the very beginning, the 80s, I don t know.
1
SPEAKER_01
Yeah, so for the 80s, a lot of it was things like register allocation.
1
SPEAKER_01
So the idea of, in a modern, like a microprocessor, what you ll end up having is you ll end up having memory, which is relatively slow, and then you have registers that are relatively fast.
1
SPEAKER_01
But registers, you don t have very many of them.
1
SPEAKER_01
Okay, and so when you re writing a bunch of code, you re just saying, like, compute this, put in a temporary variable, compute this, compute this, compute this, put in a temporary variable, I have a loop, I have some other stuff going on.
1
SPEAKER_01
Well, now you re running on an x86, like a desktop PC or something.
1
SPEAKER_01
Well, it only has, in some cases, some modes, eight registers, right?
1
SPEAKER_01
And so now the compiler has to choose what values get put in what registers at what points in the program.
1
SPEAKER_01
And this is actually a really big deal.
1
SPEAKER_01
So if you think about, you have a loop, an inner loop that executes millions of times maybe.
1
SPEAKER_01
If you re doing loads and stores inside that loop, then it s gonna be really slow.
1
SPEAKER_01
But if you can somehow fit all the values inside that loop and registers, now it s really fast.
1
SPEAKER_01
And so getting that right requires a lot of work because there s many different ways to do that.
1
SPEAKER_01
And often what the compiler ends up doing is it ends up thinking about things in a different representation than what the human wrote.
1
SPEAKER_01
You wrote into X. Well, the compiler thinks about that as four different values, each which have different lifetimes across the function that it s in, and each of those could be put in a register or memory or different memory or maybe in some parts of the code recomputed instead of stored and reloaded, and there are many of these different kinds of techniques that can be used.
1
SPEAKER_00
So it s adding almost like a time dimension to, it s trying to optimize across time, so it s considering when you re programming, you re not thinking in that way.
1
SPEAKER_01
Yeah, absolutely.
1
SPEAKER_01
And so the RISC era made things, so RISC chips, R-I-S-C, the RISC chips as opposed to CISC chips, the RISC chips made things more complicated for the compiler because
1
SPEAKER_01
What they ended up doing is ending up adding pipelines to the processor where the processor can do more than one thing at a time.
1
SPEAKER_01
But this means that the order of operations matters a lot.
1
SPEAKER_01
And so one of the classical compiler techniques that you use is called scheduling.
1
SPEAKER_01
And so moving the instructions around so that the processor can keep its pipelines full instead of stalling and getting blocked.
1
SPEAKER_01
And so there s a lot of things like that that are
1
SPEAKER_01
kind of bread and butter compiler techniques have been studied a lot over the course of decades now, but the engineering side of making them real is also still quite hard.
1
SPEAKER_01
And you talk about machine learning, this is a huge opportunity for machine learning because many of these algorithms are full of these like hokey hand-rolled heuristics, which work well on specific benchmarks, but don t generalize and full of magic numbers.
1
SPEAKER_01
And, you know, I hear there s some techniques that are good at handling that.
1
SPEAKER_00
So what would be the, if you were to apply machine learning to this, what s the thing you re trying to optimize?
1
SPEAKER_00
Is it ultimately the running time?
1
SPEAKER_01
You can pick your metric and there s running time, there s memory use, there s
1
SPEAKER_01
There s lots of different things that you can optimize for.
1
SPEAKER_01
Code size is another one that some people care about in the embedded space.
1
SPEAKER_00
Is this like thinking into the future, or has somebody actually been crazy enough to try to have machine learning-based parameter tuning for optimization of compilers?
1
SPEAKER_01
So this is something that is, I would say, research right now.
1
SPEAKER_01
There are a lot of research systems that have been applying search in various forms, and using reinforcement learning is one form, but also brute force search has been tried for quite a while, and usually these are in small problem spaces, so find the optimal way to co-generate a matrix multiply for a GPU, right?
1
SPEAKER_01
Something like that, where you say, there s a lot of,
1
SPEAKER_01
design space of do you unroll loops a lot?
1
SPEAKER_01
Do you execute multiple things in parallel?
1
SPEAKER_01
And there s many different confounding factors here because graphics cards have different numbers of threads and registers and execution ports and memory bandwidth and many different constraints that interact in nonlinear ways.
1
SPEAKER_01
And so search is very powerful for that and it gets used in certain ways, but it s not very structured.
1
SPEAKER_01
This is something that we as an industry need to fix.
1
SPEAKER_00
You said 80s, but like, so have there been like big jumps in improvement and optimization?
1
SPEAKER_00
Yeah.
1
SPEAKER_00
Yeah.
1
SPEAKER_00
Since then, what s the coolest things?
1
SPEAKER_01
It s largely been driven by hardware.
1
SPEAKER_01
So, hardware and software.
1
SPEAKER_01
So in the mid 90s, Java totally changed the world, right?
1
SPEAKER_01
And I m still amazed by how much change was introduced by Java.
1
SPEAKER_01
In a good way, or?
1
SPEAKER_01
In a good way.
1
SPEAKER_01
So like reflecting back, Java introduced things like, all at once, introduced things like JIT compilation.
1
SPEAKER_01
None of these were novel, but it pulled it together and made it mainstream and made people invest in it.
1
SPEAKER_01
JIT compilation, garbage collection, portable code, safe code, like memory safe code, like a very dynamic dispatch execution model.
1
SPEAKER_01
Like many of these things, which had been done in research systems and had been done in small ways in various places,
1
SPEAKER_01
really came to the forefront and really changed how things worked and therefore changed the way people thought about the problem.
1
SPEAKER_01
Um, JavaScript was another major world change based on the way it works.
1
SPEAKER_01
Um, but also on the hardware side of things, um, uh, multi-core and vector instructions really change the problem space and are very, um,
1
SPEAKER_01
they don t remove any of the problems that compilers faced in the past, but they, they add new kinds of problems of how do you find enough work to keep a four wide vector busy, right?
1
SPEAKER_01
Or if you re doing a matrix multiplication, how do you do different columns out of that matrix in at the same time?
1
SPEAKER_01
And how do you maximally utilize the, the arithmetic compute that one core has?
1
SPEAKER_01
And then how do you take it to multiple cores?
1
SPEAKER_00
How did the whole virtual machine thing change the compilation pipeline?
1
SPEAKER_01
Yeah, so what the Java virtual machine does is it splits, just like I was talking about before, where you have a front end that parses the code, and then you have an intermediate representation that gets transformed.
1
SPEAKER_01
What Java did was they said, we will parse the code and then compile to what s known as Java bytecode.
1
SPEAKER_01
And that bytecode is now a portable code representation that is industry standard and locked down and can t change.
1
SPEAKER_01
And then the back part of the compiler that does optimization and code generation can now be built by different vendors.
1
SPEAKER_01
And Java bytecode can be shipped around across the wire.
1
SPEAKER_01
It s memory safe and relatively trusted.
1
SPEAKER_01
And because of that, it can run in the browser.
1
SPEAKER_01
And that s why it runs in the browser.
1
SPEAKER_01
And so that way, you can be in, again, back in the day, you would write a Java applet.
1
SPEAKER_01
And as a web developer, you d build this mini app that would run on a web page.
1
SPEAKER_01
A user of that is running a web browser on their computer.
1
SPEAKER_01
You download that Java bytecode, which can be trusted, and then you do all the compiler stuff on your machine so that you know that you trust that.
1
SPEAKER_00
Is that a good idea or a bad idea?
1
SPEAKER_01
It s a great idea.
1
SPEAKER_01
I mean, it s a great idea for certain problems.
1
SPEAKER_01
And I m very much a believer that technology is itself neither good nor bad.
1
SPEAKER_01
It s how you apply it.
1
SPEAKER_01
you know, this would be a very, very bad thing for very low levels of the software stack.
1
SPEAKER_01
But, but in terms of solving some of these software portability and transparency or portability problems, I think it s been really good.
1
SPEAKER_01
Now Java ultimately didn t win out on the desktop and like there are good reasons for that.
1
SPEAKER_01
But, um, it s been very successful on servers and in many places it s been a very successful thing over, over decades.
1
SPEAKER_00
So what has been LLVM s and C-Lang s improvements in optimization throughout its history?
1
SPEAKER_00
What are some moments where you had to set back and really proud of what s been accomplished?
1
SPEAKER_01
Yeah, I think that the interesting thing about LLVM is not the innovations in compiler research.
1
SPEAKER_01
It has very good implementations of various important algorithms, no doubt.
1
SPEAKER_01
And a lot of really smart people have worked on it.
1
SPEAKER_01
But I think that the thing that s most profound about LLVM is that through standardization, it made things possible that otherwise wouldn t have happened.
1
SPEAKER_01
Okay, and so interesting things that have happened with LLVM, for example, Sony has picked up LLVM and used it to do all the graphics compilation in their movie production pipeline.
1
SPEAKER_01
And so now they re able to have better special effects because of LLVM.
1
SPEAKER_01
That s kind of cool.
1
SPEAKER_01
That s not what it was designed for, right?
1
SPEAKER_01
But that s the sign of good infrastructure when it can be used in ways it was never designed for because it has good layering and software engineering and it s composable and things like that.
1
SPEAKER_01
Which is where, as you said, it differs from GCC.
1
SPEAKER_01
Yes.
1
SPEAKER_01
GCC is also great in various ways, but it s not as good as infrastructure technology.
1
SPEAKER_01
It s really a C compiler, or it s a Fortran compiler.
1
SPEAKER_01
It s not infrastructure in the same way.
1
SPEAKER_00
Now you can tell I don t know what I m talking about because I keep saying Clang.
1
SPEAKER_00
You can always tell when a person is close by the way they pronounce something.
1
SPEAKER_00
I don t think, have I ever used Clang?
1
SPEAKER_00
Entirely possible.
1
SPEAKER_01
So you ve used code, it s generated probably.
1
SPEAKER_01
So Clang and LLVM are used to compile all the apps on the iPhone, effectively, and the OSes.
1
SPEAKER_01
It compiles Google s production server applications.
1
SPEAKER_01
It s used to build GameCube games and PlayStation 4 and things like that.
1
SPEAKER_00
So as a user, I have, but just everything I ve done that I experienced with Linux has been, I believe, always GCC.
1
SPEAKER_00
Yeah, I think Linux still defaults to GCC.
1
SPEAKER_00
And is there a reason for that, or is it because, I mean, is there a reason?
1
SPEAKER_01
It s a combination of technical and social reasons.
1
SPEAKER_01
Many Linux developers do use Clang, but the distributions, for lots of reasons, use GCC historically, and they ve not switched.
1
SPEAKER_00
Yeah.
1
SPEAKER_00
And just anecdotally online, it seems that LLVM has either reached the level of GCC or superseded on different features or whatever.
1
SPEAKER_00
The way I would say it is that they re so close it doesn t matter.
1
SPEAKER_01
Yeah, exactly.
1
SPEAKER_01
Like they re slightly better in some ways, slightly worse in other ways, but it doesn t actually really matter anymore at that level.
1
SPEAKER_00
So in terms of optimization breakthroughs, it s just been solid incremental work.
1
SPEAKER_01
Yeah, which describes a lot of compilers.
1
SPEAKER_01
The hard thing about compilers, in my experience, is the engineering, the software engineering, making it so that you can have hundreds of people collaborating on really detailed, low-level work and scaling that.
1
SPEAKER_01
And that s really hard.
1
SPEAKER_01
And that s one of the things I think LLVM has done well.
1
SPEAKER_01
And that kind of goes back to the original design goals with it to be modular and things like that.
1
SPEAKER_01
And incidentally, I don t want to take all the credit for this, right?
1
SPEAKER_01
I mean, some of the best parts about LLVM is that it was designed to be modular.
1
SPEAKER_01
And when I started, I would write, for example, a register allocator, and then somebody much smarter than me would come in and pull it out and replace it with something else that they would come up with.
1
SPEAKER_01
And because it s modular, they were able to do that.
1
SPEAKER_01
And that s one of the challenges with GCC, for example, is replacing subsystems is incredibly difficult.
1
SPEAKER_01
It can be done, but it wasn t designed for that.
1
SPEAKER_01
And that s one of the reasons that LLVM has been very successful in the research world as well.
1
SPEAKER_00
But in a community sense, Guido van Rossum, right, from Python, just retired from
1
SPEAKER_00
what is it, benevolent dictator for life, right?
1
SPEAKER_00
So in managing this community of brilliant compiler folks, did it, for a time at least, fall on you to approve things?
1
SPEAKER_01
Oh yeah, so I mean, I still have something like an order of magnitude more patches in LLVM than anybody else.
1
SPEAKER_01
And many of those I wrote myself.
1
SPEAKER_00
But you still write, I mean, you still close to the, I don t know what the expression is, to the medal.
1
SPEAKER_00
You still write code.
1
SPEAKER_00
Yeah, I still write code.
1
SPEAKER_01
Not as much as I was able to in grad school, but that s an important part of my identity.
1
SPEAKER_01
But the way that Elevium has worked over time is that when I was a grad student, I could do all the work and steer everything and review every patch and make sure everything was done exactly the way my opinionated
1
SPEAKER_01
sense, felt like it should be done, and that was fine.
1
SPEAKER_01
But as things scale, you can t do that, right?
1
SPEAKER_01
And so what ends up happening is LLVM has a hierarchical system of what s called co-donors.
1
SPEAKER_01
These co-donors are given the responsibility not to do all the work,
1
SPEAKER_01
not necessarily to review all the patches, but to make sure that the patches do get reviewed and make sure that the right thing s happening architecturally in their area.
1
SPEAKER_01
And so what you ll see is you ll see that, for example, hardware manufacturers end up owning the hardware-specific parts of their hardware.
1
SPEAKER_01
That s very common.
1
SPEAKER_01
um, leaders in the community that have done really good work naturally become the de facto owner of something.
1
SPEAKER_01
And then usually somebody else is like, how about we make them the official code owner?
1
SPEAKER_01
And then, and then we ll have somebody to make sure that all patches get reviewed in a timely manner.
1
SPEAKER_01
And then everybody s like, yes, that s obvious.
1
SPEAKER_01
And then it happens, right?
1
SPEAKER_01
And usually this is a very organic thing, which is great.
1
SPEAKER_01
And so I m nominally the top of that stack still, but I don t spend a lot of time reviewing patches.
1
SPEAKER_01
What I do is I help negotiate a lot of the technical disagreements that end up happening and making sure that the community as a whole makes progress and is moving in the right direction and doing that.
1
SPEAKER_01
So we also started a nonprofit.
1
SPEAKER_01
six years ago, seven years ago, it s time s gone away.
1
SPEAKER_01
And the nonprofit, the, the LVM foundation nonprofit, uh, helps oversee all the business sides of things and make sure that the events that the LVM community has are funded and set up and run correctly and stuff like that.
1
SPEAKER_01
But the foundation is very much, uh, stays out of the technical side of, uh, where, where the project is going.
1
SPEAKER_01
it s it sounds like a lot of it is uh... just organic just uh... yeah well then this is all of them is almost twenty years old which is hard to believe somebody point out to me recently that uh... albion is now older than gcc was one of them started
1
SPEAKER_01
So time has a way of getting away from you, but the good thing about that is it has a really robust, really amazing community of people that are in their professional lives, spread across lots of different companies, but it s a community of people that are interested in similar kinds of problems, and have been working together effectively for years, and have a lot of trust and respect for each other, and even if they don t always agree, that we re able to find a path forward.
2
SPEAKER_00
So then, in a slightly different flavor of effort, you started Apple in 2005 with the task of making, I guess, LLVM production ready.
2
SPEAKER_00
And then eventually, 2013 through 2017, leading the entire developer tools department.
2
SPEAKER_00
We re talking about LLVM, Xcode, Objective-C to Swift.
2
SPEAKER_00
So in a quick overview of your time there, what were the challenges?
2
SPEAKER_00
First of all, leading such a huge group of developers, what was the big motivator dream mission behind creating Swift, the early birth of it from Objective-C and so on and Xcode?
2
SPEAKER_01
What are some challenges?
2
SPEAKER_01
So these are different questions.
2
SPEAKER_01
Yeah, I know, but I want to talk about the other stuff too.
2
SPEAKER_01
I ll stay on the technical side, then we can talk about the big team pieces, if that s OK?
2
SPEAKER_00
Sure.
2
SPEAKER_01
So I guess to really oversimplify many years of hard work, LLVM started, joined Apple, became a thing, became successful, and became deployed.
2
SPEAKER_01
But then there s a question about how do we actually parse the source code?
2
SPEAKER_01
So LLVM is that back part, the optimizer and the code generator.
2
SPEAKER_01
And LLVM is really good for Apple as it went through a couple of hardware transitions.
2
SPEAKER_01
I joined right at the time of the Intel transition, for example, and 64-bit transitions, and then the transition to ARM with the iPhone.
2
SPEAKER_01
And so LLVM was very useful for some of these kinds of things.
2
SPEAKER_01
But at the same time, there s a lot of questions around developer experience.
2
SPEAKER_01
And so if you re a programmer pounding out, at the time, Objective-C code,
2
SPEAKER_01
the error message you get, the compile time, the turnaround cycle, the tooling in the IDE were not great, were not as good as they could be.
2
SPEAKER_01
And so, you know, as I occasionally do, I m like, well, okay, how hard is it to write a C compiler, right?
2
SPEAKER_01
And so I m not going to commit to anybody.
2
SPEAKER_01
I m not going to tell anybody.
2
SPEAKER_01
I m just going to just do it on nights and weekends and start working on it.
2
SPEAKER_01
And then, you know, I built up and see there s this thing called the preprocessor, which people don t like, but it s actually really hard and complicated and includes a bunch of
2
SPEAKER_01
really weird things like trigraphs and other stuff like that that are really nasty, and it s the crux of a bunch of the performance issues in the compiler.
2
SPEAKER_01
I started working on the parser and kind of got to the point where I m like, oh, you know what?
2
SPEAKER_01
We could actually do this.
2
SPEAKER_01
Everybody s saying that this is impossible to do, but it s actually just hard.
2
SPEAKER_01
It s not impossible.
2
SPEAKER_01
Eventually told my manager about it, and he s like, oh, wow, this is great.
2
SPEAKER_01
We do need to solve this problem.
2
SPEAKER_01
Oh, this is great.
2
SPEAKER_01
We can get you one other person to work with you on this.
2
SPEAKER_01
And slowly a team is formed, and it starts taking off.
2
SPEAKER_01
And C++, for example, huge, complicated language.
2
SPEAKER_01
People always assume that it s impossible to implement.
2
SPEAKER_01
And it s very nearly impossible, but it s just really, really hard.
2
SPEAKER_01
And the way to get there is to build it one piece at a time, incrementally.
2
SPEAKER_01
And that was only possible because we were lucky to hire some really exceptional engineers that knew various parts of it very well and could do great things.
2
SPEAKER_01
Swift was kind of a similar thing.
2
SPEAKER_01
So Swift came from, we were just finishing off the first version of C++ support in Clang.
2
SPEAKER_01
And C++ is a very formidable and very important language, but it s also ugly in lots of ways.
2
SPEAKER_01
And you can t influence C++ without thinking there has to be a better thing, right?
2
SPEAKER_01
And so I started working on Swift, again, with no hope or ambition that would go anywhere, just let s see what could be done.
2
SPEAKER_01
Let s play around with this thing.
2
SPEAKER_01
It was, you know, me in my spare time, not
2
SPEAKER_01
telling anybody about it, kind of a thing.
2
SPEAKER_01
And it made some good progress.
2
SPEAKER_01
I m like, actually, it would make sense to do this.
2
SPEAKER_01
At the same time, I started talking with the senior VP of software at the time, a guy named Bertrand Serlet.
2
SPEAKER_01
And Bertrand was very encouraging.
2
SPEAKER_01
He was like, well, you know, let s have fun.
2
SPEAKER_01
Let s talk about this.
2
SPEAKER_01
And he was a little bit of a language guy.
2
SPEAKER_01
And
2
SPEAKER_01
So he helped guide some of the early work and encouraged me and got things off the ground.
2
SPEAKER_01
And eventually I told my manager and told other people, and it started making progress.
2
SPEAKER_01
The complicating thing with Swift was that the idea of doing a new language is not obvious to anybody, including myself.
2
SPEAKER_01
And the tone at the time was that the iPhone was successful because of Objective-C.
2
SPEAKER_00
Oh, interesting.
2
SPEAKER_01
Not despite of or just because of.
2
SPEAKER_01
And you have to understand that at the time, Apple was hiring software people that loved Objective-C, right?
2
SPEAKER_01
And it wasn t that they came despite Objective-C, they loved Objective-C and that s why they got hired.
2
SPEAKER_01
And so you had a software team that the leadership in many cases went all the way back to Next, where Objective-C really became real.
2
SPEAKER_01
And so they, quote unquote, grew up writing Objective-C.
2
SPEAKER_01
And many of the individual engineers all were hired because they loved Objective-C.
2
SPEAKER_01
And so this notion of, okay, let s do new language was kind of heretical in many ways, right?
2
SPEAKER_01
Um, meanwhile, my, my sense was that the outside community wasn t really in love with objective C. Some people were, and some of the most outspoken people were, but other people were hitting challenges because it has very sharp corners and it s difficult to learn.
2
SPEAKER_01
Um, and so one of the challenges of making Swift happen that was totally non-technical is,
2
SPEAKER_01
the social part of what do we do?
2
SPEAKER_01
Like if we do a new language, which at Apple, many things happen that don t ship, right?
2
SPEAKER_01
So if we ship it, what is the metrics of success?
2
SPEAKER_01
Why would we do this?
2
SPEAKER_01
Why wouldn t we make Objective-C better?
2
SPEAKER_01
If Objective-C has problems, let s file off those rough corners and edges.
2
SPEAKER_01
And one of the major things that became the reason to do this was this notion of safety, memory safety.
2
SPEAKER_01
And the way Objective-C works is that a lot of the object system and everything else is built on top of pointers in C. Objective-C is an extension on top of C. And so pointers are unsafe.
2
SPEAKER_01
And if you get rid of the pointers, it s not Objective-C anymore.
2
SPEAKER_01
And so fundamentally, that was an issue that you could not fix safety or memory safety without fundamentally changing the language.
2
SPEAKER_01
And so once we got through that part of the
2
SPEAKER_01
the mental process and the thought process, it became a design process of saying, OK, well, if we re going to do something new, what is good?
2
SPEAKER_01
How do we think about this?
2
SPEAKER_01
And what do we like?
2
SPEAKER_01
And what are we looking for?
2
SPEAKER_01
And that was a very different phase of it.
2
SPEAKER_00
So what are some design choices early on in SWIFT?
2
SPEAKER_00
We re talking about braces.
2
SPEAKER_00
Are you making a type language or not?
2
SPEAKER_00
All those kinds of things.
2
SPEAKER_01
Yeah, so some of those were obvious given the context.
2
SPEAKER_01
So a typed language, for example, Objective-C is a typed language, and going with an untyped language wasn t really seriously considered.
2
SPEAKER_01
We wanted the performance and we wanted refactoring tools and other things like that to go with typed languages.
2
SPEAKER_00
Quick dumb question.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
Was it obvious, I think this would be a dumb question, but was it obvious that the language has to be a compiled language?
2
SPEAKER_01
Yes, that s not a dumb question.
2
SPEAKER_01
Earlier, I think late 90s, Apple had seriously considered moving its development experience to Java.
2
SPEAKER_01
But Swift started in 2010, which was several years after the iPhone.
2
SPEAKER_01
It was when the iPhone was definitely on an upper trajectory, and the iPhone was still extremely, and is still a bit memory constrained, right?
2
SPEAKER_01
And so being able to compile the code and then ship it, and then having,
2
SPEAKER_01
standalone code that is not JIT compiled is a very big deal, and it s very much part of the Apple value system.
2
SPEAKER_01
Now, JavaScript s also a thing, right?
2
SPEAKER_01
I mean, it s not that this is exclusive, and technologies are good depending on how they re applied, right?
2
SPEAKER_01
But in the design of Swift, saying like, how can we make Objective-C better, right?
2
SPEAKER_01
Objective-C is statically compiled, and that was the contiguous, natural thing to do.
2
SPEAKER_00
Just skip ahead a little bit, right back, just as a question, as you think about today in 2019, in your work at Google, TensorFlow and so on, is again, compilations, static compilation, still the right thing?
2
SPEAKER_01
Yeah, so the funny thing after working on compilers for a really long time is that, and this is one of the things that LLVM has helped with, is that I don t look at compilations being static or dynamic or interpreted or not.
2
SPEAKER_01
This is a spectrum.
2
SPEAKER_01
And one of the cool things about Swift is that Swift is not just statically compiled.
2
SPEAKER_01
It s actually dynamically compiled as well.
2
SPEAKER_01
And it can also be interpreted, though nobody s actually done that.
2
SPEAKER_01
And so what ends up happening when you use Swift in a workbook, for example, in Colab or in Jupyter, is it s actually dynamically compiling the statements as you execute them.
2
SPEAKER_01
And so this gets back to the software engineering problems, right?
2
SPEAKER_01
Where if you layer the stack properly, you can actually completely change how and when things get compiled because you have the right abstractions there.
2
SPEAKER_01
And so the way that a Colab workbook works with Swift is that when you start typing into it, it creates a process, a Unix process,
2
SPEAKER_01
and then each line of code you type in, it compiles it through the Swift compiler, the front end part, and then sends it through the optimizer, JIT compiles machine code, and then injects it into that process.
2
SPEAKER_01
And so as you re typing new stuff, it s like squirting in new code and overwriting and replacing and updating code in place.
2
SPEAKER_01
And the fact that it can do this is not an accident, like Swift was designed for this, but it s an important part of how the language was set up and how it s layered.
2
SPEAKER_01
And this is a non-obvious piece.
2
SPEAKER_01
And one of the things with Swift that was, for me, a very strong design point is to make it so that you can,
2
SPEAKER_01
learn it very quickly.
2
SPEAKER_01
And so from a language design perspective, the thing that I always come back to is this UI principle of progressive disclosure of complexity.
2
SPEAKER_01
And so in Swift, you can start by saying print quote, hello world, quote, right?
2
SPEAKER_01
And there s no slash n, just like Python, one line of code, no main, no header files, no public static class void, blah, blah, blah, string, like Java has, right?
2
SPEAKER_01
So one line of code, right?
2
SPEAKER_01
And you can teach that, and it works great.
2
SPEAKER_01
Then you can say, well, let s introduce variables.
2
SPEAKER_01
And so you can declare a variable with var.
2
SPEAKER_01
So var x equals 4.
2
SPEAKER_01
What is a variable?
2
SPEAKER_01
You can use x, x plus 1.
2
SPEAKER_01
This is what it means.
2
SPEAKER_01
Then you can say, well, how about control flow?
2
SPEAKER_01
Well, this is what an if statement is.
2
SPEAKER_01
This is what a for statement is.
2
SPEAKER_01
This is what a while statement is.
2
SPEAKER_01
And then you can say, let s introduce functions, right?
2
SPEAKER_01
And, and many languages like Python have had this, this kind of notion of let s introduce small things, and then you can add complexity, then you can introduce classes, and then you can add generics in the case of Swift, and then you can, and modules and build out in terms of,
2
SPEAKER_01
the things that you re expressing, but this is not very typical for compiled languages.
2
SPEAKER_01
And so this was a very strong design point and one of the reasons that Swift in general is designed with this factoring of complexity in mind so that the language can express powerful things.
2
SPEAKER_01
You can write firmware in Swift if you want to, but it has a very high level feel, which is really this perfect blend because
2
SPEAKER_01
Often you have very advanced library writers that want to be able to use the nitty-gritty details, but then other people just want to use the libraries and work at a higher abstraction level.
2
SPEAKER_00
It s kind of cool that I saw that you can just interoperability.
2
SPEAKER_00
I don t think I pronounced that word enough, but you can just drag in Python.
2
SPEAKER_00
It s just strange.
2
SPEAKER_00
You can import, like I saw this in the demo.
2
SPEAKER_00
How do you make that happen?
2
SPEAKER_00
Is that as easy as it looks or is it?
2
SPEAKER_01
Yes, as easy as it looks.
2
SPEAKER_01
That s not a stage magic hack or anything like that.
2
SPEAKER_00
I don t mean from the user perspective.
2
SPEAKER_00
I mean from the implementation perspective to make it happen.
2
SPEAKER_01
So it s easy once all the pieces are in place.
2
SPEAKER_01
The way it works, so if you think about a dynamically typed language like Python, right, you can think about it in two different ways.
2
SPEAKER_01
You can say it has no types, right, which is what most people would say.
2
SPEAKER_01
Or you can say it has one type.
2
SPEAKER_01
And you could say it has one type, and it s like the Python object.
2
SPEAKER_01
And the Python object gets passed around, and because there s only one type, it s implicit.
2
SPEAKER_01
And so what happens with Swift and Python talking to each other, Swift has lots of types.
2
SPEAKER_01
It has arrays, and it has strings, and all classes and that kind of stuff.
2
SPEAKER_01
But it now has a Python object.
2
SPEAKER_01
type.
2
SPEAKER_01
So there is one Python object type.
2
SPEAKER_01
And so when you say import NumPy, what you get is a Python object, which is the NumPy module.
2
SPEAKER_01
And then you say np.array, and it says, OK, hey, Python object, I have no idea what you are.
2
SPEAKER_01
Give me your array member.
2
SPEAKER_01
right?
2
SPEAKER_01
Okay, cool.
2
SPEAKER_01
And just, it just uses dynamic stuff, talks to the Python interpreter and says, Hey Python, what s the daughter a member in that Python object?
2
SPEAKER_01
It gives you back another Python object.
2
SPEAKER_01
And now you say parentheses for the call and the arguments are going to pass.
2
SPEAKER_01
And so then it says, Hey, a Python object that is the result of NP dot array call with these arguments, right?
2
SPEAKER_01
And again, calling into the Python interpreter to do that work.
2
SPEAKER_01
And so,
2
SPEAKER_01
Right now, this is all really simple.
2
SPEAKER_01
And if you dive into the code, what you ll see is that the Python module in Swift is something like 1,200 lines of code or something.
2
SPEAKER_01
It s written in pure Swift.
2
SPEAKER_01
It s super simple, and it s built on top of the C interoperability because it just talks to the Python interpreter.
2
SPEAKER_01
But making that possible required us to add two major language features to Swift to be able to express these dynamic calls and the dynamic member lookups.
2
SPEAKER_01
And so what we ve done over the last year is we ve proposed, implemented, standardized, and contributed new language features to the Swift language in order to make it so it is really trivial.
2
SPEAKER_01
And this is one of the things about Swift that is critical to the Swift for TensorFlow work, which is that we can actually add new language features.
2
SPEAKER_01
And the bar for adding those is high, but it s what makes it possible.
2
SPEAKER_00
So you re now at Google doing incredible work on several things, including TensorFlow.
2
SPEAKER_00
So TensorFlow 2.0 or whatever leading up to 2.0 has by default in 2.0 has eager execution.
2
SPEAKER_00
And yet in order to make code optimized for GPU or TPU or some of these systems, computation needs to be converted to a graph.
2
SPEAKER_00
So what s that process like?
2
SPEAKER_00
What are the challenges there?
2
SPEAKER_01
Yeah.
2
SPEAKER_01
So I, I, I m tangentially involved in this, but the, the way that it works with autograph is that you mark your, your function with the decorator.
2
SPEAKER_01
And when Python calls it, that decorator is invoked.
2
SPEAKER_01
And then it says, before I call this function, you can transform it.
2
SPEAKER_01
And so the way autograph works is,
2
SPEAKER_01
as far as I understand, is it actually uses the Python parser to go parse that, turn it into a syntax tree, and now apply compiler techniques to, again, transform this down into TensorFlow graphs.
2
SPEAKER_01
And so you can think of it as saying, hey, I have an if statement.
2
SPEAKER_01
I m going to create an if node in the graph, like you say, tf.cond.
2
SPEAKER_01
You have a multiply.
2
SPEAKER_01
Well, I ll turn that into a multiply node in the graph.
2
SPEAKER_01
And it becomes this tree transformation.
2
SPEAKER_00
So where does the Swift for TensorFlow come in?
2
SPEAKER_00
Which is, you know, parallels, for one, Swift is an interface, like Python is an interface to TensorFlow, but it seems like there s a lot more going on in just a different language interface.
2
SPEAKER_00
There s optimization methodology.
2
SPEAKER_01
So the TensorFlow world has a couple of different, what I d call front-end technologies.
2
SPEAKER_01
And so Swift and Python and Go and Rust and Julia and all these things share the TensorFlow graphs and all the runtime and everything that s later.
2
SPEAKER_01
And so Swift for TensorFlow is merely another front-end for TensorFlow, just like any of these other systems are.
2
SPEAKER_01
There s a major difference between, I would say, three camps of technologies here.
2
SPEAKER_01
There s Python, which is a special case, because the vast majority of the community efforts go into the Python interface.
2
SPEAKER_01
And Python has its own approaches for automatic differentiation.
2
SPEAKER_01
It has its own APIs and all this kind of stuff.
2
SPEAKER_01
There s Swift, which I ll talk about in a second.
2
SPEAKER_01
And then there s kind of everything else.
2
SPEAKER_01
And so the everything else are effectively language bindings.
2
SPEAKER_01
So they call into the TensorFlow runtime.
2
SPEAKER_01
But they usually don t have automatic differentiation, or they usually don t provide anything other than APIs that call the C APIs in TensorFlow.
2
SPEAKER_01
And so they re kind of wrappers for that.
2
SPEAKER_01
Swift is really kind of special.
2
SPEAKER_01
And it s a very different approach.
2
SPEAKER_01
Swift for TensorFlow, that is, is a very different approach.
2
SPEAKER_01
Because there we re saying, let s look at all the problems that need to be solved in the full stack of
2
SPEAKER_01
the TensorFlow compilation process, if you think about it that way.
2
SPEAKER_01
Because TensorFlow is fundamentally a compiler.
2
SPEAKER_01
It takes models, and then it makes them go fast on hardware.
2
SPEAKER_01
That s what a compiler does.
2
SPEAKER_01
And it has a front end, it has an optimizer, and it has many back ends.
2
SPEAKER_01
And so if you think about it the right way, or if you look at it in a particular way, it is a compiler, okay?
2
SPEAKER_01
And so Swift is merely another front end, but it s saying, and the design principle is saying, let s look at all the problems that we face as machine learning practitioners and what is the best possible way we can do that, given the fact that we can change literally anything in this entire stack.
2
SPEAKER_01
In Python, for example, where the vast majority of the engineering and effort has gone into, it s constrained by being the best possible thing you can do with a Python library.
2
SPEAKER_01
There are no Python language features that are added because of machine learning that I m aware of.
2
SPEAKER_01
They added a matrix multiplication operator with that, but that s as close as you get.
2
SPEAKER_01
And so with Swift, it s hard, but you can add language features to the language and there s a community process for that.
2
SPEAKER_01
And so we look at these things and say, well, what is the right division of labor between the human programmer and the compiler?
2
SPEAKER_01
And Swift has a number of things that shift that balance.
2
SPEAKER_01
So because it has a type system, for example, it makes certain things possible for analysis of the code and
2
SPEAKER_01
the compiler can automatically build graphs for you without you thinking about them.
2
SPEAKER_01
That s a big deal for a programmer.
2
SPEAKER_01
You just get free performance, you get clustering and fusion and optimization, things like that, without you as a programmer having to manually do it because the compiler can do it for you.
2
SPEAKER_01
Automatic differentiation is another big deal, and I think one of the key contributions of
3
SPEAKER_01
of the Swift for TensorFlow project, is that there s this entire body of work on automatic differentiation that dates back to the Fortran days.
3
SPEAKER_01
People doing a tremendous amount of numerical computing in Fortran used to write these, what they call source-to-source translators, where you take a bunch of code, shove it into a mini compiler, and it would push out more Fortran code.
3
SPEAKER_01
But it would generate the backwards passes for your functions for you, the derivatives, right?
3
SPEAKER_01
And so,
3
SPEAKER_01
in that work in the seventies, a tremendous number of optimizations, a tremendous number of, uh, techniques for fixing numerical instability and other, other kinds of problems were developed, but they re very difficult to port into a world where in eager execution, you get an op by op at a time.
3
SPEAKER_01
Like you need to be able to look at an entire function and be able to reason about what s going on.
3
SPEAKER_01
And so when you have a language-integrated automatic differentiation, which is one of the things that the SWIFT project is focusing on, you can open all these techniques and reuse them in familiar ways.
3
SPEAKER_01
But the language integration piece has a bunch of design room in it, and it s also complicated.
3
SPEAKER_00
The other piece of the puzzle here that s kind of interesting is TPUs at Google.
3
SPEAKER_00
Yes.
3
SPEAKER_00
So, you know, we re in a new world with deep learning.
3
SPEAKER_00
Yes.
3
SPEAKER_00
It constantly is changing, and I imagine, without disclosing anything, I imagine, you know, you re still innovating on the TPU front, too.
3
SPEAKER_00
Indeed.
3
SPEAKER_00
So, how much sort of interplay is there between software and hardware in trying to figure out how together move towards an optimized solution?
3
SPEAKER_01
There s an incredible amount, so we re on our third generation of TPUs, which are now 100 petaflops in a very large liquid-cooled box, a virtual box with no cover.
3
SPEAKER_01
And as you might imagine, we re not out of ideas yet.
3
SPEAKER_01
The great thing about TPUs is that they re a perfect example of hardware-software co-design.
3
SPEAKER_01
And so it s about saying what hardware do we build to solve certain classes of machine learning problems?
3
SPEAKER_01
Well, the algorithms are changing.
3
SPEAKER_01
Like the hardware takes, you know, some cases years to produce, right?
3
SPEAKER_01
And so you have to make bets and decide what is going to happen.
3
SPEAKER_01
And so, and what is the best way to spend the transistors to get the maximum, you know, performance per watt or area per cost or like whatever it is that you re optimizing for.
3
SPEAKER_01
And so one of the amazing things about TPUs is this numeric format called bfloat16.
3
SPEAKER_01
bfloat16 is a compressed 16-bit floating point format, but it puts the bits in different places.
3
SPEAKER_01
In numeric terms, it has a smaller mantissa and a larger exponent.
3
SPEAKER_01
That means that it s less precise, but it can represent larger ranges of values, which in the machine learning context is really important and useful because
3
SPEAKER_01
sometimes you have very small gradients you want to accumulate and very, very small numbers that are important to move things as you re learning, but sometimes you have very large magnitude numbers as well.
3
SPEAKER_01
And BFLOAT16 is not as precise, the mantissa is small, but it turns out the machine learning algorithms actually want to generalize.
3
SPEAKER_01
And so there s, you know, theories that this actually increases
3
SPEAKER_01
the ability for the network to generalize across data sets.
3
SPEAKER_01
And regardless of whether it s good or bad, it s much cheaper at the hardware level to implement because the area and time of a multiplier is n squared in the number of bits in the mantissa, but it s linear with size of the exponent.
3
SPEAKER_01
And you re connected to both efforts here, both on the hardware and the software side?
3
SPEAKER_01
Yeah, and so that was a breakthrough coming from the research side and people working on optimizing network transport of weights across the network originally and trying to find ways to compress that.
3
SPEAKER_01
But then it got burned into silicon, and it s a key part of what makes TPU performance so amazing.
3
SPEAKER_01
And great.
3
SPEAKER_01
Now, TPUs have many different aspects that are important.
3
SPEAKER_01
But the co-design between the low-level compiler bits and the software bits and the algorithms is all super important.
3
SPEAKER_01
And it s this amazing trifecta that only Google can do.
3
SPEAKER_00
Yeah, that s super exciting.
3
SPEAKER_00
So can you tell me about MLIR project, previously the secretive one?
3
SPEAKER_01
Yeah, so MLIR is a project that we announced at a compiler conference.
3
SPEAKER_01
three weeks ago or something, at the Compilers for Machine Learning conference.
3
SPEAKER_01
Basically, again, if you look at TensorFlow as a compiler stack, it has a number of compiler algorithms within it.
3
SPEAKER_01
It also has a number of compilers that get embedded into it, and they re made by different vendors.
3
SPEAKER_01
For example,
3
SPEAKER_01
Google has XLA, which is a great compiler system.
3
SPEAKER_01
NVIDIA has TensorRT.
3
SPEAKER_01
Intel has Ngraph.
3
SPEAKER_01
There s a number of these different compiler systems, and they re very hardware specific, and they re trying to solve different parts of the problems.
3
SPEAKER_01
But they re all kind of similar in a sense of they want to integrate with TensorFlow.
3
SPEAKER_01
Now, TensorFlow has an optimizer.
3
SPEAKER_01
and it has these different code generation technologies built in.
3
SPEAKER_01
The idea of MLIR is to build a common infrastructure to support all these different subsystems.
3
SPEAKER_01
And initially, it s to be able to make it so that they all plug in together and they can share a lot more code and can be reusable.
3
SPEAKER_01
But over time, we hope that the industry will start collaborating and sharing code, and instead of reinventing the same things over and over again, that we can actually foster some of that working together to solve common problem energy that has been useful in the compiler field before.
3
SPEAKER_01
Beyond that, MLIR is, some people have joked that it s kind of LLVM too.
3
SPEAKER_01
It learns a lot about what LLVM has been good and what LLVM has done wrong.
3
SPEAKER_01
And it s a chance to fix that.
3
SPEAKER_01
And also there are challenges in the LLVM ecosystem as well, where LLVM is very good at the thing it was designed to do, but you know, 20 years later, the world has changed and people are trying to solve higher level problems and we need some new technology.
3
SPEAKER_00
And what s the future of open source in this context?
3
SPEAKER_01
Very soon.
3
SPEAKER_01
So it is not yet open source, but it will be hopefully in the next couple months.
3
SPEAKER_01
So you still believe in the value of open source in these kinds of companies?
3
SPEAKER_01
Oh yeah, absolutely.
3
SPEAKER_01
And I think that the TensorFlow community at large fully believes in open source.
3
SPEAKER_00
So I mean, there is a difference between Apple, where you were previously, and Google now, in spirit and culture.
3
SPEAKER_00
And I would say the open sourcing of TensorFlow was a seminal moment.
3
SPEAKER_00
in the history of software, because here s this large company releasing a very large code base that s open sourcing.
3
SPEAKER_00
What are your thoughts on that?
3
SPEAKER_00
Happy or not, were you to see that kind of degree of open sourcing?
3
SPEAKER_00
So between the two, I prefer the Google approach, if that s what you re saying.
3
SPEAKER_01
The Apple approach makes sense given the historical context that Apple came from, but that s been 35 years ago.
3
SPEAKER_01
And I think that Apple is definitely adapting.
3
SPEAKER_01
And the way I look at it is that there s different kinds of concerns in this space, right?
3
SPEAKER_01
It is very rational for a business to care about making money.
3
SPEAKER_01
That fundamentally is what a business is about.
3
SPEAKER_01
Right.
3
SPEAKER_01
But I think it s also incredibly realistic to say it s not your string library.
3
SPEAKER_01
That s the thing that s going to make you money.
3
SPEAKER_01
It s going to be the amazing UI product differentiating features and other things like that, that you build on top of your string library.
3
SPEAKER_01
And so, um, keeping your string library proprietary and secret and things like that, maybe not the, the important thing anymore.
3
SPEAKER_01
Right.
3
SPEAKER_01
Where before platforms were, were different.
3
SPEAKER_01
Right.
3
SPEAKER_01
And,
3
SPEAKER_01
Even 15 years ago, things were a little bit different.
3
SPEAKER_01
But the world is changing.
3
SPEAKER_01
So Google strikes a very good balance, I think.
3
SPEAKER_01
And I think that TensorFlow being open source really changed the entire machine learning field, and it caused a revolution in its own right.
3
SPEAKER_01
And so I think it s amazingly forward-looking because
3
SPEAKER_01
I could have imagined, and I wasn t at Google at the time, but I could imagine a different context and a different world where a company says, machine learning is critical to what we re doing, we re not gonna give it to other people.
3
SPEAKER_01
And so that decision is a profoundly brilliant insight that I think has really led to the world being better, and better for Google as well.
3
SPEAKER_00
And has all kinds of ripple effects.
3
SPEAKER_00
I think it is,
3
SPEAKER_00
Really, I mean, you can t understate Google deciding how profound that is for software.
3
SPEAKER_01
It s awesome.
3
SPEAKER_01
And again, I can understand the concern about if we release our machine learning software, our competitors could go faster.
3
SPEAKER_01
But on the other hand, I think that open sourcing TensorFlow has been fantastic for Google.
3
SPEAKER_01
And I m sure that decision was very non-obvious at the time.
3
SPEAKER_01
But I think it s worked out very well.
3
SPEAKER_00
So let s try this real quick.
3
SPEAKER_00
You were at Tesla for five months as the VP of Autopilot Software.
3
SPEAKER_00
You led the team during the transition from H Hardware 1 to Hardware 2.
3
SPEAKER_00
I have a couple questions.
3
SPEAKER_00
So one, first of all, to me, that s one of the bravest engineering decisions undertaking really ever in the automotive industry, software-wise, starting from scratch.
3
SPEAKER_00
It s a really brave engineering decision.
3
SPEAKER_00
So my one question there is, what was that like?
3
SPEAKER_00
What was the challenge of that?
3
SPEAKER_00
Do you mean the career decision of jumping from a comfortable good job into the unknown, or?
3
SPEAKER_00
That combined, so at the individual level, you making that decision, and then when you show up, you know, it s a really hard engineering problem.
3
SPEAKER_00
Yes.
3
SPEAKER_00
So you could just stay
3
SPEAKER_00
Maybe slow down, say hardware one, or those kinds of decisions.
3
SPEAKER_00
Just taking it full on, let s do this from scratch.
3
SPEAKER_00
What was that like?
3
SPEAKER_01
Well, so, I mean, I don t think Tesla has a culture of taking things slow and seeing how it goes.
3
SPEAKER_01
And one of the things that attracts me about Tesla is it s very much a gung-ho, let s change the world, let s figure it out kind of a place.
3
SPEAKER_01
And so I have a huge amount of respect for that.
3
SPEAKER_01
Tesla has done very smart things with Hardware 1 in particular, and the Hardware 1 design was originally designed to be very simple automation features in the car for traffic-aware cruise control and things like that.
3
SPEAKER_01
And the fact that they were able to effectively feature creep it into lane holding and a very useful driver assistance feature is pretty astounding, particularly given the details of the hardware.
3
SPEAKER_01
Hardware 2 built on that in a lot of ways.
3
SPEAKER_01
And the challenge there was that they were transitioning from a third-party provided vision stack to an in-house built vision stack.
3
SPEAKER_01
And so for the first step, which I mostly helped with, was getting onto that new vision stack, and that was very challenging.
3
SPEAKER_01
And it was time critical for various reasons, and it was a big leap, but it was fortunate that it built on a lot of the knowledge and expertise and the team that had built Hardware 1 s driver assistance features.
3
SPEAKER_00
So you spoke in a collected and kind way about your time at Tesla, but it was ultimately not a good fit.
3
SPEAKER_00
Elon Musk, we ve talked on this podcast, several guests of the course, Elon Musk continues to do some of the most bold and innovative engineering work in the world, at times at the cost to some of the members of the Tesla team.
3
SPEAKER_00
What did you learn about working in this chaotic world with Elon?
3
SPEAKER_01
Yeah, so I guess I would say that when I was at Tesla, I experienced and saw the highest degree of turnover I d ever seen in a company, which was a bit of a shock.
3
SPEAKER_01
But one of the things I learned and I came to respect is that Elon s able to attract amazing talent because he has a very clear vision of the future and he can get people to buy into it because they want that future to happen.
3
SPEAKER_01
Right.
3
SPEAKER_01
And the power of vision is something that I have a tremendous amount of respect for.
3
SPEAKER_01
And I think that Elon is fairly singular in the world in terms of the, uh, things he s able to get people to believe in.
3
SPEAKER_01
And it s, it s a very, it s very, there are many people that stand in the street corner and say, ah, we re going to go to Mars.
3
SPEAKER_01
Right.
3
SPEAKER_01
But then, but then there are a few people that can get other others to buy into it and believe in build the path and make it happen.
3
SPEAKER_01
And so,
3
SPEAKER_01
I respect that.
3
SPEAKER_01
I don t respect all of his methods, but I have a huge amount of respect for that.
3
SPEAKER_00
You ve mentioned in a few places, including in this context, working hard.
3
SPEAKER_00
What does it mean to work hard?
3
SPEAKER_00
And when you look back at your life, what were some of the most brutal periods of having to really sort of put everything you have into something?
3
SPEAKER_01
Yeah, good question.
3
SPEAKER_01
So working hard can be defined a lot of different ways, so a lot of hours, and so that is true.
3
SPEAKER_01
The thing to me that s the hardest is both being short-term focused on delivering and executing and making a thing happen, while also thinking about the longer term and trying to balance that, right?
3
SPEAKER_01
Because if you are myopically focused on solving a task and getting that done,
3
SPEAKER_01
And only think about that incremental next step, you will miss the next big hill you should jump over to, right?
3
SPEAKER_01
And so I ve been really fortunate that I ve been able to kind of oscillate between the two.
3
SPEAKER_01
And historically at Apple, for example, that was made possible because I was able to work with some really amazing people and build up teams and leadership structures.
3
SPEAKER_01
and allow them to grow in their careers and take on responsibility, thereby freeing up me to be a little bit crazy and thinking about the next thing.
3
SPEAKER_01
And so it s a lot of that, but it s also about, you know, with experience, you make connections that other people don t necessarily make.
3
SPEAKER_01
And so I think that is, that s a big part as well.
3
SPEAKER_01
But the bedrock is just a lot of hours and
3
SPEAKER_01
you know, that s, that s okay with me.
3
SPEAKER_01
Um, there s different theories on work life balance and my theory for myself, which I do not project onto the team, but my theory for myself is that, you know, I, I want to love what I m doing and work really hard.
3
SPEAKER_01
And my purpose, I feel like my goal is to change the world and make it a better place.
3
SPEAKER_01
And that s, that s what I m really motivated to do.
3
SPEAKER_00
So last question, LLVM logo s a dragon.
3
SPEAKER_00
You explain that this is because dragons have connotations of power, speed, intelligence.
3
SPEAKER_00
It can also be sleek, elegant, and modular, though you remove the modular part.
3
SPEAKER_00
What is your favorite dragon-related character from fiction, video, or movies?
3
SPEAKER_01
Those are all very kind ways of explaining it.
3
SPEAKER_01
Do you want to know the real reason it s a dragon?
3
SPEAKER_01
Yeah.
3
SPEAKER_01
Is that better?
3
SPEAKER_01
So there s a seminal book on compiler design called The Dragon Book.
3
SPEAKER_01
And so this is a really old now book on compilers.
3
SPEAKER_01
And so the dragon logo for LLVM came about because at Apple, we kept talking about LLVM-related technologies, and there s no logo to put on a slide.
3
SPEAKER_01
It s sort of like, what do we do?
3
SPEAKER_01
And somebody s like, well, what kind of logo should a compiler technology have?
3
SPEAKER_01
And I m like, I don t know.
3
SPEAKER_01
I mean, the dragon is the best thing that we ve got.
3
SPEAKER_01
And, you know, Apple somehow magically came up with the logo and it was a great thing and the whole community rallied around it.
3
SPEAKER_01
And then it got better as other graphic designers got involved, but that s originally where it came from.
3
SPEAKER_00
Is there dragons from fiction that you connect with?
3
SPEAKER_00
That Game of Thrones, Lord of the Rings, that kind of thing?
3
SPEAKER_01
Lord of the Rings is great.
3
SPEAKER_01
I also like role-playing games and things like computer role-playing games.
3
SPEAKER_01
And so dragons often show up in there.
3
SPEAKER_01
But it really comes back to the book.
3
SPEAKER_01
Oh no, we need a thing.
3
SPEAKER_01
And hilariously, one of the funny things about LLVM is that my wife, who s amazing, runs the LLVM Foundation.
3
SPEAKER_01
and she goes to Grace Hopper and is trying to get more women involved in the, she s also a compiler engineer, so she s trying to get other women to get interested in compilers and things like this, and so she hands out the stickers, and people like the LLVM sticker because of Game of Thrones.
3
SPEAKER_01
And so sometimes culture has this helpful effect to get the next generation of compiler engineers engaged with the cause.
3
SPEAKER_00
Okay, awesome.
3
SPEAKER_00
Chris, thanks so much for talking to me.
3
SPEAKER_01
It s been great talking with you.