0
SPEAKER_01
The following is a conversation with Jeremy Howard.
0
SPEAKER_01
He s the founder of Fast AI, a research institute dedicated to making deep learning more accessible.
0
SPEAKER_01
He s also a distinguished research scientist at the University of San Francisco, a former president of Kaggle, as well as a top-ranking competitor there.
0
SPEAKER_01
And in general, he s a successful entrepreneur, educator, researcher, and an inspiring personality in the AI community.
0
SPEAKER_01
When someone asks me,
0
SPEAKER_01
How do I get started with deep learning?
0
SPEAKER_01
Fast.ai is one of the top places I point them to.
0
SPEAKER_01
It s free.
0
SPEAKER_01
It s easy to get started.
0
SPEAKER_01
It s insightful and accessible.
0
SPEAKER_01
And if I may say so, it has very little BS.
0
SPEAKER_01
They can sometimes dilute the value of educational content on popular topics like deep learning.
0
SPEAKER_01
Fast.ai has a focus on practical application of deep learning and hands-on exploration of the cutting edge that is incredibly both accessible to beginners and useful to experts.
0
SPEAKER_01
This is the Artificial Intelligence Podcast.
0
SPEAKER_01
If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.
0
SPEAKER_01
And now, here s my conversation with Jeremy Howard.
0
SPEAKER_01
What s the first program you ve ever written?
0
SPEAKER_00
First program I wrote that I remember would be at high school.
0
SPEAKER_00
I did an assignment where I decided to try to find out if there were some, like, better musical scales than the normal 12 tone, 12 interval scale.
0
SPEAKER_00
So I wrote a program on my Commodore 64 in BASIC that searched through other scale sizes to see if it could find one where there were more accurate, you know, harmonies.
0
SPEAKER_01
Like mid-tone?
0
SPEAKER_00
Like you want an actual exactly 3 to 2 ratio, whereas with a 12 interval scale, it s not exactly 3 to 2, for example.
0
SPEAKER_00
So that s well-tempered, as they say.
0
SPEAKER_01
And basic on a Commodore 64.
0
SPEAKER_01
Yeah.
0
SPEAKER_01
Where was the interest in music from?
0
SPEAKER_00
I did music all my life.
0
SPEAKER_00
So I played saxophone and clarinet and piano and guitar and drums and whatever.
0
SPEAKER_01
How does that thread go through your life?
0
SPEAKER_01
Where s music today?
0
SPEAKER_00
It s not where I wish it was.
0
SPEAKER_00
I, for various reasons, couldn t really keep it going, particularly because I had a lot of problems with RSI with my fingers.
0
SPEAKER_00
And so I had to kind of like cut back anything that used hands and fingers.
0
SPEAKER_00
I hope one day I ll be able to get back to it health-wise.
0
SPEAKER_01
So there s a love for music underlying it all?
0
SPEAKER_01
What s your favorite instrument?
0
SPEAKER_00
Saxophone.
0
SPEAKER_00
Baritone saxophone.
0
SPEAKER_00
Well, probably bass saxophone, but they re awkward.
0
SPEAKER_01
Well, I always love it when music is coupled with programming.
0
SPEAKER_01
There s something about a brain that utilizes those that emerges with creative ideas.
0
SPEAKER_01
So you ve used and studied quite a few programming languages.
0
SPEAKER_01
Can you give an overview of what you ve used?
0
SPEAKER_01
What are the pros and cons of each?
0
SPEAKER_00
Well, my favorite programming environment
0
SPEAKER_00
almost certainly was Microsoft Access back in like the earliest days.
0
SPEAKER_00
So that was Visual Basic for Applications, which is not a good programming language, but the programming environment was fantastic.
0
SPEAKER_00
It s like the ability to create, you know, user interfaces and tie data and actions to them and create reports and all that.
0
SPEAKER_00
I ve never seen anything as good.
0
SPEAKER_00
There s things nowadays like Airtable, which are like
0
SPEAKER_00
small subsets of that, which people love for good reason.
0
SPEAKER_00
But unfortunately, nobody s ever achieved anything like that.
0
SPEAKER_01
What is that?
0
SPEAKER_01
If you could pause on that for a second.
0
SPEAKER_01
Oh, Access.
0
SPEAKER_00
It s a database program that Microsoft produced, part of Office.
0
SPEAKER_00
And it kind of withered, you know, but basically it lets you, in a totally graphical way, create tables and relationships and queries and tie them to forms.
0
SPEAKER_00
set up, you know, event handlers and calculations.
0
SPEAKER_00
And it was a very complete, powerful system designed for not massive scalable things, but for like useful little applications that I loved.
0
SPEAKER_01
So what s the connection between Excel and Access?
0
SPEAKER_00
So very close.
0
SPEAKER_00
So Access kind of was the relational database equivalent, if you like.
0
SPEAKER_00
So people still do a lot of that stuff that should be in Access in Excel because they know it.
0
SPEAKER_00
Excel is great as well.
0
SPEAKER_00
But it s just not as
0
SPEAKER_00
richer programming model as VBA combined with a relational database.
0
SPEAKER_00
And so I ve always loved relational databases, but today, programming on top of a relational database is just a lot more of a headache.
0
SPEAKER_00
You know, you generally either need to kind of, you know, you need something that connects, that runs some kind of database server, unless you use SQLite, which has its own issues.
0
SPEAKER_00
Then you can often, if you want to get a nice programming model, you ll need to like create
0
SPEAKER_00
add an ORM on top and then, I don t know, there s all these pieces tied together.
0
SPEAKER_00
And it s just a lot more awkward than it should be.
0
SPEAKER_00
There are people that are trying to make it easier.
0
SPEAKER_00
So in particular, I think of F sharp, you know, Don Syme, who, um, him and his team have done a great job of, um, making, uh, something like a database appear in the type system.
0
SPEAKER_00
So you actually get like tab completion for fields and tables and stuff like that.
0
SPEAKER_00
Anyway, so that was kind of, anyway, so like that whole VBA office thing, I guess, was a starting point, which I still miss.
0
SPEAKER_00
And I got into standard Visual Basic, which... That s interesting.
0
SPEAKER_01
Just to pause on that for a second.
0
SPEAKER_01
It s interesting that you re connecting programming languages to the ease of management of data.
0
SPEAKER_00
Yeah.
0
SPEAKER_01
So in your use of programming languages, you always had a love and a connection with data.
0
SPEAKER_00
I ve always been interested in doing useful things for myself and for others, which generally means getting some data and doing something with it and putting it out there again.
0
SPEAKER_00
So that s been my interest throughout.
0
SPEAKER_00
So I also did a lot of stuff with AppleScript back in the early days.
0
SPEAKER_00
So it s kind of nice being able to
0
SPEAKER_00
get the computer and computers to talk to each other and to do things for you.
0
SPEAKER_00
And then I think that one, the programming language I most loved then would have been Delphi, which was Object Pascal.
0
SPEAKER_00
created by Anders Helsberg, who previously did Turbo Pascal and then went on to create .NET and then went on to create TypeScript.
0
SPEAKER_00
Delphi was amazing because it was like a compiled, fast language that was as easy to use as Visual Basic.
0
SPEAKER_01
Delphi, what is it similar to in more modern languages?
0
SPEAKER_00
Visual Basic.
0
SPEAKER_01
Visual Basic.
0
SPEAKER_00
Yeah, but a compiled fast version.
0
SPEAKER_00
So I m not sure there s anything quite like it anymore.
0
SPEAKER_00
If you took like C sharp or Java and got rid of the virtual machine and replaced it with something you could compile a small type binary.
0
SPEAKER_00
I feel like it s where Swift could get to with the new Swift UI and the cross platform development going on.
0
SPEAKER_00
Like that s one of my.
0
SPEAKER_00
dreams is that we ll hopefully get back to where Delphi was.
0
SPEAKER_00
There is actually a free Pascal project nowadays called Lazarus, which is also attempting to kind of recreate Delphi.
0
SPEAKER_00
So they re making good progress.
0
SPEAKER_01
So, okay, Delphi, that s one of your favorite programming languages.
0
SPEAKER_00
Or at least programming environments.
0
SPEAKER_00
Again, I d say Pascal s not a nice language.
0
SPEAKER_00
If you wanted to know specifically about what languages I like, I would definitely pick J as being an amazingly wonderful language.
0
SPEAKER_01
What s J?
0
SPEAKER_00
J, are you aware of APL?
0
SPEAKER_01
I am not, except from doing a little research on the work you ve done.
0
SPEAKER_00
Okay, so not at all surprising you re not familiar with it because it s not well known, but it s actually one of the main
0
SPEAKER_00
families of programming languages going back to the late 50s, early 60s.
0
SPEAKER_00
So there was a couple of major directions.
0
SPEAKER_00
One was the kind of lambda calculus, Alonzo Church direction, which I guess kind of Lisp and Scheme and whatever, which has a history going back to the early days of computing.
0
SPEAKER_00
The second was the kind of imperative slash OO, you know, algo, similar going on to C, C++, so forth.
0
SPEAKER_00
There was a third, which are called array-oriented languages, which started with a paper by a guy called Ken Iverson, which was actually a math theory paper, not a programming paper.
0
SPEAKER_00
notation as a tool for thought, and it was the development of a new type of math notation.
0
SPEAKER_00
And the idea is that this math notation was much more flexible, expressive, and also well-defined than traditional math notation, which is none of those things.
0
SPEAKER_00
Math notation is awful.
0
SPEAKER_00
And so he actually turned that into a programming language.
0
SPEAKER_00
And because this was the early 50s, or the, sorry, late 50s, all the names were available.
0
SPEAKER_00
So he called his language Programming Language, or APL.
0
SPEAKER_00
APL, wow.
0
SPEAKER_00
So APL is an implementation of notation as a tool for thought, by which he means math notation.
0
SPEAKER_00
And Ken and his son went on to do many things, but eventually they actually produced a new language that was built on top of all the learnings of APL, and that was called J. And J is the most expressive, composable,
0
SPEAKER_00
language of, you know, beautifully designed language I ve ever seen.
0
SPEAKER_01
Does it have object oriented components?
0
SPEAKER_01
Does it have that kind of thing?
0
SPEAKER_00
Not really.
0
SPEAKER_00
It s an array oriented language.
0
SPEAKER_00
It s a new, it s a, it s a, it s, it s the third path.
0
SPEAKER_01
Are you saying array?
0
SPEAKER_00
Array oriented.
0
SPEAKER_01
What does it mean to be array-oriented?
0
SPEAKER_00
So array-oriented means that you generally don t use any loops, but the whole thing is done with kind of an extreme version of broadcasting, if you re familiar with that NumPy slash Python concept.
0
SPEAKER_00
You do a lot with one line of code.
0
SPEAKER_00
It looks a lot like math notation, highly compact.
0
SPEAKER_00
And the idea is that you can kind of, because you can do so much with one line of code, a single screen of code is very unlikely to, you very rarely need more than that to express your program.
0
SPEAKER_00
And so you can kind of keep it all in your head and you can kind of clearly communicate it.
0
SPEAKER_00
It s interesting that APL created two main branches, K and J.
0
SPEAKER_00
J is this kind of like open source niche community of crazy enthusiasts like me.
0
SPEAKER_00
And then the other path, K, is fascinating.
0
SPEAKER_00
It s an astonishingly expensive programming language which many of the world s most ludicrously rich hedge funds use.
0
SPEAKER_00
So the entire K machine is so small it sits inside level 3 cache on your CPU and it easily wins every benchmark I ve ever seen in terms of data processing speed.
0
SPEAKER_00
But you don t come across it very much because it s like $100,000 per CPU to
0
SPEAKER_00
to run it.
0
SPEAKER_00
But it s like this, this, this, this path of programming languages is just so much, I don t know, so much more powerful in every way than the ones that almost anybody uses every day.
0
SPEAKER_01
So it s all, it s all about computation.
0
SPEAKER_00
It s pretty heavily focused on computation.
0
SPEAKER_00
I mean, so much of programming is data processing by definition.
0
SPEAKER_00
And so there s a lot of things you can,
0
SPEAKER_00
do with it.
0
SPEAKER_00
But yeah, there s not much work being done on making like user interface toolkits or whatever.
0
SPEAKER_00
I mean, there s some, but they re not great.
0
SPEAKER_01
At the same time, you ve done a lot of stuff with Perl and Python.
0
SPEAKER_00
Yeah.
0
SPEAKER_01
So where does that fit into the picture of J and K and APL?
0
SPEAKER_00
Well, you know, it s just much more pragmatic.
0
SPEAKER_00
Like in the end, you kind of have to end up where the
0
SPEAKER_00
Where the libraries are, you know, like it s a meme.
0
SPEAKER_00
I m my focus is on Productivity.
0
SPEAKER_00
I just want to get stuff done and solve problems.
0
SPEAKER_00
So Pal was great.
0
SPEAKER_00
I created an email company called fast mail and pal was Great because back in the late 90s early 2000s it just
0
SPEAKER_00
Had a lot of stuff it could do.
0
SPEAKER_00
I still had to write my own Monitoring system and my own web framework my own whatever because like none of that stuff existed, but It was a super flexible language to do that in and you used pearl For fast mail you used it as a back-end.
0
SPEAKER_01
Thank you.
0
SPEAKER_01
So everything was written in pearl.
0
SPEAKER_00
Yep.
0
SPEAKER_00
Yeah everything Everything was pearl.
0
SPEAKER_01
Why do you think pearl hasn t?
0
SPEAKER_01
succeeded or hasn t dominated the market where Python really takes over a lot.
0
SPEAKER_00
Yeah.
0
SPEAKER_00
Well, I mean, Perl did dominate.
0
SPEAKER_00
It was everything, everywhere.
0
SPEAKER_00
But then the guy that ran Perl, Larry Wall, kind of just didn t put the time in anymore.
0
SPEAKER_00
And
0
SPEAKER_00
no project can be successful if there isn t, you know, particularly one that started with a strong leader that loses that strong leadership.
0
SPEAKER_00
So then Python has kind of replaced it.
0
SPEAKER_00
You know, Python is a lot less elegant language in nearly every way, but it has the data science libraries and a lot of them are pretty great.
0
SPEAKER_00
So I kind of use it
0
SPEAKER_00
because it s the best we have, but it s definitely not good enough.
0
SPEAKER_01
But what do you think the future of programming looks like?
0
SPEAKER_01
What do you hope the future of programming looks like if we zoom in on the computational fields, on data science, on machine learning?
0
SPEAKER_00
I hope Swift is successful.
0
SPEAKER_00
Because the goal of Swift, the way Chris Latner describes it, is to be infinitely hackable.
0
SPEAKER_00
And that s what I want.
0
SPEAKER_00
I want something where me and the people I do research with and my students can look at and change everything from top to bottom.
0
SPEAKER_00
There s nothing mysterious and magical and inaccessible.
0
SPEAKER_00
Unfortunately with Python, it s the opposite of that.
0
SPEAKER_00
Because Python is so slow, it s extremely unhackable.
0
SPEAKER_00
You get to a point where it s like, okay, from here on down it s C. So your debugger doesn t work in the same way, your profiler doesn t work in the same way, your build system doesn t work in the same way.
0
SPEAKER_00
It s really not very hackable.
0
SPEAKER_01
What s the part you would like to be hackable?
0
SPEAKER_01
Is it for the objective of optimizing training of neural networks, inference of neural networks?
0
SPEAKER_01
Is it performance of the system?
0
SPEAKER_01
Or is there some non-performance related just... It s everything.
0
SPEAKER_00
I mean, in the end, I want to be productive as a practitioner.
0
SPEAKER_00
So that means that, so like at the moment our understanding of deep learning is incredibly primitive, there s very little we understand, most things don t work very well, even though it works better than anything else out there.
0
SPEAKER_00
There s so many opportunities to make it better.
0
SPEAKER_00
So you look at any domain area, like
0
SPEAKER_00
I don t know, speech recognition with deep learning or natural language processing classification with deep learning or whatever.
0
SPEAKER_00
Every time I look at an area with deep learning, I always see like, oh, it s terrible.
0
SPEAKER_00
There s lots and lots of obviously stupid ways to do things that need to be fixed.
0
SPEAKER_00
So then I want to be able to jump in there and quickly experiment and make them better.
0
SPEAKER_01
Do you think the programming language has a role in that?
0
SPEAKER_01
Huge role.
0
SPEAKER_00
Yeah.
0
SPEAKER_00
So currently Python has a big gap in terms of our ability to innovate, particularly around recurrent neural networks and natural language processing.
0
SPEAKER_00
Because it s so slow, the actual loop where we actually loop through words, we have to do that whole thing in CUDA C. So we actually can t innovate with the kernel, the heart of that most important algorithm.
0
SPEAKER_00
And it s just a huge problem.
0
SPEAKER_00
And this happens all over the place.
0
SPEAKER_00
So we hit, you know, research limitations.
0
SPEAKER_00
Another example, convolutional neural networks, which are actually the most popular architecture for lots of things, maybe most things in deep learning.
0
SPEAKER_00
We almost certainly should be using sparse convolutional neural networks.
0
SPEAKER_00
But only, like, two people are.
0
SPEAKER_00
Because to do it, you have to rewrite all of that courtesy level stuff.
0
SPEAKER_00
And yeah, just researchers and practitioners don t.
0
SPEAKER_00
So, like, there s just big gaps in, like, what people actually research on, what people actually implement because of the programming language problem.
0
SPEAKER_01
So you think it s just too difficult to write in Kudasi that a higher level programming language like Swift should enable the easier fooling around creative stuff with RNNs or with parsed convolutional networks?
0
SPEAKER_00
Kind of.
1
SPEAKER_01
Who s at fault?
1
SPEAKER_01
Who s, who s in charge of making it easy for a researcher to play around?
1
SPEAKER_00
I mean, no one s at fault.
1
SPEAKER_00
It s just nobody s got around to it yet.
1
SPEAKER_00
Or it s just, it s hard.
1
SPEAKER_00
Right.
1
SPEAKER_00
And I mean, part of the fault is that we ignored that whole APL kind of direction.
1
SPEAKER_00
Most or nearly everybody did for 60 years, 50 years.
1
SPEAKER_00
But recently people have been starting to reinvent pieces of that and kind of create some interesting new directions in the compiler technology.
1
SPEAKER_00
So the place where that s particularly happening right now is something called MLIR, which is something that, again, Chris Latner, the Swift guy, is leading.
1
SPEAKER_00
Because it s actually not going to be Swift on its own that solves this problem.
1
SPEAKER_00
Because the problem is that currently writing an acceptably fast GPU program is too complicated regardless of what language you use.
1
SPEAKER_00
And that s just because if you have to deal with the fact that I ve got, you know, 10,000 threads and I have to synchronize between them all and I have to put my thing into grid blocks and think about warps and all this stuff, it s just, it s just so much boilerplate that to do that well you have to be a specialist at that and it s going to be a year s work to
1
SPEAKER_00
you know, optimize that algorithm in that way.
1
SPEAKER_00
But with things like tensor comprehensions and TILE and MLIR and TVM, there s all these various projects which are all about saying, let s let people create domain-specific languages for tensor computations.
1
SPEAKER_00
These are the kinds of things we do generally on the GPU for deep learning.
1
SPEAKER_00
and then have a compiler which can optimize that tensor computation.
1
SPEAKER_00
A lot of this work is actually sitting on top of a project called Halide, which is a mind-blowing project where they came up with such a domain-specific language.
1
SPEAKER_00
In fact, two.
1
SPEAKER_00
One domain-specific language for expressing this is what my tensor computation is.
1
SPEAKER_00
And another domain-specific language for expressing, this is the kind of the way I want you to structure the compilation of that, like do it block by block and do these bits in parallel.
1
SPEAKER_00
And they were able to show how you can compress the amount of code by 10x compared to optimized GPU code and get the same performance.
1
SPEAKER_00
So that s like, so these other things are kind of sitting on top of that kind of research and MLIR is pulling a lot of those best practices together.
1
SPEAKER_00
And now we re starting to see work done on making all of that directly accessible through Swift so that I could use Swift to kind of write those domain specific.
1
SPEAKER_00
Languages and hopefully we ll get then Swift CUDA kernels written in a very expressive and concise way that looks a bit like J in APL and then Swift layers on top of that and then a Swift UI on top of that and You know, it ll be so nice if we can get to that point
1
SPEAKER_01
Does it all eventually boil down to CUDA and NVIDIA GPUs?
1
SPEAKER_00
Unfortunately, at the moment it does.
1
SPEAKER_00
But one of the nice things about MLIR, if AMD ever gets their act together, which they probably won t, is that they or others could write MLIR backends for other GPUs or other tensor computation devices.
1
SPEAKER_00
of which today there are increasing number like Graphcore or Vertex AI or whatever.
1
SPEAKER_00
So yeah, being able to target lots of backends would be another benefit of this and the market really needs competition because at the moment NVIDIA is massively overcharging for their kind of enterprise class cards because there is no serious competition because nobody else is doing the software properly.
1
SPEAKER_01
In the cloud, there is some competition, right?
1
SPEAKER_01
But not really.
1
SPEAKER_00
Other than TPUs, perhaps.
1
SPEAKER_00
TPUs are almost unprogrammable at the moment.
1
SPEAKER_01
The TPUs has the same problem.
1
SPEAKER_00
It s even worse.
1
SPEAKER_00
So TPUs, Google actually made an explicit decision to make them almost entirely unprogrammable because they felt that there was too much IP in there.
1
SPEAKER_00
And if they gave people direct access to program them, people would learn their secrets.
1
SPEAKER_00
So you can t actually directly
1
SPEAKER_00
program the memory in a TPU.
1
SPEAKER_00
You can t even directly like create code that runs on and that you look at on the machine that has the GPU.
1
SPEAKER_00
It all goes through a virtual machine.
1
SPEAKER_00
So all you can really do is this kind of cookie cutter thing of like plug in high level stuff together, which is just super tedious and annoying and totally unnecessary.
1
SPEAKER_01
So what was the, tell me if you could, the origin story of fast AI.
1
SPEAKER_01
what is the motivation, its mission, its dream?
1
SPEAKER_00
So I guess the founding story is heavily tied to my previous startup, which is a company called Inletic, which was the first company to focus on deep learning for medicine.
1
SPEAKER_00
And I created that because I saw there was a huge opportunity to, there s about a 10X shortage of the number of doctors in the world, in the developing world that we need.
1
SPEAKER_00
Expected it would take about 300 years to train enough doctors to meet that gap.
1
SPEAKER_00
But I guess that maybe if we used deep learning for some of the analytics, we could maybe make it so you don t need as highly trained doctors.
1
SPEAKER_00
For diagnosis?
1
SPEAKER_00
For diagnosis and treatment planning.
1
SPEAKER_01
Where s the biggest benefit, just before we get to fast AI, where s the biggest benefit of AI in medicine that you see today?
1
SPEAKER_00
Not much happening today in terms of like stuff that s actually out there.
1
SPEAKER_00
It s very early.
1
SPEAKER_00
But in terms of the opportunity, it s to take markets like India and China and Indonesia, which have big populations, Africa, small numbers of doctors.
1
SPEAKER_00
and provide diagnostic, particularly treatment planning and triage kind of on device so that if you do a, you know, test for malaria or tuberculosis or whatever, you immediately get something that even a healthcare worker that s had a month of training can get a very high quality assessment of whether the patient might be at risk and tell, you know, okay, we ll send them off to a hospital.
1
SPEAKER_00
So, for example, in Africa, outside of South Africa, there s only five pediatric radiologists for the entire continent.
1
SPEAKER_00
So most countries don t have any.
1
SPEAKER_00
So if your kid is sick and they need something diagnosed through medical imaging, the person, even if you re able to get medical imaging done, the person that looks at it will be, you know, a nurse at best.
1
SPEAKER_00
But actually in India, for example, and China, almost no x-rays are read by anybody, by any trained professional, because they don t have enough.
1
SPEAKER_00
So if instead we had an algorithm that could take the most likely high-risk 5% and say, triage basically, say, okay, someone needs to look at this.
1
SPEAKER_00
It would massively Change the kind of way that what s possible with medicine in the developing world and remember they have Increasingly they have money.
1
SPEAKER_00
They re the developing world.
1
SPEAKER_00
They re not the poor world.
1
SPEAKER_00
They re developing well, so they have the money so they re building the hospitals they re getting the Diagnostic equipment, but they just there s no way for a very long time.
1
SPEAKER_00
Will they be able to?
1
SPEAKER_00
have the expertise
1
SPEAKER_01
shortage of expertise, okay, and that s where the deep learning systems can step in and magnify the expertise they do have, essentially.
1
SPEAKER_01
So you do see, just to linger it a little bit longer, the interaction, do you still see the human experts still at the core of these systems?
1
SPEAKER_01
Is there something in medicine that can be automated almost completely?
1
SPEAKER_00
I don t see the point of even thinking about that, because we have such a shortage of people.
1
SPEAKER_00
Why would we want to find a way not to use them?
1
SPEAKER_00
We have people.
1
SPEAKER_00
So the idea of, even from an economic point of view, if you can make them 10x more productive,
1
SPEAKER_00
Getting rid of the person doesn t impact your unit economics at all.
1
SPEAKER_00
And it totally ignores the fact that there are things people do better than machines.
1
SPEAKER_00
So it s just to me, that s not a useful way of framing the problem.
1
SPEAKER_01
I guess, just to clarify, I guess I meant there may be some problems.
1
SPEAKER_01
where you can avoid even going to the expert ever, sort of maybe preventative care or some basic stuff, the low-hanging fruit, allowing the expert to focus on the things that are really that.
1
SPEAKER_00
Well, that s what the triage would do, right?
1
SPEAKER_00
So the triage would say, okay, it s 99% sure there s nothing here.
1
SPEAKER_00
So that can be done on device and they can just say, okay, go home.
1
SPEAKER_00
So the experts are being used to look at the stuff which has some chance it s worth looking at, which most things it s not, you know, it s fine.
1
SPEAKER_01
Why do you think we haven t quite made progress on that yet in terms of the scale of
1
SPEAKER_01
how much AI is applied in the medical field?
1
SPEAKER_00
Oh, there s a lot of reasons.
1
SPEAKER_00
I mean, one is it s pretty new.
1
SPEAKER_00
I only started in Lytic in like 2014.
1
SPEAKER_00
And before that, like, it s hard to express to what degree the medical world was not aware of the opportunities here.
1
SPEAKER_00
So I went to RSNA, which is the world s largest radiology conference.
1
SPEAKER_00
And I told everybody I could, you know, like, I m doing this thing with deep learning, please come and check it out.
1
SPEAKER_00
And no one had any idea what I was talking about and no one had any interest in it.
1
SPEAKER_00
So like we ve come from absolute zero, which is hard.
1
SPEAKER_00
And then the whole regulatory framework, education system, everything is just set up to think of doctoring in a very different way.
1
SPEAKER_00
So today there is a small number of people who are
1
SPEAKER_00
deep learning practitioners and doctors at the same time.
1
SPEAKER_00
And we re starting to see the first ones come out of their Ph.D.
1
SPEAKER_00
programs.
1
SPEAKER_00
Zach Kahane over in Boston, Cambridge, has a number of students now who are data
1
SPEAKER_00
data science experts, deep learning experts, and actual medical doctors.
1
SPEAKER_00
Quite a few doctors have completed our fast AI course now and are publishing papers and creating journal reading groups in the American Council of Radiology.
1
SPEAKER_00
And it s just starting to happen.
1
SPEAKER_00
But it s going to be a long process.
1
SPEAKER_00
The regulators have to learn how to regulate this.
1
SPEAKER_00
They have to build
1
SPEAKER_00
guidelines, and then the lawyers at hospitals have to develop a new way of understanding that sometimes it makes sense for data to be
1
SPEAKER_00
you know, looked at in raw form in large quantities in order to create world-changing results.
1
SPEAKER_01
Yeah, so regulation around data, all that, it sounds probably the hardest problem, but sounds reminiscent of autonomous vehicles as well.
1
SPEAKER_01
Many of the same regulatory challenges, many of the same data challenges.
1
SPEAKER_00
Yeah, I mean, funnily enough, the problem is less the regulation and more the interpretation of that regulation by lawyers in hospitals.
1
SPEAKER_00
So HIPAA is actually,
1
SPEAKER_00
The P in HIPAA does not stand for privacy.
1
SPEAKER_00
It stands for portability.
1
SPEAKER_00
It s actually meant to be a way that data can be used.
1
SPEAKER_00
And it was created with lots of gray areas because the idea is that would be more practical and would help people to use this
1
SPEAKER_00
this legislation to actually share data in a more thoughtful way.
1
SPEAKER_00
Unfortunately, it s done the opposite, because when a lawyer sees a gray area, they say, oh, if we don t know, we won t get sued, then we can t do it.
1
SPEAKER_00
Right.
1
SPEAKER_00
Not exactly the problem.
1
SPEAKER_00
The problem is more that hospital lawyers are not incented to make bold decisions about data portability.
1
SPEAKER_01
Or even to embrace technology that saves lives.
1
SPEAKER_01
They more want to not get in trouble for embracing that technology.
1
SPEAKER_00
It also saves lives in a very abstract way, which is like, oh, we ve been able to release these 100,000 anonymized records.
1
SPEAKER_00
I can t point at the specific person whose life that saved.
1
SPEAKER_00
I can say like, oh, we ended up with this paper, which found this result, which, you know, diagnosed a thousand more people than we would have otherwise.
1
SPEAKER_00
But it s like, which ones were helped?
1
SPEAKER_00
It s very abstract.
1
SPEAKER_01
And on the counter side of that, you may be able to point to a life that was taken because of something that was... Yeah, or a person whose privacy was violated.
1
SPEAKER_00
It s like, oh, this specific person, you know, was de-identified.
1
SPEAKER_01
So just a fascinating topic, we re jumping around, we ll get back to fast AI, but on the question of privacy, data is the fuel for so much innovation in deep learning.
1
SPEAKER_01
What s your sense on privacy, whether we re talking about Twitter, Facebook, YouTube, just the technologies like in the medical field that rely on people s data in order to create impact.
1
SPEAKER_01
How do we get that right?
1
SPEAKER_01
Respecting people s privacy and yet creating technology that learns from data.
1
SPEAKER_00
One of my areas of focus is on doing more with less data.
1
SPEAKER_00
Um, which so most vendors unfortunately are strongly incented to find ways to require more data and more computation.
1
SPEAKER_00
So Google and IBM being the most obvious, uh, IBM.
1
SPEAKER_00
Yeah.
1
SPEAKER_00
So Watson, you know, so Google and IBM both strongly push the idea that you have to be, you know, that they have more data and more computation and more intelligent people than anybody else.
1
SPEAKER_00
And so you have to trust them to do things cause nobody else can do it.
1
SPEAKER_00
And Google s very upfront about this like Jeff Dean has gone out there and given talks and said our goal is to require a thousand times more computation but less people our goal is
1
SPEAKER_00
is to use the people that you have better and the data you have better and the computation you have better.
1
SPEAKER_00
So one of the things that we ve discovered is, or at least highlighted, is that you very, very, very often don t need much data at all.
1
SPEAKER_00
And so the data you already have in your organization will be enough to get state-of-the-art results.
1
SPEAKER_00
So like my starting point would be to kind of say around privacy is
1
SPEAKER_00
A lot of people are looking for ways to share data and aggregate data, but I think often that s unnecessary.
1
SPEAKER_00
They assume that they need more data than they do because they re not familiar with the basics of transfer learning, which is this critical technique for needing orders of magnitude less data.
1
SPEAKER_01
Is your sense, one reason you might want to collect data from everyone is like in the recommender system context, where your individual, Jeremy Howard s individual data is the most useful for providing a product that s impactful for you.
1
SPEAKER_01
So for giving you advertisements, for recommending to you movies, for doing medical diagnosis.
1
SPEAKER_01
Is your sense we can build with a small amount of data
1
SPEAKER_01
general models that will have a huge impact for most people that we don t need to have data from each individual?
1
SPEAKER_00
On the whole, I d say yes.
1
SPEAKER_00
I mean, there are things like
1
SPEAKER_00
You know, recommender systems have this cold start problem where, you know, Jeremy is a new customer.
1
SPEAKER_00
We haven t seen him before, so we can t recommend him things based on what else he s bought and liked with us.
1
SPEAKER_00
And there s various workarounds to that.
1
SPEAKER_00
Like in a lot of music programs, we ll start out by saying, which of these artists do you like?
1
SPEAKER_00
Which of these albums do you like?
1
SPEAKER_00
Which of these songs do you like?
1
SPEAKER_00
Netflix used to do that.
1
SPEAKER_00
Nowadays, they tend not to.
1
SPEAKER_00
People kind of don t like that because they think, oh, we don t want to bother the user.
1
SPEAKER_00
So you could work around that by having some kind of data sharing where you get my marketing record from Axiom or whatever and try to guestion that.
1
SPEAKER_00
To me, the benefit to me and to society of
1
SPEAKER_00
saving me five minutes on answering some questions versus the negative externalities of the privacy issue doesn t add up.
1
SPEAKER_00
So I think like a lot of the time the places where people are invading our privacy in order to provide convenience is
1
SPEAKER_00
Really about just trying to make them more money and and They move these negative externalities and to places that they don t have to pay for them so when you actually see
1
SPEAKER_00
Regulations appear that actually cause the companies that create these negative externalities to have to pay for it themselves.
1
SPEAKER_00
They say well we can t Do it anymore.
1
SPEAKER_00
So the cost is actually too high right but for something like medicine yeah, I mean the hospital has my you know medical imaging my pathology studies my medical records and
1
SPEAKER_00
Um, and also I own my medical data.
1
SPEAKER_00
So you can, um, so I, I help a startup called doc.ai.
1
SPEAKER_00
One of the things doc.ai does is that there s, there s an app that you can connect to, you know, Sutter Health and LabCorp and Walgreens and download your medical data to your phone.
1
SPEAKER_00
And then.
2
SPEAKER_00
Upload it again at your discretion to share it as you wish So with that kind of approach we can share our medical information with the people we want to
2
SPEAKER_01
Yeah, so control.
2
SPEAKER_01
I mean, really being able to control who you share with and so on.
2
SPEAKER_00
Yeah.
2
SPEAKER_01
So that has a beautiful, interesting tangent.
2
SPEAKER_01
But to return back to the origin story of Fast.ai.
2
SPEAKER_00
Right.
2
SPEAKER_00
So before I started Fast.ai, I spent a year researching where are the biggest opportunities for
2
SPEAKER_00
deep learning because I knew from my time at Kaggle in particular that deep learning had kind of hit this threshold point where it was rapidly becoming the state-of-the-art approach in every area that looked at it.
2
SPEAKER_00
And I d been working with neural nets for over 20 years.
2
SPEAKER_00
I knew that from a theoretical point of view, once it hit that point, it would do that in kind of just about every
2
SPEAKER_00
domain.
2
SPEAKER_00
And so I kind of spent a year researching what are the domains that s going to have the biggest low-hanging fruit in the shortest time period.
2
SPEAKER_00
I picked medicine, but there were so many I could have picked.
2
SPEAKER_00
And so there was a kind of level of frustration for me of like, okay, I m really glad we ve opened up the medical deep learning world and today it s huge, as you know, but we can t do
2
SPEAKER_00
You know, I can t do everything.
2
SPEAKER_00
I don t even know, like, it took, like in medicine, it took me a really long time to even get a sense of, like, what kind of problems do medical practitioners solve?
2
SPEAKER_00
What kind of data do they have?
2
SPEAKER_00
Who has that data?
2
SPEAKER_00
So I kind of felt like I need to approach this differently if I want to maximize the positive impact of deep learning, rather than
2
SPEAKER_00
me picking an area and trying to become good at it and building something, I should let people who are already domain experts in those areas and who already have the data do it themselves.
2
SPEAKER_00
So that was the reason for fast AI is to basically try and figure out how to get deep learning into the hands of people who could benefit from it and help them to do so in as quick and easy and effective a way as possible.
2
SPEAKER_01
Got it.
2
SPEAKER_01
So sort of empower the domain experts.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
And like partly it s because like, unlike most people in this field, my background is very applied and industrial.
2
SPEAKER_00
Like my first job was at McKinsey and Company.
2
SPEAKER_00
I spent 10 years in management consulting.
2
SPEAKER_00
I spend a lot of time with domain experts, you know, so I kind of respect them and appreciate them.
2
SPEAKER_00
And I know that s where the value generation in society is.
2
SPEAKER_00
And so I also know how most of them can t code and most of them don t have the time to invest, you know, three years in a graduate degree or whatever.
2
SPEAKER_00
So it s like, how do I
2
SPEAKER_00
upskill those domain experts.
2
SPEAKER_00
I think that would be a super powerful thing.
2
SPEAKER_00
You know, the biggest societal impact I could have.
2
SPEAKER_00
So yeah, that was the thinking.
2
SPEAKER_01
So much of fast AI students and researchers and the things you teach are pragmatically minded, practically minded, figuring out ways how to solve real problems and fast.
2
SPEAKER_01
So from your experience, what s the difference between theory and practice of deep learning?
2
SPEAKER_00
Well,
2
SPEAKER_00
Most of the research in the deep learning world is a total waste of time.
2
SPEAKER_00
Right, that s what I was getting at.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
It s a problem in science in general.
2
SPEAKER_00
Scientists need to be published, which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area.
2
SPEAKER_00
So that means that they all need to work on the same thing.
2
SPEAKER_00
And so it really, and the thing they work on, there s nothing to encourage them to work on things that are practically useful.
2
SPEAKER_00
So you get just a whole lot of research which is minor advances and stuff that s been very highly studied and has no significant practical impact.
2
SPEAKER_00
Whereas the things that really make a difference, like I mentioned transfer learning, like if we can do better at transfer learning, then it s this like world-changing thing where suddenly like lots more people can do world-class work with less resources and less data.
2
SPEAKER_00
But almost nobody works on that.
2
SPEAKER_00
Or another example, active learning, which is the study of like how do we get more out of the human beings in the loop.
2
SPEAKER_01
That s my favorite topic.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
So active learning is great, but it s almost nobody working on it because it s just not a trendy thing right now.
2
SPEAKER_00
You know what?
2
SPEAKER_01
Somebody started to interrupt.
2
SPEAKER_01
He was saying that nobody is publishing on active learning, but there s people inside companies, anybody who actually has to solve a problem, they re going to innovate on active learning.
2
SPEAKER_00
Yeah, everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think, gosh, this is taking a long time and it s very expensive.
2
SPEAKER_00
And then they start thinking, well, why am I labeling everything?
2
SPEAKER_00
I m only, the machine s only making mistakes on those two classes.
2
SPEAKER_00
They re the hard ones.
2
SPEAKER_00
Maybe I ll just start labeling those two classes.
2
SPEAKER_00
And then you start thinking,
2
SPEAKER_00
But why did I do that manually?
2
SPEAKER_00
Why can t I just get the system to tell me which things are going to be hardest?
2
SPEAKER_00
It s an obvious thing to do, but yeah, it s just like transfer learning.
2
SPEAKER_00
It s understudied, and the academic world just has no reason to care about practical results.
2
SPEAKER_00
The funny thing is, like, I ve only really ever written one paper.
2
SPEAKER_00
I hate writing papers.
2
SPEAKER_00
And I didn t even write it.
2
SPEAKER_00
It was my colleague, Sebastian Ruder, who actually wrote it.
2
SPEAKER_00
I just did the research for it.
2
SPEAKER_00
But it was basically introducing transfer learning, successful transfer learning, to NLP for the first time.
2
SPEAKER_00
The algorithm is called ULMfit.
2
SPEAKER_00
It actually, I actually wrote it for the course, for the fast AI course.
2
SPEAKER_00
I wanted to teach people NLP and I thought I only want to teach people practical stuff.
2
SPEAKER_00
And I think the only practical stuff is transfer learning.
2
SPEAKER_00
And I couldn t find any examples of transfer learning in NLP.
2
SPEAKER_00
So I just did it.
2
SPEAKER_00
And I was shocked to find that as soon as I did it, which, you know, the basic prototype took a couple of days, it smashed the state of the art on one of the most important data sets in a field that I knew nothing about.
2
SPEAKER_00
And I just thought, well, this is,
2
SPEAKER_00
Ridiculous.
2
SPEAKER_00
And so I spoke to Sebastian about it and he kindly offered to write it up, the results.
2
SPEAKER_00
And so it ended up being published in ACL, which is the top computational linguistics conference.
2
SPEAKER_00
So like people do actually care once you do it, but I guess it s difficult for maybe like
2
SPEAKER_00
junior researchers or like Like I don t care whether I get citations or papers or whatever.
2
SPEAKER_00
I don t there s nothing in my life that makes that important Which is why I ve never actually bothered to write a paper myself now for people who do I guess they have to pick the kind of safe option which is like yeah, make a slight improvement on something that everybody s already working on and
2
SPEAKER_01
Yeah, nobody does anything interesting or succeeds in life with the safe option.
2
SPEAKER_00
Although, I mean, the nice thing is nowadays everybody is now working on NLP transfer learning, because since that time we ve had GPT and GPT-2 and BERT and, you know, it s like, it s, so yeah, once you show that something s possible, everybody jumps in, I guess, so.
2
SPEAKER_01
I hope to be a part of, and I hope to see more innovation in active learning in the same way.
2
SPEAKER_01
I think transfer learning and active learning are fascinating public open work.
2
SPEAKER_00
I actually helped start a startup called Platform AI, which is really all about active learning.
2
SPEAKER_00
And yeah, it s been interesting trying to kind of see what research is out there and make the most of it.
2
SPEAKER_00
And there s basically none.
2
SPEAKER_00
So we ve had to do all our own research.
2
SPEAKER_01
Once again, just as you described.
2
SPEAKER_01
Can you tell the story of the Stanford competition DawnBench and FastAI s achievement on it?
2
SPEAKER_00
Sure.
2
SPEAKER_00
So something which I really enjoy is that I basically teach two courses a year, the practical deep learning for coders, which is kind of the introductory course, and then cutting edge deep learning for coders, which is the kind of research level course.
2
SPEAKER_00
And while I teach those courses, I basically have a big office at the University of San Francisco, big enough for like 30 people.
2
SPEAKER_00
And I invite anybody, any student who wants to come and hang out with me while I build the course.
2
SPEAKER_00
And so generally it s full.
2
SPEAKER_00
And so we have 20 or 30 people in a big office with nothing to do but study deep learning.
2
SPEAKER_00
So it was during one of these times that somebody in the group said, oh, there s a thing called DawnBench that looks interesting.
2
SPEAKER_00
And I was like, what the hell is that?
2
SPEAKER_00
And they set out some competition to see how quickly you can train a model.
2
SPEAKER_00
Seems kind of
2
SPEAKER_00
Not exactly relevant to what we re doing, but it sounds like the kind of thing which you might be interested in.
2
SPEAKER_00
I checked it out and I was like, oh crap, there s only 10 days till it s over.
2
SPEAKER_00
It s too late.
2
SPEAKER_00
And we re kind of busy trying to teach this course.
2
SPEAKER_00
But we re like, it would make an interesting...
2
SPEAKER_00
Case study for the course like it s all the stuff we re already doing.
2
SPEAKER_00
Why don t we just put together our current best practices and ideas?
2
SPEAKER_00
So me and I guess about four students Just decided to give it a go and we focused on this small one called sci-fi 10, which is little 32 by 32 pixel images and
2
SPEAKER_00
Can you say what dimensions yeah, so it s a competition to train a model as fast as possible It was run by Stanford and as cheap as possible to that s also another one for as cheap as possible And there s a couple of categories Image net and so far ten so image nets is big 1.3 million
2
SPEAKER_00
image thing that took a couple of days to train.
2
SPEAKER_00
I remember a friend of mine, Pete Warden, who s now at Google, I remember he told me how he trained ImageNet a few years ago and he basically like had this
2
SPEAKER_00
little granny flat out the back that he turned into his ImageNet training center.
2
SPEAKER_00
And he figured, you know, after like a year of work, he figured out how to train it in like 10 days or something.
2
SPEAKER_00
It s like, that was a big job.
2
SPEAKER_00
Well, Cypher 10 at that time, you could train in a few hours, you know, it s much smaller and easier.
2
SPEAKER_00
So we thought we d try Cypher 10.
2
SPEAKER_00
And yeah, I d really never done that before.
2
SPEAKER_00
Like I d never really, like things like using more than one GPU at a time was something I tried to avoid.
2
SPEAKER_00
Cause to me, it s like very against the whole idea of accessibility is, you should better do things with one GPU.
2
SPEAKER_01
I mean, have you asked in the past before, after having accomplished something, how do I do this faster, much faster?
2
SPEAKER_00
Oh, always.
2
SPEAKER_00
But it s always, for me, it s always, how do I make it much faster?
2
SPEAKER_00
on a single GPU that a normal person could afford in their day-to-day life.
2
SPEAKER_00
It s not, how could I do it faster by, you know, having a huge data center?
2
SPEAKER_00
Because to me, it s all about, like, as many people should be able to use something as possible without fussing around with infrastructure.
2
SPEAKER_00
So anyway, so in this case, it s like, well, we can use eight GPUs just by renting a AWS machine.
2
SPEAKER_00
So we thought we d try that.
2
SPEAKER_00
And yeah, basically using the stuff we were already doing, we were able to get, you know, the speed, you know, within a few days, we had the speed down to
2
SPEAKER_00
A very small number of minutes.
2
SPEAKER_00
I can t remember exactly how many minutes it was but it might have been like 10 minutes or something And so yeah, we found ourselves at the top of the leaderboard easily for both time and money which really shocked me because the other people competing this were like Google and Intel and stuff were like Know a lot more about this stuff than I think we do.
2
SPEAKER_00
So then we were emboldened we thought oh
2
SPEAKER_00
Let s try the image net one too.
2
SPEAKER_00
I mean, it seemed way out of our league, but our goal was to get under 12 hours.
2
SPEAKER_00
Um, and we did, which was really exciting.
2
SPEAKER_00
And, but we didn t put anything up on the leaderboard, but we were down to like 10 hours.
2
SPEAKER_00
But then, uh, Google put in some, uh, uh, like five hours or something.
2
SPEAKER_00
And we re just like,
2
SPEAKER_00
We re so screwed.
2
SPEAKER_00
But we kind of thought, we ll keep trying, you know, if Google can do it.
2
SPEAKER_00
I mean, Google did on five hours on some, on like a TPU pod or something, like a lot of hardware.
2
SPEAKER_00
But we kind of like had a bunch of ideas to try, like a really simple thing was why are we using these big images?
2
SPEAKER_00
They re like 224 or 256 by 256 pixels.
2
SPEAKER_00
You know, why don t we try smaller ones?
2
SPEAKER_01
And just to elaborate, there s a constraint on the accuracy that your trained model is supposed to achieve.
2
SPEAKER_00
Yeah, you ve got to achieve 93%, I think it was, for ImageNet.
2
SPEAKER_00
Exactly.
2
SPEAKER_00
Which is very tough, so you have to... Yeah, 93%, like they picked a good threshold.
2
SPEAKER_00
It was a little bit higher than what the most commonly used ResNet-50 model could achieve at that time.
2
SPEAKER_00
So, yeah, so it s quite a difficult problem to solve.
2
SPEAKER_00
But yeah, we realized if we actually just use 64x64 images,
2
SPEAKER_00
It trained a pretty good model.
2
SPEAKER_00
And then we could take that same model and just give it a couple of epochs to learn 224 by 224 images.
2
SPEAKER_00
And it was basically already trained, which makes a lot of sense.
2
SPEAKER_00
Like if you teach somebody, like, here s what a dog looks like, and you show them low res versions.
2
SPEAKER_00
And then you say, here s a really clear picture of a dog.
2
SPEAKER_00
They already know what a dog looks like.
2
SPEAKER_00
So that, like, just, we, we, we jumped to the front and we ended up winning, uh, uh,
2
SPEAKER_00
parts of that competition.
2
SPEAKER_00
We actually ended up doing a distributed version over multiple machines a couple of months later and ended up at the top of the leaderboard.
2
SPEAKER_00
We had 18 minutes.
2
SPEAKER_01
Yeah.
2
SPEAKER_00
And it was, uh, and people have just kept on blasting through again and again since then.
2
SPEAKER_01
So, so what s your view on multi GPU or multiple machine training in general as, as a way to speed code up?
2
SPEAKER_00
I think it s largely a waste of time.
2
SPEAKER_00
Both multi-GPU on a single machine and... Yeah, particularly multi-machines, because it s just clunky.
2
SPEAKER_00
Multi-GPUs is less clunky than it used to be.
2
SPEAKER_00
But to me, anything that slows down your iteration speed is a waste of time.
2
SPEAKER_00
So you could maybe do your very last, you know, perfecting of the model on multi-GPUs if you need to.
2
SPEAKER_00
But, so for example,
2
SPEAKER_00
I think doing stuff on ImageNet is generally a waste of time.
2
SPEAKER_00
Why test things on 1.3 million images?
2
SPEAKER_00
Most of us don t use 1.3 million images.
2
SPEAKER_00
And we ve also done research that shows that doing things on a smaller subset of images gives you the same relative answers anyway.
2
SPEAKER_00
So from a research point of view, why waste that time?
2
SPEAKER_00
So actually I released a couple of new datasets recently.
2
SPEAKER_00
One is called ImageNet, the French ImageNet, which is a small subset of ImageNet, which is designed to be easy to classify.
2
SPEAKER_01
What s, how do you spell image net?
2
SPEAKER_00
It s got an extra T and E at the end, because it s very French.
2
SPEAKER_00
And then another one called image wolf, which is a subset of image net that only contains dog breeds.
2
SPEAKER_00
And that s a hard one, right?
2
SPEAKER_00
That s a hard one.
2
SPEAKER_00
And I ve discovered that if you just look at these two subsets, you can train things
2
SPEAKER_00
on a single GPU in 10 minutes.
2
SPEAKER_00
And the results you get are directly transferable to ImageNet nearly all the time.
2
SPEAKER_00
And so now I m starting to see some researchers start to use these much smaller data sets.
2
SPEAKER_01
I so deeply love the way you think, because I think you might have written a blog post saying that sort of going to these big data sets is encouraging people to not think creatively.
2
SPEAKER_00
Absolutely.
2
SPEAKER_01
So you re too, it sort of constrains you to train on large resources.
2
SPEAKER_01
And because you have these resources, you think more resources will be better.
2
SPEAKER_01
And then you start, so like somehow you kill the creativity.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
And even worse than that, Lex, I keep hearing from people who say, I decided not to get into deep learning because I don t believe it s accessible to people outside of Google to do useful work.
2
SPEAKER_00
So like I see a lot of people make an explicit decision to not learn this incredibly valuable tool because they ve drunk the Google Kool-Aid, which is that only Google s big enough and smart enough to do it.
2
SPEAKER_00
And I just find that so disappointing and it s so wrong.
2
SPEAKER_01
And I think all of the major breakthroughs in AI in the next 20 years will be doable on a single GPU.
2
SPEAKER_01
Like I would say, my sense is all the big sort of... Well, let s put it this way.
2
SPEAKER_00
None of the big breakthroughs of the last 20 years have required multiple GPUs.
2
SPEAKER_00
So like BatchNorm, ReLU, Dropout... To demonstrate that there s something to them.
2
SPEAKER_00
Every one of them, none of them has required multiple GPUs.
2
SPEAKER_01
GANs, the original GANs, didn t require multiple GPUs.
2
SPEAKER_00
Well, and we ve actually recently shown that you don t even need GANs.
2
SPEAKER_00
So we ve developed GAN level outcomes without needing GANs.
2
SPEAKER_00
And we can now do it with, again, by using transfer learning, we can do it in a couple of hours on a single GPU.
2
SPEAKER_01
Just using a generated model, like without the adversarial part?
2
SPEAKER_00
Yeah, so we ve found loss functions that work super well without the adversarial part.
2
SPEAKER_00
And then one of our students, a guy called Jason Antich, has created a system called DeOldify, which uses this technique to colorize old black and white movies.
2
SPEAKER_00
You can do it on a single GPU, colorize a whole movie in a couple of hours.
2
SPEAKER_00
And one of the things that Jason and I did together was we figured out how to add a little bit of GAN at the very end, which it turns out for colorization makes it just a bit brighter and nicer.
2
SPEAKER_00
And then Jason did masses of experiments to figure out exactly how much to do, but it s still all done on his home machine on a single GPU in his lounge room.
2
SPEAKER_00
And if you think about colorizing Hollywood movies, that sounds like something a huge studio would have to do.
2
SPEAKER_00
But he has the world s best results on this.
3
SPEAKER_01
There s this problem of microphones.
3
SPEAKER_01
We re just talking about microphones now?
3
SPEAKER_00
Yeah.
3
SPEAKER_01
It s such a pain in the ass to have these microphones to get good quality audio.
3
SPEAKER_01
And I tried to see if it s possible to plop down a bunch of cheap sensors and reconstruct higher quality audio from multiple sources.
3
SPEAKER_01
Because right now, I haven t seen work from, okay, we can save inexpensive mics, automatically combining audio from multiple sources to improve the combined audio.
3
SPEAKER_01
People haven t done that.
3
SPEAKER_01
And that feels like a learning problem.
3
SPEAKER_01
So hopefully somebody can
3
SPEAKER_00
Well, I mean, it s, it s evidently doable and it should have been done by now.
3
SPEAKER_00
Um, I felt, I felt the same way about computational photography four years ago.
3
SPEAKER_00
That s right.
3
SPEAKER_00
Why are we investing in big lenses when three cheap lenses plus actually a little bit of, um, intentional movement.
3
SPEAKER_00
So like hold, you know, like take a few frames, gives you enough information to get excellent sub-pixel resolution, which particularly with deep learning, you would know exactly what you meant to be looking at.
3
SPEAKER_00
We can totally do the same thing with audio.
3
SPEAKER_00
I think it s madness that it hasn t been done yet.
3
SPEAKER_01
Has there been progress on the photography company?
3
SPEAKER_00
Yeah, photography is basically standard now.
3
SPEAKER_00
So the Google Pixel Night Light, I don t know if you ve ever tried it, but it s astonishing.
3
SPEAKER_00
You take a picture in almost pitch black and you get back a very high quality image.
3
SPEAKER_00
And it s not because of the lens.
3
SPEAKER_00
Same stuff with like adding the bokeh to the, you know, the background blurring done computationally.
3
SPEAKER_00
This is the picture right here.
3
SPEAKER_00
Yeah, basically everybody now is doing most of the fanciest stuff on their phones with computational photography and also increasingly people are putting more than one lens on the back of the camera.
3
SPEAKER_00
So the same will happen for audio for sure.
3
SPEAKER_01
And there s applications in the audio side.
3
SPEAKER_01
If you look at an Alexa type device, most people I ve seen, especially I worked at Google before, when you look at noise background removal, you don t think of multiple sources of audio.
3
SPEAKER_01
You don t play with that as much as I would hope people would.
3
SPEAKER_00
But I mean, you can still do it even with one.
3
SPEAKER_00
Like, again, not much work s been done in this area.
3
SPEAKER_00
So we re actually going to be releasing an audio library soon, which hopefully will encourage development of this because it s so underused.
3
SPEAKER_00
The basic approach we used for our super resolution, which Jason uses for DeOldify of generating high quality images, the exact same approach would work for audio.
3
SPEAKER_00
No one s done it yet, but it would be a couple of months work.
3
SPEAKER_01
Okay.
3
SPEAKER_01
Also learning rate in terms of DawnBench.
3
SPEAKER_01
There s some magic on learning rate that you played around with.
3
SPEAKER_00
Yeah.
3
SPEAKER_01
Interesting.
3
SPEAKER_00
Yeah.
3
SPEAKER_00
So this is all work that came from a guy called Leslie Smith.
3
SPEAKER_00
Leslie s a researcher who, like us, cares a lot about just the practicalities of training neural networks quickly and accurately.
3
SPEAKER_00
Which you would think is what everybody should care about, but almost nobody does.
3
SPEAKER_00
And he discovered something very interesting, which he calls superconvergence, which is there are certain networks that with certain settings of high parameters could suddenly be trained 10 times faster by using a 10 times higher learning rate.
3
SPEAKER_00
Now, no one published that paper because it s not
3
SPEAKER_00
an area of kind of active research in the academic world.
3
SPEAKER_00
No academics recognize that this is important.
3
SPEAKER_00
And also deep learning in academia is not considered a experimental science.
3
SPEAKER_00
So unlike in physics, where you could say like, I just saw a subatomic particle do something which the theory doesn t explain, you could publish that without an explanation.
3
SPEAKER_00
And then in the next 60 years, people can try to work out how to explain it.
3
SPEAKER_00
We don t allow this in the deep learning world.
3
SPEAKER_00
So it s literally impossible for Leslie to publish a paper that says, I ve just seen something amazing happen.
3
SPEAKER_00
This thing trained 10 times faster than it should have.
3
SPEAKER_00
I don t know why.
3
SPEAKER_00
And so the reviewers were like, well, you can t publish that because you don t know why.
3
SPEAKER_01
So anyway, that s important to pause on because there s so many discoveries that would need to start like that.
3
SPEAKER_00
Every other scientific field I know of works that way.
3
SPEAKER_00
I don t know why ours is uniquely disinterested in publishing unexplained experimental results, but there it is.
3
SPEAKER_00
So it wasn t published.
3
SPEAKER_00
Having said that, I read a lot more unpublished papers than published papers because that s where you find the interesting insights.
3
SPEAKER_00
So I absolutely read this paper and I was just like, this is...
3
SPEAKER_00
astonishingly mind-blowing and weird and awesome.
3
SPEAKER_00
And like, why isn t everybody only talking about this?
3
SPEAKER_00
Because like, if you can train these things 10 times faster, they also generalize better because you re, you re doing less epochs, which means you look at the data less, you get better accuracy.
3
SPEAKER_00
So I ve been kind of studying that ever since.
3
SPEAKER_00
And eventually Leslie kind of figured out a lot of how to get this done and we added minor tweaks.
3
SPEAKER_00
And a big part of the trick is starting at a very low learning rate, very gradually increasing it.
3
SPEAKER_00
So as you re training your model, you would take very small steps at the start and you gradually make them bigger and bigger until eventually you re taking much bigger steps than anybody thought was possible.
3
SPEAKER_00
There s a few other little tricks to make it work, but basically we can reliably get superconvergence.
3
SPEAKER_00
And so for the DawnBench thing, we were using just much higher learning rates than people expected to work.
3
SPEAKER_01
What do you think the future of, I mean, it makes so much sense for that to be a critical hyperparameter learning rate that you vary.
3
SPEAKER_01
What do you think the future of learning rate magic looks like?
3
SPEAKER_00
Well there s been a lot of great work in the last 12 months in this area.
3
SPEAKER_00
It s, and people are increasingly realizing that optimize, like we just have no idea really how optimizers work.
3
SPEAKER_00
And the combination of weight decay, which is how we regularize optimizers, and the learning rate, and then other things like the epsilon we use in the atom optimizer, they all work together in weird ways.
3
SPEAKER_00
And different parts of the model, this is another thing we ve done a lot of work on, is research into how different parts of the model should be trained at different rates in different ways.
3
SPEAKER_00
So we do something we call discriminative learning rates, which is really important, particularly for transfer learning.
3
SPEAKER_00
So really, I think in the last 12 months, a lot of people have realized that all this stuff is important.
3
SPEAKER_00
There s been a lot of great work coming out.
3
SPEAKER_00
And we re starting to see algorithms appear which have very, very few dials, if any, that you have to touch.
3
SPEAKER_00
So I think what s going to happen is the idea of a learning rate, well, it almost already has disappeared in the latest research.
3
SPEAKER_00
And instead, it s just like, you know, we know enough about how to
3
SPEAKER_00
interpret the gradients and the change of gradients we see to know how to set every parameter.
3
SPEAKER_01
So you see the future of deep learning where really, where s the input of a human expert needed?
3
SPEAKER_00
Well, hopefully the input of the human expert will be almost entirely unneeded from the deep learning point of view.
3
SPEAKER_00
So, again, like Google s approach to this is to try and use thousands of times more compute to run lots and lots of models at the same time and hope that one of them is good.
3
SPEAKER_00
AutoML kind of?
3
SPEAKER_00
Yeah, autoML kind of stuff, which I think is insane.
3
SPEAKER_00
When you better understand the mechanics of how models learn, you don t have to try a thousand different models to find which one happens to work the best.
3
SPEAKER_00
You can just jump straight to the best one, which means that it s more accessible in terms of compute.
3
SPEAKER_00
Cheaper and also with less hyper parameters to set it means you don t need deep learning experts to train your deep learning model for you Which means that domain experts can do more of the work which means that now you can focus the human time on the kind of interpretation Data gathering identifying model errors and stuff like that
3
SPEAKER_01
Yeah, the data side.
3
SPEAKER_01
How often do you work with data these days in terms of the cleaning, looking at it?
3
SPEAKER_01
Like Darwin looked at different species while traveling about.
3
SPEAKER_01
Do you look at data?
3
SPEAKER_01
Have you in your roots in Kaggle?
3
SPEAKER_00
Always yeah, I mean, it s a key part of our course.
3
SPEAKER_00
It s like before we train a model in the course We see how to look at the data and then after the first thing we do after we train our first model Which we fine-tune an image net model for five minutes and then the thing we immediately do after that is we learn how to analyze
3
SPEAKER_00
the results of the model by looking at examples of misclassified images, looking at a classification matrix, and then doing research on Google to learn about the kinds of things that it s misclassifying.
3
SPEAKER_00
So to me, one of the really cool things about machine learning models in general
3
SPEAKER_00
is that you can, when you interpret them, they tell you about things like what are the most important features, which groups you re misclassifying, and they help you become a domain expert more quickly because you can focus your time on the bits that the model is telling you is important.
3
SPEAKER_00
So it lets you deal with things like data leakage, for example, if it says, oh, the main feature I m looking at is customer ID, you know, and you re like, oh, customer ID shouldn t be predictive, and then you can talk to the,
3
SPEAKER_00
people that manage customer IDs and they ll tell you like, oh yes, as soon as a customer s application is accepted, we add a one on the end of their customer ID or something, you know.
3
SPEAKER_00
So yeah, model, looking at data, particularly from the lens of which parts of the data the model says is important is super important.
3
SPEAKER_01
Yeah, and using the model to almost debug the data, to learn more about the data.
3
SPEAKER_01
What are the different cloud options for training your own networks?
3
SPEAKER_01
Last question related to DAWNBench.
3
SPEAKER_01
Well, it s part of a lot of the work you do, but from a perspective of performance, I think you ve written this in a blog post.
3
SPEAKER_01
There s AWS, there s TPU from Google.
3
SPEAKER_01
What s your sense, what the future holds, what would you recommend now in terms of training the cloud?
3
SPEAKER_00
Google s TPUs and the best NVIDIA GPUs are similar.
3
SPEAKER_00
I mean, maybe the TPUs are like 30% faster, but they re also much harder to program.
3
SPEAKER_00
There isn t a clear leader in terms of hardware right now, although much more importantly, NVIDIA s GPUs are much more programmable.
3
SPEAKER_00
They ve got much more written for them.
3
SPEAKER_00
So like that s the clear leader for me and where I would spend my time as a researcher and practitioner.
3
SPEAKER_00
But then in terms of the platform, I mean, we re, we re super lucky now with stuff like Google, uh, GCP, Google cloud and, um, AWS that you can access a GPU pretty quickly and easily.
3
SPEAKER_00
But I mean, for AWS, it s still too hard.
3
SPEAKER_00
Like you have to.
3
SPEAKER_00
find an AMI and get the instance running and then install the software you want and blah, blah, blah.
3
SPEAKER_00
GCP is currently the best way to get started on a full server environment because they have a fantastic fast AI and PyTorch ready to go instance.
3
SPEAKER_00
which has all the courses pre-installed.
3
SPEAKER_00
It has Jupyter Notebook pre-running.
3
SPEAKER_00
Jupyter Notebook is this wonderful interactive computing system which everybody basically should be using for any kind of data-driven research.
3
SPEAKER_00
But then even better than that, there are platforms like Salamander, which we own, and Paperspace, where literally you click a single button and it pops up a Jupyter Notebook.
3
SPEAKER_00
straight away without any kind of installation or anything.
3
SPEAKER_00
And all the course notebooks are all pre-installed.
3
SPEAKER_00
So like for me, this is one of the things we spent a lot of time kind of curating and working on.
3
SPEAKER_00
Because when we first started our courses, the biggest problem was people dropped out of lesson one because they couldn t get an AWS instance running.
3
SPEAKER_00
So things are so much better now.
3
SPEAKER_00
And like we actually have, if you go to course.fast.ai, the first thing it says is, here s how to get started with your GPU.
3
SPEAKER_00
And there s like, you just click on the link and you click start.
3
SPEAKER_01
And it s going you will go GCP.
3
SPEAKER_01
I have to confess.
3
SPEAKER_00
I ve never used the Google GCP Yeah, GCP gives you $300 of compute for free, which is really nice
3
SPEAKER_00
But as I say, salamander and paper space are even easier still.
3
SPEAKER_01
So from the perspective of deep learning frameworks, you work with fast AI, particularly this framework, and PyTorch and TensorFlow.
3
SPEAKER_01
What are the strengths of each platform in your perspective?
3
SPEAKER_00
So in terms of what we ve done our research on and taught in our course, we started with Theano and Keras, and then we switched to TensorFlow and Keras, and then we switched to PyTorch, and then we switched to PyTorch and Fast.ai.
3
SPEAKER_00
And that kind of reflects a growth and development of the ecosystem of deep learning libraries.
3
SPEAKER_00
Theano and TensorFlow
3
SPEAKER_00
were great, but were much harder to teach and do research and development on because they define what s called a computational graph up front, a static graph, where you basically have to say, here are all the things that I m going to eventually do in my model.
3
SPEAKER_00
And then later on you say, okay, do those things with this data.
3
SPEAKER_00
And you can t like debug them.
3
SPEAKER_00
You can t do them step by step.
3
SPEAKER_00
You can t program them interactively in a Jupyter notebook and so forth.
3
SPEAKER_00
PyTorch was not the first but PyTorch was certainly the the strongest entrant to come along and say let s not do it that way let s just use normal Python and Everything you know about in Python is just going to work and we ll figure out how to make that run on the GPU as and when necessary that turned out to be a huge a huge leap in terms of what we could do with our research and what we could do with our teaching and Because it wasn t limiting
3
SPEAKER_00
Yeah, I mean it was critical for us for something like DawnBench to be able to rapidly try things.
3
SPEAKER_00
It s just so much harder to be a researcher and practitioner when you have to do everything up front and you can t inspect it.
3
SPEAKER_00
The problem with PyTorch is it s not at all accessible to newcomers because you have to like write your own training loop and manage the gradients and all this stuff.
3
SPEAKER_00
And it s also like not great for researchers because you re spending your time dealing with all this boilerplate and overhead rather than thinking about your algorithm.
3
SPEAKER_00
So we ended up writing this very multi-layered API that at the top level, you can train a state-of-the-art neural network and three lines of code.
3
SPEAKER_00
And which kind of talks to an API, which talks to an API, which talks to an API, which like you can dive into at any level and get progressively
3
SPEAKER_00
closer to the machine kind of levels of control.
3
SPEAKER_00
And this is the fast AI library.
3
SPEAKER_00
That s been critical for us and for our students and for lots of people that have won big learning competitions with it and written academic papers with it.
3
SPEAKER_00
It s made a big difference.
3
SPEAKER_00
We re still limited though by Python, and particularly this problem with things like recurrent neural nets, say, where you just can t change things unless you accept it going so slowly that it s impractical.
3
SPEAKER_00
So in the latest incarnation of the course and with some of the research we re now starting to do, we re starting to do stuff, some stuff in Swift.
3
SPEAKER_00
I think we re three years away from that being super practical, but I m in no hurry.
3
SPEAKER_00
I m very happy to invest the time to get there.
3
SPEAKER_00
But, you know, with that, we actually already have a nascent version of the Fast.ai library for Vision running on Swish and TensorFlow.
3
SPEAKER_00
Because Python for TensorFlow is not going to cut it.
3
SPEAKER_00
It s just a disaster.
3
SPEAKER_00
What they did was they tried to replicate the bits that people were saying they like about PyTorch, this kind of interactive computation, but they didn t actually change their foundational runtime components.
3
SPEAKER_00
So they kind of added this like syntax sugar, they re called TF Eager, TensorFlow Eager, which makes it look a lot like PyTorch, but it s 10 times slower than PyTorch to actually do a step.
3
SPEAKER_00
So because they didn t invest the time in like retooling the foundations, because their code base is so horribly complex.
3
SPEAKER_01
Yeah, I think it s probably very difficult to do that kind of retooling.
3
SPEAKER_00
Yeah, well, particularly the way TensorFlow was written.
3
SPEAKER_00
It was written by a lot of people very quickly in a very disorganized way.
3
SPEAKER_00
So, like, when you actually look in the code, as I do often, I m always just like, oh, God, what were they thinking?
3
SPEAKER_00
It s just, it s pretty awful.
3
SPEAKER_00
So, I m really extremely negative about the potential future for Python TensorFlow.
3
SPEAKER_00
But Swift for TensorFlow can be a different beast altogether.
3
SPEAKER_00
It can basically be a layer on top of MLIR that takes advantage of all the great compiler stuff that Swift builds on with LLVM.
3
SPEAKER_00
And yeah, I think it will be absolutely fantastic.
3
SPEAKER_01
Well, you re inspiring me to try.
3
SPEAKER_01
I haven t truly felt the pain of TensorFlow 2.0 Python.
3
SPEAKER_00
It s fine by me, but... Yeah, I mean, it does the job if you re using like predefined things that somebody s already written.
3
SPEAKER_00
But if you actually compare, you know, like I ve had to do, because I ve been having to do a lot of stuff with TensorFlow recently, you actually compare like, okay, I want to write something from scratch.
3
SPEAKER_00
And you re like, I just keep finding it s like, oh, it s running 10 times slower than PyTorch.
3
SPEAKER_01
So is the biggest cost, let s throw running time out the window, how long it takes you to program?
3
SPEAKER_00
That s not too different now, thanks to TensorFlow Eager.
3
SPEAKER_00
That s not too different.
3
SPEAKER_00
But because so many things take so long to run, you wouldn t run it at 10 times slower.
3
SPEAKER_00
You d just go like, oh, this is taking too long.
3
SPEAKER_00
And also, there s a lot of things which are just less programmable, like tf.data, which is the way data processing works in TensorFlow, is just this big mess.
3
SPEAKER_00
It s incredibly inefficient.
3
SPEAKER_00
And they kind of had to write it that way because of the TPU problems I described earlier.
4
SPEAKER_00
So I just, you know, I just feel like they ve got this huge technical debt which they re not going to solve without starting from scratch.
4
SPEAKER_01
So here s an interesting question then.
4
SPEAKER_01
If there s a new student starting today, what would you recommend they use?
4
SPEAKER_00
Well, I mean, we obviously recommend Fast.ai and PyTorch because we teach new students and that s what we teach with.
4
SPEAKER_00
So we would very strongly recommend that because it will let you get on top of the concepts much more quickly.
4
SPEAKER_00
So then you ll become an actual, and you ll also learn the actual state-of-the-art techniques, you know, so you actually get world-class results.
4
SPEAKER_00
Honestly, it doesn t much matter what library you learn, because switching from Chainer to MXNet to TensorFlow to PyTorch is going to be a couple of days  work, as long as you understand the foundations well.
4
SPEAKER_01
But you think, will Swift creep in there as a thing that people start using?
4
SPEAKER_00
Not for a few years, particularly because Swift has no data science community, libraries, tooling.
4
SPEAKER_00
And the Swift community has
4
SPEAKER_00
a total lack of appreciation and understanding of numeric computing.
4
SPEAKER_00
So like they keep on making stupid decisions, you know, for years they ve just done dumb things around performance and prioritization.
4
SPEAKER_00
That s clearly changing now because
4
SPEAKER_00
The developer of Swift, Chris Latner, is working at Google on Swift for TensorFlow, so that s a priority.
4
SPEAKER_00
It ll be interesting to see what happens with Apple, because Apple hasn t shown any sign of caring about numeric programming in Swift, so hopefully they ll
4
SPEAKER_00
Get off their ass and start appreciating this because currently all of their low-level libraries are Not written in Swift.
4
SPEAKER_00
They re not particularly Swifty at all stuff like core ML.
4
SPEAKER_00
They re really pretty rubbish, so
4
SPEAKER_00
Yeah, so there s a long way to go but at least one nice thing is that Swift for TensorFlow can actually Directly use Python code and Python libraries in a literally the entire Lesson one notebook of fast AI runs in Swift right now in Python mode.
4
SPEAKER_00
So that s that s a nice intermediate thing
4
SPEAKER_01
How long does it take, if you look at the two Fast.ai courses, how long does it take to get from point zero to completing both courses?
4
SPEAKER_00
It varies a lot.
4
SPEAKER_00
Somewhere between two months and two years, generally.
4
SPEAKER_01
So for two months, how many hours a day on average?
4
SPEAKER_00
Somebody who is a very competent coder can do 70 hours per course and pick up 70.
4
70?
4
SPEAKER_00
That s it?
4
SPEAKER_00
Okay.
4
SPEAKER_00
But a lot of people I know take a year off to study fast AI full-time, and say at the end of the year they feel pretty competent.
4
SPEAKER_00
Because generally there s a lot of other things you do, like generally they ll be entering Kaggle competitions, they might be reading Ian Goodfellow s book, they might, you know, they ll be doing a bunch of stuff.
4
SPEAKER_00
And often, you know, particularly if they are a domain expert, their coding skills might be a little on the pedestrian side.
4
SPEAKER_00
So part of it s just like doing a lot more writing.
4
SPEAKER_01
What do you find is the bottleneck for people usually, except getting started and setting stuff up?
4
SPEAKER_00
I would say coding.
4
SPEAKER_00
Yeah.
4
SPEAKER_00
I would say the best, the people who are strong coders pick it up the best.
4
SPEAKER_00
Although another bottleneck is people who have a lot of experience of classic statistics can really struggle because it, the intuition is so the opposite of what they re used to.
4
SPEAKER_00
They re very used to like trying to reduce the number of parameters in their model and
4
SPEAKER_00
looking at individual coefficients and stuff like that.
4
SPEAKER_00
So I find people who have a lot of coding background and know nothing about statistics are generally going to be the best off.
4
SPEAKER_01
So you taught several courses on deep learning and as Feynman says, the best way to understand something is to teach it.
4
SPEAKER_01
What have you learned about deep learning from teaching it?
4
SPEAKER_00
a lot.
4
SPEAKER_00
It s a key reason for me to teach the courses.
4
SPEAKER_00
I mean obviously it s going to be necessary to achieve our goal of getting domain experts to be familiar with deep learning, but it was also necessary for me to achieve my goal of being really familiar with deep learning.
4
SPEAKER_00
I mean, to see so many domain experts from so many different backgrounds, it s definitely, I wouldn t say taught me, but convinced me something that I d like to believe was true, which was anyone can do it.
4
SPEAKER_00
So there s a lot of snobbishness out there about only certain people can learn to code, only certain people are going to be smart enough to do AI.
4
SPEAKER_00
That s definitely bullshit.
4
SPEAKER_00
I ve seen so many people from so many different backgrounds get state-of-the-art results in their domain areas now.
4
SPEAKER_00
It s definitely taught me that the key differentiator between people that succeed and people that fail is tenacity.
4
SPEAKER_00
That seems to be basically the only thing that matters.
4
SPEAKER_00
The people, a lot of people give up.
4
SPEAKER_00
And, but of the ones who don t give up, pretty much everybody succeeds.
4
SPEAKER_00
You know, even if at first I m just kind of like thinking like, wow, they really aren t quite getting it yet, are they?
4
SPEAKER_00
But eventually people get it and they succeed.
4
SPEAKER_00
So I think that s been, I think they re both things I liked to believe was true, but I don t feel like I really had strong evidence for them to be true.
4
SPEAKER_00
But now I can say I ve seen it again and again.
4
SPEAKER_01
So what advice do you have for someone who wants to get started in deep learning?
4
SPEAKER_00
train lots of models, that s how you learn it.
4
SPEAKER_00
So I think, it s not just me, I think our course is very good, but also lots of people independently have said it s very good.
4
SPEAKER_00
It recently won the COGx award for AI courses as being the best in the world.
4
SPEAKER_00
So I d say come to our course, course.fast.ai.
4
SPEAKER_00
The thing I keep on hopping on in my lessons is train models Print out the inputs to the models print out to the outputs to the models like study You know change change the inputs a bit.
4
SPEAKER_00
Look at how the outputs vary just run lots of experiments to get a you know an intuitive understanding of what s going on and
4
SPEAKER_01
to get hooked, do you think, you mentioned training, do you think just running the models inference, like if we talk about getting started?
4
SPEAKER_00
No, you ve got to fine tune the models.
4
SPEAKER_00
So that s, that s, that s the critical thing.
4
SPEAKER_00
Cause at that point you now have a model that s in your domain area.
4
SPEAKER_00
So there s, there s, there s no point running somebody else s model cause it s not your model.
4
SPEAKER_00
Like, so it only takes five minutes to fine tune a model for the data you care about.
4
SPEAKER_00
And in Lesson 2 of the course, we teach you how to create your own dataset from scratch by scripting Google Image Search.
4
SPEAKER_00
And we show you how to actually create a web application running online.
4
SPEAKER_00
So I create one in the course that differentiates between a teddy bear, a grizzly bear, and a brown bear.
4
SPEAKER_00
And it does it with basically 100% accuracy.
4
SPEAKER_00
Took me about four minutes to scrape the images from Google search in the script.
4
SPEAKER_00
There s a little graphical widgets we have in the notebook that help you clean up the data set.
4
SPEAKER_00
There s other widgets that help you study the results to see where the errors are happening.
4
SPEAKER_00
So now we ve got over a thousand replies in our share your work here thread of students saying here s the thing I built and so there s people who like and a lot of them are state-of-the-art like somebody said Oh, I tried looking at Devon Gary characters and I couldn t believe it The thing that came out was more accurate than the best academic paper after lesson one and then there s others which are just more kind of fun like somebody who s doing
4
SPEAKER_00
Trinidad and Tobago hummingbirds.
4
SPEAKER_00
She said that s kind of their national bird and Susie s got something that can now classify a Trinidad and Tobago hummingbirds So yeah train models fine-tune models with your data set and then study their inputs and outputs How much is fast AI courses free?
4
SPEAKER_00
Everything we do is free.
4
SPEAKER_00
We have no revenue sources of any kind.
4
SPEAKER_00
It s just a service to the community and
4
SPEAKER_01
You re a saint.
4
SPEAKER_01
Okay.
4
SPEAKER_01
Once a person understands the basics, trains a bunch of models, if we look at the scale of years, what advice do you have for someone wanting to eventually become an expert?
4
SPEAKER_00
Train lots of models, but specifically train lots of models in your domain area.
4
SPEAKER_00
So an expert what, right?
4
SPEAKER_00
We don t need more expert, like,
4
SPEAKER_00
create slightly evolutionary research in areas that everybody s studying.
4
SPEAKER_00
We need experts at using deep learning to diagnose malaria, or we need experts at using deep learning to analyze language to study media bias, or we need experts in
4
SPEAKER_00
analyzing fisheries to identify problem areas in the ocean.
4
SPEAKER_00
That s what we need.
4
SPEAKER_00
So become the expert in your passion area.
4
SPEAKER_00
And this is a tool which you can use for just about anything.
4
SPEAKER_00
And you ll be able to do that thing better than other people, particularly by combining it with your passion and domain expertise.
4
SPEAKER_01
So that s really interesting.
4
SPEAKER_01
Even if you do want to innovate on transfer learning or active learning, your thought is, I mean, it s one I certainly share, is you also need to find a domain or a data set that you actually really care for.
4
SPEAKER_00
If you re not working on a real problem that you understand, how do you know if you re doing it any good?
4
SPEAKER_00
How do you know if your results are good?
4
SPEAKER_00
How do you know if you re getting bad results?
4
SPEAKER_00
Why are you getting bad results?
4
SPEAKER_00
Is it a problem with the data?
4
SPEAKER_00
How do you know you re doing anything useful?
4
SPEAKER_00
Yeah, the only, to me, the only really interesting research is, not the only, but the vast majority of interesting research is like try and solve an actual problem and solve it really well.
4
SPEAKER_01
So both understanding sufficient tools on the deep learning side and becoming a domain expert in a particular domain are really things within reach for anybody.
4
SPEAKER_00
Yeah, I mean, to me, I would compare it to like studying self-driving cars, having never looked at a car or been in a car or turned a car on, you know, which is like the way it is for a lot of people.
4
SPEAKER_00
They ll study some academic data set.
4
SPEAKER_00
where they literally have no idea about that.
4
SPEAKER_01
By the way, I m not sure how familiar you are with autonomous vehicles, but that is literally, you describe a large percentage of robotics folks working in self-driving cars, is they actually haven t considered driving.
4
SPEAKER_01
They haven t actually looked at what driving looks like.
4
SPEAKER_01
They haven t driven.
4
SPEAKER_00
And it s a problem, because you know, when you ve actually driven, you know, like, these are the things that happened to me when I was driving.
4
SPEAKER_01
There s nothing that beats the real world examples of just experiencing them.
4
SPEAKER_01
You ve created many successful startups.
4
SPEAKER_01
What does it take to create a successful startup?
4
SPEAKER_00
Same thing as becoming a successful deep learning practitioner, which is not giving up.
4
SPEAKER_00
So you can
4
SPEAKER_00
You run out of money or run out of time or run out of something, you know, but if you keep costs super low and try and save up some money beforehand so you can afford to have some time.
4
SPEAKER_00
then just sticking with it is one important thing.
4
SPEAKER_00
Doing something you understand and care about is important.
4
SPEAKER_00
By something I don t mean... The biggest problem I see with deep learning people is they do a PhD in deep learning and then they try and commercialize their PhD.
4
SPEAKER_00
Which is a waste of time because that doesn t solve an actual problem.
4
SPEAKER_00
You picked your PhD topic because it was an interesting kind of engineering or math or research exercise.
4
SPEAKER_00
But yeah, if you ve actually spent time as a recruiter, and you know that most of your time was spent sifting through resumes, and you know that most of the time you re just looking for certain kinds of things, and you can try doing that with a model for a few minutes and see whether that s something which the model seems to be able to do as well as you could, then you re on the right track to creating a startup.
4
SPEAKER_00
And then I think just, yeah, being,
4
SPEAKER_00
Just be pragmatic and try and stay away from venture capital money as long as possible, preferably forever.
4
SPEAKER_01
So, yeah, on that point, do you, venture capital, so did you, were you able to successfully run startups with self-funded for quite a while?
4
SPEAKER_00
Yeah, so my first two were self-funded and that was the right way to do it.
4
SPEAKER_01
Is that scary?
4
SPEAKER_00
No, VC startups are much more scary because you have these people on your back who do this all the time and who have done it for years telling you grow, grow, grow, grow.
4
SPEAKER_00
And they don t care if you fail.
4
SPEAKER_00
They only care if you don t grow fast enough.
4
SPEAKER_00
So that s scary.
4
SPEAKER_00
Whereas doing the ones myself, well, with partners,
4
SPEAKER_00
Who were friends it s nice because like we just went along at a pace that made sense and we were able to build it to something which Was big enough that we never had to work again, but was not big enough that any VC would think it was impressive and That was enough for us to be excited, you know so I I thought that s a much better way to do things for most people and
4
SPEAKER_01
in generally speaking, not for yourself, but how do you make money during that process?
4
SPEAKER_01
Do you cut into savings?
4
SPEAKER_00
So yeah, so I started Fastmail and Optimal Decisions at the same time in 1999 with two different friends.
4
SPEAKER_00
And the Fastmail
4
SPEAKER_00
I guess I spent $70 a month on the server, and when the server ran out of space, I put a payments button on the front page and said, if you want more than 10 megaspace, you have to pay $10 a year.
4
SPEAKER_01
So run low, like keep your cost down.
4
SPEAKER_00
Yes, I kept my cost down.
4
SPEAKER_00
And once, you know, once once I needed to spend more money, I asked people to spend the money for me.
4
SPEAKER_00
And that that was that basically from then on, we were making money and I was profitable from then.
4
SPEAKER_00
For optimal decisions, it was a bit harder because we were trying to sell something that was more like a $1 million sale.
4
SPEAKER_00
But what we did was we would sell scoping projects, so kind of like prototype-y projects.
4
SPEAKER_00
But rather than doing it for free, we would sell them for $50,000 to $100,000.
4
SPEAKER_00
So again, we were covering our costs and also making the client feel like we were doing something valuable.
4
SPEAKER_00
So in both cases, we were profitable from
4
SPEAKER_00
six months in.
4
SPEAKER_01
Ah, nevertheless, it s scary.
4
SPEAKER_00
I mean, yeah, sure.
4
SPEAKER_00
I mean, it s scary before you jump in.
4
SPEAKER_00
And I just, I guess I was comparing it to the scariness of VC.
4
SPEAKER_00
I felt like with VC stuff, it was more scary, kind of.
4
SPEAKER_00
Much more in somebody else s hands, you know, will they fund you or not?
4
SPEAKER_00
And what do they think of what you re doing?
4
SPEAKER_00
I also found it very difficult with VCs back startups to actually Do the thing which I thought was important for the company rather than doing the thing which I thought would make the VC happy Now VCs always tell you not to do the thing that makes them happy But then if you don t do the thing that makes them happy they get sad so
4
SPEAKER_01
And do you think optimizing for the, whatever they call it, the exit, is a good thing to optimize for?
4
SPEAKER_00
I mean, it can be, but not at the VC level, because the VC exit needs to be, you know, a thousand X. So, where else the lifestyle exit, if you can sell something for $10 million and you ve made it, right?
4
SPEAKER_00
So, I don t, it depends.
4
SPEAKER_00
If you want to build something that s going to, you re kind of happy to do forever, then fine.
4
SPEAKER_00
If you want to build something you want to sell,
4
SPEAKER_00
In three years time, that s fine, too.
4
SPEAKER_00
I mean, they re both perfectly good outcomes.
4
SPEAKER_01
So you re learning Swift now, in a way.
4
SPEAKER_01
I mean, you re trying to.
4
SPEAKER_01
And I read that you use, at least in some cases, spaced repetition as a mechanism for learning new things.
4
SPEAKER_00
Yeah.
4
SPEAKER_01
I use Anki quite a lot myself.
4
SPEAKER_00
Me too.
4
SPEAKER_01
I actually never talked to anybody about it.
4
SPEAKER_01
Don t know how many people do it, but it works incredibly well for me.
4
SPEAKER_01
Can you talk to your experience?
4
SPEAKER_01
First of all, okay, let s back it up.
4
SPEAKER_01
What is spaced repetition?
4
SPEAKER_00
Spaced repetition is a
4
SPEAKER_00
An idea created by a psychologist named Ebbinghaus, I don t know, must be a couple hundred years ago or something, 150 years ago.
4
SPEAKER_00
He did something which sounds pretty damn tedious.
4
SPEAKER_00
He wrote down random sequences of letters on cards and tested how well he would remember those random sequences.
4
SPEAKER_00
day later, a week later, whatever.
4
SPEAKER_00
He discovered that there was this kind of a curve where his probability of remembering one of them would be dramatically smaller the next day, and then a little bit smaller the next day, and a little bit smaller the next day.
4
SPEAKER_00
What he discovered is that if he revised those cards after a day, the probabilities would decrease at a smaller rate.
4
SPEAKER_00
And then if he revised them again a week later, they would decrease at a smaller rate again.
4
SPEAKER_00
And so he basically figured out a roughly optimal equation for when you should revise something you want to remember.
4
SPEAKER_00
So it s best repetition learning is using this simple algorithm, just something like revise something after a day and then three days and then a week and then three weeks and so forth.
4
SPEAKER_00
And so if you use a program like Anki, as you know, it will just do that for you.
5
SPEAKER_00
And it will say, did you remember this?
5
SPEAKER_00
And if you say no, it will reschedule it back to appear again like 10 times faster than it otherwise would have.
5
SPEAKER_00
It s a kind of a way of being guaranteed to learn something.
5
SPEAKER_00
Because by definition, if you re not learning it, it will be rescheduled to be revised more quickly.
5
SPEAKER_00
Unfortunately, though, it s also like it doesn t let you fool yourself if you re not learning something.
5
SPEAKER_00
you know like your revisions will just get more and more.
5
SPEAKER_00
So you have to find ways to learn things productively and effectively, like treat your brain well.
5
SPEAKER_00
So using like mnemonics and stories and context and stuff like that.
5
SPEAKER_00
Um, so yeah, it s, it s a super great technique.
5
SPEAKER_00
It s like learning how to learn is something which everybody should learn before they actually learn anything.
5
SPEAKER_00
Um, but almost nobody does.
5
SPEAKER_01
So what have you, so it certainly works well for learning new languages for, I mean, for learning like small projects almost, but do you, you know, I started using it for, I forget who wrote a blog post about this inspired me.
5
SPEAKER_01
It might have been you, I m not sure.
5
SPEAKER_01
I started when I read papers, concepts and ideas, I ll put them.
5
SPEAKER_01
Was it Michael Nielsen?
5
SPEAKER_01
It was Michael Nielsen.
5
SPEAKER_00
Michael started doing this recently and has been writing about it.
5
SPEAKER_00
So the kind of today s Ebbinghaus is a guy called Peter Wozniak, who developed a system called SuperMemo.
5
SPEAKER_00
And he s been basically trying to become like
5
SPEAKER_00
the world s greatest renaissance man over the last few decades.
5
SPEAKER_00
He s basically lived his life with spaced repetition learning for everything.
5
SPEAKER_00
I, and sort of like, Michael s only very recently got into this, but he started really getting excited about doing it for a lot of different things.
5
SPEAKER_00
For me personally, I actually don t use it for anything except Chinese.
5
SPEAKER_00
And the reason for that is that Chinese is specifically a thing I made a conscious decision that I want to continue to remember, even if I don t get much of a chance to exercise it, because I m not often in China, so I don t.
5
SPEAKER_00
Or else something like programming languages or papers.
5
SPEAKER_00
Have a very different approach which is I try not to learn anything from them.
5
SPEAKER_00
But instead I try to Identify the important concepts and like actually ingest them.
5
SPEAKER_00
So like really understand that concept Deeply and study it carefully.
5
SPEAKER_00
I will decide if it really is important if it is like incorporated into our library, you know incorporated into how I do things or decide it s
5
SPEAKER_00
not worth it, say.
5
SPEAKER_00
So I find
5
SPEAKER_00
I find I then remember the things that I care about because I m using it all the time.
5
SPEAKER_00
So for the last 25 years, I ve committed to spending at least half of every day learning or practicing something new, which all my colleagues have always hated because it always looks like I m not working on what I m meant to be working on, but it always means I do everything faster because I ve been practicing a lot of stuff.
5
SPEAKER_00
So I kind of give myself a lot of opportunity to practice new things and so I find now I don t Yeah, I don t often kind of find myself wishing I could remember something because if it s something that s useful then I ve Been using it a lot.
5
SPEAKER_00
It s easy enough to look it up on Google But speaking Chinese, you can t look it up on Google.
5
SPEAKER_00
So
5
SPEAKER_01
Do you have advice for people learning new things?
5
SPEAKER_01
What have you learned as a process?
5
SPEAKER_01
I mean, it all starts with just making the hours in the day available.
5
SPEAKER_00
Yeah, you ve got to stick with it, which is, again, the number one thing that 99% of people don t do.
5
SPEAKER_00
So the people I started learning Chinese with, none of them were still doing it 12 months later.
5
SPEAKER_00
I m still doing it 10 years later.
5
SPEAKER_00
I tried to stay in touch with them, but they just, no one did it.
5
SPEAKER_00
For something like Chinese like study how human learning works so my every one of my Chinese flashcards is associated with a Story and that story is specifically designed to be memorable and we find things memorable which are like funny or disgusting or sexy or Related to people that we know or care about so I try to make sure all those stories that are in my head have those characteristics and
5
SPEAKER_00
Yeah, so you have to, you know, you won t remember things well if they don t have some context.
5
SPEAKER_00
And yeah, you won t remember them well if you don t regularly practice them, whether it be just part of your day-to-day life or, for Chinese, I mean, flashcards.
5
SPEAKER_00
I mean, the other thing is, let yourself fail sometimes.
5
SPEAKER_00
So like, I ve had various medical problems over the last few years.
5
SPEAKER_00
Basically, my flashcards just stopped for about three years.
5
SPEAKER_00
And there have been other times I ve stopped for a few months and it s so hard because you get back to it and it s like, you have 18,000 cards due.
5
SPEAKER_00
And so you just have to go, all right, well, I can either stop and give up everything or just decide to do this every day for the next two years until I get back to it.
5
SPEAKER_00
The amazing thing has been that even after three years, I
5
SPEAKER_00
you know, the Chinese was still in there.
5
SPEAKER_00
Like, it was so much faster to relearn than it was to learn the first time.
5
SPEAKER_01
Yeah, absolutely.
5
SPEAKER_01
It s in there.
5
SPEAKER_01
I have the same with guitar, with music and so on.
5
SPEAKER_01
It s sad because the work sometimes takes away and then you won t play for a year.
5
SPEAKER_01
But really, if you then just get back to it every day, you re right there again.
5
SPEAKER_01
What do you think is the next big breakthrough in artificial intelligence?
5
SPEAKER_01
What are your hopes in deep learning or beyond that people should be working on or you hope there ll be breakthroughs?
5
SPEAKER_00
I don t think it s possible to predict.
5
SPEAKER_00
I think what we already have is an incredibly powerful platform to solve lots of societally important problems that are currently unsolved.
5
SPEAKER_00
So I just hope that lots of people will learn this toolkit and try to use it.
5
SPEAKER_00
I don t think we need a lot of new technological breakthroughs to do a lot of great work right now.
5
SPEAKER_01
And when do you think we re going to create a human level intelligence system?
5
SPEAKER_01
Do you think?
5
SPEAKER_01
I don t know.
5
SPEAKER_01
How hard is it?
5
SPEAKER_01
How far away are we?
5
SPEAKER_00
I don t know.
5
SPEAKER_00
Don t have no way to know.
5
SPEAKER_00
I don t know.
5
SPEAKER_00
Like, I don t know why people make predictions about this because there s no data and nothing to go on.
5
SPEAKER_00
And that s right.
5
SPEAKER_00
It s just like there s so many societally important problems to solve right now.
5
SPEAKER_00
I just don t find it a really interesting question to even answer.
5
SPEAKER_01
So in terms of societally important problems, what s the problem that is within reach?
5
SPEAKER_00
Well, I mean, for example, there are problems that AI creates, right?
5
SPEAKER_00
So more specifically,
5
SPEAKER_00
labor force displacement is going to be huge and people keep making this frivolous econometric argument of being like, oh, there s been other things that aren t AI that have come along before and haven t created massive labor force displacement, therefore AI won t.
5
SPEAKER_01
So that s a serious concern for you?
5
SPEAKER_00
Oh, yeah.
5
SPEAKER_01
Andrew Yang is running on it.
5
SPEAKER_00
Yeah, it s, I m desperately concerned.
5
SPEAKER_00
And you see already that the changing workplace has led to a hollowing out of the middle class.
5
SPEAKER_00
You re seeing that students coming out of school today have a less rosy financial future ahead of them than their parents did, which has never happened in recent, in the last few hundred years.
5
SPEAKER_00
You know, we ve always had progress before.
5
SPEAKER_00
And you see this turning into
5
SPEAKER_00
anxiety and despair and even violence.
5
SPEAKER_00
So I very much worry about that.
5
SPEAKER_01
You ve written quite a bit about ethics too.
5
SPEAKER_00
I do think that every data scientist working with
5
SPEAKER_00
deep learning needs to recognize they have an incredibly high leverage tool that they re using that can influence society in lots of ways and if they re doing research that that research is going to be used by people doing this kind of work and they have a responsibility to consider the consequences and to think about things like
5
SPEAKER_00
How will humans be in the loop here?
5
SPEAKER_00
How do we avoid runaway feedback loops?
5
SPEAKER_00
How do we ensure an appeals process for humans that are impacted by my algorithm?
5
SPEAKER_00
How do I ensure that the constraints of my algorithm are adequately explained to the people that end up using them?
5
SPEAKER_00
There s all kinds of human issues, which only data scientists are actually in the right place to educate people about, but data scientists tend to think of themselves as
5
SPEAKER_00
just engineers and that they don t need to be part of that process, which is wrong.
5
SPEAKER_01
Well, you re in a perfect position to educate them better, to read literature, to read history, to learn from history.
5
SPEAKER_01
Well, Jeremy, thank you so much for everything you do, for inspiring a huge amount of people, getting them into deep learning and having the ripple effects, the flap of a butterfly s wings that will probably change the world.
5
SPEAKER_01
So thank you very much.