0
SPEAKER_00
The following is a conversation with Vladimir Vapnik.
0
SPEAKER_00
He s the co-inventor of the support vector machine, support vector clustering, VC theory, and many foundational ideas in statistical learning.
0
SPEAKER_00
He was born in the Soviet Union and worked at the Institute of Controlled Sciences in Moscow.
0
SPEAKER_00
Then, in the United States, he worked at AT&T, NEC Labs, Facebook Research, and now is a professor at Columbia University.
0
SPEAKER_00
His work has been cited over 170,000 times.
0
SPEAKER_00
He has some very interesting ideas about artificial intelligence and the nature of learning, especially on the limits of our current approaches and the open problems in the field.
0
SPEAKER_00
This conversation is part of MIT course on Artificial General Intelligence and the Artificial Intelligence Podcast.
0
SPEAKER_00
If you enjoy it, please subscribe on YouTube or rate it on iTunes or your podcast provider of choice.
0
SPEAKER_00
Or simply connect with me on Twitter or other social networks at Lex Friedman, spelled F-R-I-D.
0
SPEAKER_00
And now, here s my conversation with Vladimir Vapnik.
0
SPEAKER_00
Einstein famously said that God doesn t play dice.
0
SPEAKER_00
Yeah.
0
SPEAKER_00
You have studied the world through the eyes of statistics, so let me ask you in terms of the nature of reality, fundamental nature of reality, does God play dice?
0
SPEAKER_01
We don t know some factors.
0
SPEAKER_01
And because we don t know some factors, which could be important, it looks like God plays dice.
0
SPEAKER_01
But we should describe.
0
SPEAKER_01
In philosophy, they distinguish between two
0
SPEAKER_01
positions of instrumentalism, where you re creating theory for prediction, and position of realism, where you re trying to understand what God did.
0
SPEAKER_00
Can you describe instrumentalism and realism a little bit?
0
SPEAKER_01
For example, if you have some mechanical laws, what is that?
0
SPEAKER_01
Is it law which true always and everywhere?
0
SPEAKER_01
Or it is law which allow you to predict position of moving element?
0
SPEAKER_01
what you believe.
0
SPEAKER_01
You believe that it is God s law, that God created the world, which obey to this physical law, or it is just law for predictions.
0
SPEAKER_00
And which one is instrumentalism?
0
SPEAKER_01
for predictions.
0
SPEAKER_01
If you believe that this is law of God, and it s always true everywhere, that means that you re a realist.
0
SPEAKER_01
You re trying to really understand God s thought.
0
SPEAKER_00
So the way you see the world is as an instrumentalist?
0
SPEAKER_01
You know, I m working for some models, models of machine learning.
0
SPEAKER_01
So in this model, we can see a setting and we try to solve, resolve the setting, to solve the problem.
0
SPEAKER_01
And you can do it in two different ways, from the point of view of instrumentalists, and that s what everybody does now.
0
SPEAKER_01
because they say that the goal of machine learning is to find the rule for classification.
0
SPEAKER_01
That is true, but it is an instrument for prediction.
0
SPEAKER_01
But I can say the goal of machine learning is to learn about conditional probability, so how God played with
0
SPEAKER_01
And is he play?
0
SPEAKER_01
What is probability for one?
0
SPEAKER_01
What is probability for another given situation?
0
SPEAKER_01
But for prediction I don t need this.
0
SPEAKER_01
I need the rule.
0
SPEAKER_01
But for understanding, I need conditional probability.
0
SPEAKER_00
So let me just step back a little bit first to talk about, you mentioned, which I read last night, the parts of the 1960 paper by Eugene Wigner, Unreasonable Effectiveness of Mathematics and Natural Sciences.
0
SPEAKER_00
Such a beautiful paper, by the way.
0
SPEAKER_00
made me feel, to be honest, to confess my own work in the past few years on deep learning, heavily applied, made me feel that I was missing out on some of the beauty of nature in the way that math can uncover.
0
SPEAKER_00
So let me just step away from the poetry of that for a second.
0
SPEAKER_00
How do you see the role of math in your life?
0
SPEAKER_00
Is it a tool?
0
SPEAKER_00
Is it poetry?
0
SPEAKER_00
Where does it sit and does math for you have limits of what it can describe?
0
SPEAKER_01
Some people say that math is language which use God.
0
SPEAKER_01
So I believe... Speak to God or use God?
0
SPEAKER_00
Use God.
0
SPEAKER_00
Use God.
0
SPEAKER_00
Yeah.
0
SPEAKER_01
So I believe that this article
0
SPEAKER_01
about effectiveness, unreasonable effectiveness of math, is that if you re looking at mathematical structures, they know something about reality.
0
SPEAKER_01
And the most scientists from natural science, they re looking on equation and trying to understand reality.
0
SPEAKER_01
So the same in machine learning.
0
SPEAKER_01
If you try very carefully look on all equations which define conditional probability, you can understand something about reality more than from your fantasy.
0
SPEAKER_00
So math can reveal the simple underlying principles of reality, perhaps.
0
SPEAKER_01
You know what means simple?
0
SPEAKER_01
It is very hard to discover them.
0
SPEAKER_01
But then when you discover them and look at them, you see how beautiful they are.
0
SPEAKER_01
And it is surprising why people did not see that before.
0
SPEAKER_01
You re looking at an equation and derive it from equations.
0
SPEAKER_01
For example, I talked yesterday about least square method.
0
SPEAKER_01
And people have a lot of fantasy how to improve least square method.
0
SPEAKER_01
But if you re going step by step by solving some equations, you suddenly will get some term which, after thinking, you understand that it describes position of observation point.
0
SPEAKER_01
In least square method, we throw out a lot of information.
0
SPEAKER_01
We don t look in composition of point of observations.
0
SPEAKER_01
We re looking only on residuals.
0
SPEAKER_01
But when you understood that, that s a very simple idea.
0
SPEAKER_01
But it s not too simple to understand.
0
SPEAKER_01
And you can derive this just from equations.
0
SPEAKER_00
So some simple algebra, a few steps will take you to something surprising that when you think about, you understand.
0
SPEAKER_01
And that is proof that human intuition not too rich and very primitive.
0
SPEAKER_01
And it does not see very simple situations.
0
SPEAKER_00
So let me take a step back.
0
SPEAKER_00
In general, yes.
0
SPEAKER_00
But what about human, as opposed to intuition, ingenuity, the moments of brilliance?
0
SPEAKER_00
Do you have to be so hard on human intuition?
0
SPEAKER_00
Are there moments of brilliance in human intuition that can leap ahead of math, and then the math will catch up?
0
SPEAKER_01
I don t think so.
0
SPEAKER_01
I think that the best human intuition, it is putting in axioms, and then it is technical.
0
SPEAKER_00
See where the axioms take you.
0
SPEAKER_01
Yeah.
0
SPEAKER_01
But if they correctly take axioms, but axiom polished during generations of scientists.
0
SPEAKER_01
And this is integral wisdom.
0
SPEAKER_00
So, that s beautifully put.
0
SPEAKER_00
But if you maybe look at, when you think of Einstein and special relativity, what is the role of imagination coming first there, in the moment of discovery of an idea?
0
SPEAKER_00
So there s obviously a mix of math and out-of-the-box imagination there.
0
SPEAKER_01
That I don t know.
0
SPEAKER_01
Whatever I did, I exclude any imagination.
0
SPEAKER_01
Because whatever I saw in machine learning that comes from imagination, like features, like deep learning, they are not relevant to the problem.
0
SPEAKER_01
When you re looking very carefully from mathematical equations, you re deriving very simple theory, which goes far beyond theoretically than whatever people can imagine.
0
SPEAKER_01
Because it is not good fantasy.
0
SPEAKER_01
It is just interpretation.
0
SPEAKER_01
It is just fantasy.
0
SPEAKER_01
But it is not what you need.
0
SPEAKER_01
You don t need any imagination to derive, say, main principle of machine learning.
0
SPEAKER_00
When you think about learning and intelligence, maybe thinking about the human brain and trying to describe mathematically the process of learning, that is something like what happens in the human brain.
0
SPEAKER_00
Do you think we have the tools currently?
0
SPEAKER_00
Do you think we will ever have the tools to try to describe that process of learning?
0
SPEAKER_01
It is not description what s going on.
0
SPEAKER_01
It is interpretation.
0
SPEAKER_01
It is your interpretation.
0
SPEAKER_01
Your vision can be wrong.
0
SPEAKER_01
You know, one guy invent microscope, Levenhuk, for the first time.
0
SPEAKER_01
Only he got this instrument and nobody else.
0
SPEAKER_01
He kept secret about microscope.
0
SPEAKER_01
But he wrote a report in London Academia of Science.
0
SPEAKER_01
And in his report, when he s looking at the blood, he looked everywhere, on the water, on the blood, on the skin.
0
SPEAKER_01
But he described blood like fight between queen and king.
0
SPEAKER_01
So he saw blood cells, red cells, and he imagined that it is army fighting each other.
0
SPEAKER_01
And it was his interpretation of situation.
0
SPEAKER_01
And he sent this report in the Academy of Science.
0
SPEAKER_01
They very carefully looked because they believed that he s right.
0
SPEAKER_01
He saw something, but he gave wrong interpretation.
0
SPEAKER_01
And I believe the same can happen with the brain.
0
SPEAKER_01
Because the most important part... You know, I believe in human language.
0
SPEAKER_01
In some proverb, there s so much wisdom.
0
SPEAKER_01
For example, people say that it is better than a thousand days of diligent studies one day with a great teacher.
0
SPEAKER_01
But I will ask you what a teacher does.
0
SPEAKER_01
Nobody knows.
0
SPEAKER_01
And that is intelligence.
0
SPEAKER_01
But we know from history, and now from math and machine learning, that teacher can do a lot.
0
SPEAKER_00
So what, from a mathematical point of view, is a great teacher?
0
SPEAKER_01
I don t know.
0
SPEAKER_01
But we can say what teacher can do.
0
SPEAKER_01
He can introduce some invariants, some predicate for creating invariants.
0
SPEAKER_01
How he s doing it, I don t know, because teacher knows reality and can describe from this reality a predicate, invariants.
0
SPEAKER_01
But you know that when you re using invariant, you can decrease number of observations hundred times.
0
SPEAKER_00
That s... So, but maybe try to pull that apart a little bit.
0
SPEAKER_00
I think you mentioned like a piano teacher saying to the student, play like a butterfly.
0
SPEAKER_00
Yeah.
0
SPEAKER_00
I played piano, I played guitar for a long time.
0
SPEAKER_00
Yeah, maybe it s romantic, poetic, but it feels like there s a lot of truth in that statement.
0
SPEAKER_00
There is a lot of instruction in that statement.
0
SPEAKER_00
Can you pull that apart?
0
SPEAKER_00
What is that?
0
SPEAKER_00
The language itself may not contain this information.
0
SPEAKER_01
It does not blah blah blah.
0
SPEAKER_00
It does not blah blah blah, yeah.
0
SPEAKER_01
It affects you.
0
SPEAKER_00
It s what?
0
SPEAKER_00
It affects you.
0
SPEAKER_01
It affects your playing.
0
SPEAKER_00
Yes, it does, but it s not the lang... It feels like a... What is the information being exchanged there?
0
SPEAKER_00
What is the nature of information?
0
SPEAKER_00
What is the representation of that information?
0
SPEAKER_01
I believe that it is sort of predicated, but I don t know.
0
SPEAKER_01
That is exactly what intelligence in machine learning should be.
0
SPEAKER_01
Because the rest is just mathematical technique.
0
SPEAKER_01
I think that what was discovered recently is that there is two types, two mechanisms of learning.
0
SPEAKER_01
one called strong convergence mechanism, and weak convergence mechanism.
0
SPEAKER_01
Before, people used only one convergence.
0
SPEAKER_01
In weak convergence mechanism, you can use predicate.
0
SPEAKER_01
That s what play like butterfly, and it will immediately affect your playing.
0
SPEAKER_01
You know, there is English proverb, great.
0
SPEAKER_01
If it looks like a duck, swims like a duck,
0
SPEAKER_01
And quack like a duck, then it is probably duck.
0
SPEAKER_01
But this is exact about predicates.
0
SPEAKER_01
Looks like a duck, what it means.
0
SPEAKER_01
So you saw many ducks that you re training data.
0
SPEAKER_01
So you have description of how
0
SPEAKER_01
looks integral, looks ducks.
0
SPEAKER_00
Yeah, the visual characteristics of a duck, yeah.
0
SPEAKER_01
Yeah, but you want, and you have model for the cognition doubts.
0
SPEAKER_01
So you would like so that theoretical description from model coincide with empirical description, which you saw on teletext there.
0
SPEAKER_01
So about looks like a duck, it is general.
0
SPEAKER_01
But what about swims like a duck?
0
SPEAKER_01
You should know the duck swims.
0
SPEAKER_01
You can say it plays chess like a duck.
0
SPEAKER_01
Okay, duck doesn t play chess.
0
SPEAKER_01
And it is completely legal predicate, but it is useless.
0
SPEAKER_01
So half teacher can recognize not useless predicate.
0
SPEAKER_01
So up to now, we don t use this predicate in existing machine learning.
0
SPEAKER_01
So why do we need zillions of data?
0
SPEAKER_01
But in this English proverb, they use only three predicates.
0
SPEAKER_01
Looks like a duck, swims like a duck, and quacks like a duck.
0
SPEAKER_00
So you can t deny the fact that swims like a duck and quacks like a duck has humor in it, has ambiguity.
0
SPEAKER_01
Let s talk about swim like a duck.
0
SPEAKER_01
It does not say jump like a duck.
0
SPEAKER_01
Why?
0
SPEAKER_01
Because... It s not relevant.
0
SPEAKER_01
But that means that you know ducks, you know different birds, you know animals, and you derive from this that it is relevant to say swim like a duck.
0
SPEAKER_00
So underneath, in order for us to understand Swims Like a Duck, it feels like we need to know millions of other little pieces of information, which we pick up along the way.
0
SPEAKER_00
You don t think so.
0
SPEAKER_00
There doesn t need to be this knowledge base.
0
SPEAKER_00
In those statements carries some rich information that helps us understand the essence of duck.
0
SPEAKER_00
How far are we from integrating predicates?
0
SPEAKER_01
Now, you know that when you consider complete theory of machine learning, so what it does, you have a lot of functions, and then you re talking, it looks like a duck.
0
SPEAKER_01
You see your training data.
0
SPEAKER_01
From training data, you recognize, like, expected
0
SPEAKER_01
doc should look.
0
SPEAKER_01
Then you remove all functions which does not look like you think it should look from training date.
0
SPEAKER_01
So you decrease amount of function from which you pick up one.
0
SPEAKER_01
Then you give a second predicate and again decrease the set of functions.
0
SPEAKER_01
And after that you pick up the best function you can find.
0
SPEAKER_01
It is standard machine learning.
0
SPEAKER_01
So why you need not too many examples?
0
SPEAKER_00
Your predicates aren t very good?
0
SPEAKER_01
That means the predicate is very good.
0
SPEAKER_01
Because every predicate is invented to decrease admissible set of functions.
0
SPEAKER_00
So you talk about admissible set of functions and you talk about good functions.
0
SPEAKER_00
So what makes a good function?
0
SPEAKER_01
So admissible set of function is set of function which has small capacity or small diversity, small VC dimension example, which contain good function inside.
0
SPEAKER_00
So by the way, for people who don t know,
1
SPEAKER_01
VC.
1
SPEAKER_01
You re the V in the VC.
1
SPEAKER_01
So how would you describe to a layperson what VC theory is?
1
SPEAKER_01
How would you describe VC?
1
SPEAKER_00
So when you have a machine capable to pick up one function from that miscible set of functions.
1
SPEAKER_00
But set of admissible functions can be big, say contain all continuous functions and useless.
1
SPEAKER_00
You don t have so many examples to pick up functions.
1
SPEAKER_00
But it can be small.
1
SPEAKER_00
We call it capacity, but maybe better called diversity.
1
SPEAKER_00
So not very different function in the set.
1
SPEAKER_00
It s an infinite set of function, but not very diverse.
1
SPEAKER_00
So it is small VC dimension.
1
SPEAKER_00
When VC dimension is small, you need small amount of training data.
1
SPEAKER_00
So the goal is to create admissible set of functions, which have small VC dimension and contain good function.
1
SPEAKER_00
Then you will be able to pick up the function using small amount of observations.
1
SPEAKER_01
So that is the task of learning, is creating a set of admissible functions that has a small VC dimension, and then you figure out a clever way of picking up
1
SPEAKER_00
No, that is goal of learning which I formulated yesterday.
1
SPEAKER_00
Statistical learning theory does not involve in creating admissible set of function.
1
SPEAKER_00
In classical learning theory, everywhere, 100% in textbook, the set of function, admissible set of function is given.
1
SPEAKER_00
But this is science about nothing, because the most difficult problem to create admissible set of functions.
1
SPEAKER_00
given, say, a lot of functions, continuum set of functions, create admissible set of functions, that means that it has finite VC dimension, small VC dimension, and contain good function.
1
SPEAKER_00
So this was out of consideration.
1
SPEAKER_01
So what s the process of doing that?
1
SPEAKER_01
I mean, it s fascinating.
1
SPEAKER_01
What is the process of creating this admissible set of functions?
1
SPEAKER_01
That is invariance.
1
SPEAKER_01
That s invariance.
1
SPEAKER_01
Can you describe invariance?
1
SPEAKER_00
Yeah, you re looking of properties of training data, and properties means that you have some function,
1
SPEAKER_00
and you just count what is the average value of a function on training data.
1
SPEAKER_00
You have a model and what is the expectation of this function on the model.
1
SPEAKER_00
And they should coincide.
1
SPEAKER_00
So the problem is about how to pick up functions.
1
SPEAKER_00
It can be any function.
1
SPEAKER_00
In fact, it is true for all functions, but because when we re talking, say, duck does not jump, so you don t ask question, jump like a duck, because it is trivially does not jump and doesn t help you to recognize jump, but you know something.
1
SPEAKER_00
which question to ask.
1
SPEAKER_00
When you re asking, it seems like a duck.
1
SPEAKER_00
But it looks like a duck at its general situation.
1
SPEAKER_00
It looks like, say, a guy who has this illness, this disease.
1
SPEAKER_00
It is legal.
1
SPEAKER_00
So there is a general type of predicate that looks like a special type of predicate, which is related to this specific problem.
1
SPEAKER_00
And that is the intelligence part of all this business.
1
SPEAKER_00
And that s where the teacher is involved.
1
SPEAKER_01
incorporating the specialized predicates.
1
SPEAKER_01
What do you think about deep learning as neural networks, these arbitrary architectures, as helping accomplish some of the tasks you re thinking about?
1
SPEAKER_01
Their effectiveness or lack thereof, what are the weaknesses and what are the possible strengths?
1
SPEAKER_00
You know, I think that this is fantasy.
1
SPEAKER_00
Everything which, like deep learning, like features.
1
SPEAKER_00
Let me give you this example.
1
SPEAKER_00
One of the greatest books is Churchill s book about history of Second World War.
1
SPEAKER_00
And he s starting this book describing that in all time, when war is over, so
1
SPEAKER_00
The great kings, they gathered together.
1
SPEAKER_00
Almost all of them were relatives.
1
SPEAKER_00
And they discussed what should be done, how to create peace.
1
SPEAKER_00
And they came to agreement.
1
SPEAKER_00
And when happened First World War, the general public came in power.
1
SPEAKER_00
And they were so greedy that robbed Germany.
1
SPEAKER_00
And it was clear for everybody that it is not peace.
1
SPEAKER_00
That piece will last only 20 years because they were not professionals.
1
SPEAKER_00
It s the same I see in machine learning.
1
SPEAKER_00
There are mathematicians who are looking for the problem from a very deep point of view, mathematical point of view.
1
SPEAKER_00
And there are computer scientists who mostly do not know mathematics.
1
SPEAKER_00
They just have interpretation of that.
1
SPEAKER_00
and they invented a lot of blah, blah, blah interpretations like deep learning.
1
SPEAKER_00
Why you did deep learning?
1
SPEAKER_00
Mathematics does not know deep learning.
1
SPEAKER_00
Mathematics does not know neurons.
1
SPEAKER_00
It is just function.
1
SPEAKER_00
If you like to say piecewise linear function, say that.
1
SPEAKER_00
And do it in class of piecewise linear function.
1
SPEAKER_00
But they invent something.
1
SPEAKER_00
And then they try to prove
1
SPEAKER_00
Advantage of that, through interpretations, which mostly wrong.
1
SPEAKER_00
And when it s not enough, they appeal to brain, which they know nothing about that.
1
SPEAKER_00
Nobody knows what s going on in the brain.
1
SPEAKER_00
So I think that more reliable, look on math.
1
SPEAKER_00
This is mathematical problem.
1
SPEAKER_00
Do your best to solve this problem.
1
SPEAKER_00
Try to understand that there is not only one way of convergence, which is a strong way of convergence.
1
SPEAKER_00
There is a weak way of convergence, which requires predicate.
1
SPEAKER_00
And if you will go through all this stuff,
1
SPEAKER_00
you will see that you don t need deep learning.
1
SPEAKER_00
Even more, I would say one of the theories, which is called the representative theory, it says that the optimal solution of a mathematical problem, which describes learning, is on a shadow network, not on deep learning.
1
SPEAKER_01
And a shallow network, yeah.
1
SPEAKER_00
The ultimate problem is there.
1
SPEAKER_01
Absolutely.
1
SPEAKER_01
So in the end, what you re saying is exactly right.
1
SPEAKER_01
The question is, you have no value for throwing something on the table, playing with it, not math.
1
SPEAKER_01
It s like in neural network where you said throwing something in the bucket or the biological example and looking at kings and queens with the cells with a microscope.
1
SPEAKER_01
You don t see value in imagining the cells are kings and queens and using that as inspiration and imagination for where the math will eventually lead you.
1
SPEAKER_01
You think that interpretation basically deceives you in a way that s not productive.
1
SPEAKER_00
I think that if you re trying to analyze this business of learning, and especially discussion about deep learning, it is discussion about interpretation, not about things, about what you can say about things.
1
SPEAKER_01
That s right, but aren t you surprised by the beauty of it?
1
SPEAKER_01
Not mathematical beauty, but the fact that it works at all.
1
SPEAKER_01
You were criticizing that very beauty, our human desire to interpret, to find our silly interpretations in these constructs.
1
SPEAKER_01
Let me ask you this.
1
SPEAKER_01
surprised and does it inspire you?
1
SPEAKER_01
How do you feel about the success of a system like AlphaGo at beating the game of Go?
1
SPEAKER_01
Using neural networks to estimate the quality of a board and the quality of the picture.
1
SPEAKER_00
That is your interpretation, quality of the board.
1
SPEAKER_01
Yeah, yes.
1
SPEAKER_01
But it s not our interpretation.
1
SPEAKER_01
The fact is a neural network system, doesn t matter, a learning system that we don t, I think, mathematically understand that well, beats the best human player, does something that was thought impossible.
1
SPEAKER_00
That means that it s not a very difficult problem.
1
SPEAKER_01
We empirically have discovered that this is not a very difficult problem.
1
SPEAKER_01
It s true.
1
SPEAKER_01
So maybe... I can t argue.
1
SPEAKER_00
Even more, I would say, that if they use deep learning, it is not the most effective way of learning theory.
1
SPEAKER_00
And usually, when people use deep learning,
1
SPEAKER_00
they re using zillions of training data.
1
SPEAKER_00
Yeah, but you don t need this.
1
SPEAKER_00
So I describe challenge.
1
SPEAKER_00
Can we do some problems which do well deep learning method with deep net using 100 times less training data?
1
SPEAKER_00
Even more, some problems deep learning cannot solve.
1
SPEAKER_00
because it s not necessary they create admissible set of functions.
1
SPEAKER_00
To create deep architecture means to create admissible set of functions.
1
SPEAKER_00
You cannot say that you re creating good admissible set of functions.
1
SPEAKER_00
It s your fantasy.
1
SPEAKER_00
It does not come from us.
1
SPEAKER_00
But it is possible to create admissible set of functions because you have your training data.
1
SPEAKER_00
Actually, for mathematicians, when you consider invariant, you need to use law of large numbers.
1
SPEAKER_00
When you re making training in existing algorithm, you need uniform law of large numbers.
1
SPEAKER_00
which is much more difficult, it requires VC dimension and all this stuff.
1
SPEAKER_00
But nevertheless, if you use both weak and strong way of convergence, you can decrease a lot of training data.
1
SPEAKER_01
You could do the three, the swims like a duck and quacks like a duck.
1
SPEAKER_01
So let s step back and
1
SPEAKER_01
Think about human intelligence in general.
1
SPEAKER_01
Clearly that has evolved in a non-mathematical way.
1
SPEAKER_01
As far as we know, God or whoever didn t come up with a model and place it in our brain of admissible functions.
1
SPEAKER_01
It kind of evolved.
1
SPEAKER_01
I don t know, maybe you have a view on this.
1
SPEAKER_01
So Alan Turing in the 50s, in his paper, asked and rejected the question, can machines think?
1
SPEAKER_01
It s not a very useful question, but can you briefly entertain this useless question?
1
SPEAKER_01
Can machines think?
1
SPEAKER_01
So talk about intelligence and your view of it.
1
SPEAKER_00
I don t know that.
1
SPEAKER_00
I know that Turing described imitation.
1
SPEAKER_00
If a computer can imitate a human being, let s call it intelligent.
1
SPEAKER_00
And he understands that it is not a thinking computer.
1
SPEAKER_00
He completely understands what he s doing.
1
SPEAKER_00
But he set up a problem of imitation.
1
SPEAKER_00
So now we understand that the problem is not in imitation.
1
SPEAKER_00
I m not sure
1
SPEAKER_00
that intelligence is just inside of us.
1
SPEAKER_00
It may be also outside of us.
1
SPEAKER_00
I have several observations.
1
SPEAKER_00
So, when I prove some theorem, it s a very difficult theorem, in a couple of years, in several places,
1
SPEAKER_00
people prove the same theory and say, Sauer Lemma after us was done.
1
SPEAKER_00
Then another guy proves the same theory.
1
SPEAKER_00
In the history of science, it s happened all the time.
1
SPEAKER_00
For example, geometry.
1
SPEAKER_00
It s happened simultaneously.
1
SPEAKER_00
Lobachevsky, and then Gauss, and Boyai, and other guys.
1
SPEAKER_00
And it s approximately in 10 times period, 10 years period of time.
1
SPEAKER_00
And I saw a lot of examples like that.
1
SPEAKER_00
And many mathematicians think that when they develop something, they re developing something in general which affects everybody.
1
SPEAKER_00
So maybe our model that intelligence is only inside of us is incorrect.
1
SPEAKER_01
It s our interpretation, yeah.
1
SPEAKER_00
It may be there exists some connection
1
SPEAKER_00
this world intelligence, I don t know.
1
SPEAKER_01
You re almost like plugging in into... Yeah, exactly.
1
SPEAKER_01
And contributing to this... Into a big network.
1
SPEAKER_01
Into a big, maybe a neural network.
1
SPEAKER_01
No, no, no.
1
SPEAKER_01
On the flip side of that, maybe you can comment
1
SPEAKER_01
on big O complexity and how you see classifying algorithms by worst case running time in relation to their input.
1
SPEAKER_01
So that way of thinking about functions.
1
SPEAKER_01
Do you think P equals NP?
1
SPEAKER_01
Do you think that s an interesting question?
1
SPEAKER_00
Yeah, it is an interesting question.
1
SPEAKER_00
But let me talk about complexity in the both worst case scenario.
1
SPEAKER_00
There is a mathematical setting.
1
SPEAKER_00
When I came to the United States in 1991, people did not know.
1
SPEAKER_00
They did not know statistical learning.
1
SPEAKER_00
So in Russia it was published in monographs, but in America they did not know.
1
SPEAKER_00
Then they learned.
1
SPEAKER_00
And somebody told me that it is worst case theory and they will create real case theory, but till now it did not.
1
SPEAKER_00
Because it is a mathematical tool.
1
SPEAKER_00
You can do only what you can do using mathematics.
1
SPEAKER_00
And which has a clear understanding and clear description.
1
SPEAKER_00
And for this reason, we introduce complexity.
1
SPEAKER_00
And we need this because using... Actually, it is diversity.
1
SPEAKER_00
I like this one more.
1
SPEAKER_00
You see dimension, you can prove some theorems.
1
SPEAKER_00
But we also create theory for case when you know probability measure.
1
SPEAKER_00
And that is the best case which can happen in this entropy theory.
1
SPEAKER_00
So from mathematical point of view,
1
SPEAKER_00
You know the best possible case and the worst possible case.
1
SPEAKER_00
You can derive different models, but it s not so interesting.
1
SPEAKER_01
You think the edges are interesting?
1
SPEAKER_00
The edges are interesting.
1
SPEAKER_00
It is not so easy to get good bound, exact bound.
1
SPEAKER_00
It s not many cases where you have the bound is not exact.
1
SPEAKER_00
But interesting principles which discover the mass
1
SPEAKER_01
Do you think it s interesting because it s challenging and reveals interesting principles that allow you to get those bounds?
1
SPEAKER_01
Or do you think it s interesting because it s actually very useful for understanding the essence of a function, of an algorithm?
1
SPEAKER_01
So it s like me judging your life as a human being by the worst thing you did and the best thing you did versus all the stuff in the middle.
1
SPEAKER_01
It seems not productive.
1
SPEAKER_00
I don t think so, because you cannot describe situation in the middle, or it will be not general.
1
SPEAKER_00
So you can describe cases, and it is clear it has some model, but you cannot describe model for every new case.
1
SPEAKER_00
So you will be never accurate.
1
SPEAKER_01
But from a statistical point of view, the way you ve studied functions and the nature of learning and the world, don t you think that the real world has a very long tail?
1
SPEAKER_01
That the edge cases are very far away from the mean?
1
SPEAKER_01
The stuff in the middle?
1
SPEAKER_01
Or no?
1
SPEAKER_00
I don t know that.
1
SPEAKER_00
I think that, from my point of view, if you will use formal statistics,
2
SPEAKER_01
You need uniform law of large numbers.
2
SPEAKER_01
If you will use this invariance business, you will need just law of large numbers.
2
SPEAKER_01
And there s this huge difference between uniform law of large numbers and large numbers.
2
SPEAKER_00
Is it useful to describe that a little more?
2
SPEAKER_00
Or should we just take it to
2
SPEAKER_01
For example, when I was talking about DAC, I gave three predicates and that was enough.
2
SPEAKER_01
But if you will try to do formal, distinguish, you will need a lot of observation.
2
SPEAKER_01
So that means that information about looks like a duck contains a lot of bits of information, formal bits of information.
2
SPEAKER_01
So we don t know that how much bits of information contain things from artificial intelligence.
2
SPEAKER_01
And that is the subject of analysis.
2
SPEAKER_01
all business.
2
SPEAKER_01
I don t like how people consider artificial intelligence.
2
SPEAKER_01
They consider us some codes which imitate activity of human beings.
2
SPEAKER_01
It is not science.
2
SPEAKER_01
It is applications.
2
SPEAKER_01
You would like to imitate, go ahead.
2
SPEAKER_01
It is very useful and a good problem.
2
SPEAKER_01
But
2
SPEAKER_01
You need to learn something more.
2
SPEAKER_01
How people can develop, say, predicate, swims like a duck, or play like a butterfly, or something like that.
2
SPEAKER_01
Not that the teacher tells you.
2
SPEAKER_01
how it came in his mind, how he chose this image.
2
SPEAKER_01
That is the problem of intelligence.
2
SPEAKER_00
That is the problem of intelligence.
2
SPEAKER_00
And you see that connected to the problem of learning?
2
SPEAKER_00
Absolutely.
2
SPEAKER_01
Because you immediately give this predicate, like a specific predicate, swims like a dog, or quacks like a dog.
2
SPEAKER_01
It was chosen somehow.
2
SPEAKER_00
So what is the line of work, would you say, if you were to formulate it as a set of open problems?
2
SPEAKER_00
That will take us there.
2
SPEAKER_00
We ll play like a butterfly.
2
SPEAKER_00
We ll get a system to be able to... Let s separate two stories.
2
SPEAKER_01
One mathematical story, that if you have predicate, you can do something.
2
SPEAKER_01
And another story, how to get predicate.
2
SPEAKER_01
It is intelligence problem, and people even did not start to understand intelligence.
2
SPEAKER_01
Because to understand intelligence, first of all, try to understand what doing teachers, how teacher teach.
2
SPEAKER_01
Why one teacher better than another one?
2
SPEAKER_00
Yeah, so you think we really even haven t started on the journey of generating the predicates?
2
SPEAKER_01
No, we don t understand.
2
SPEAKER_01
We even don t understand that this problem exists.
2
SPEAKER_01
Because, did you hear?
2
SPEAKER_01
No, I just know name.
2
SPEAKER_01
I want to understand why one teacher better than another.
2
SPEAKER_01
and have affect teacher, student.
2
SPEAKER_01
It is not because he repeating the problem which is in textbook.
2
SPEAKER_01
He make some remarks.
2
SPEAKER_01
He make some philosophy of reasoning.
2
SPEAKER_00
Yeah, that s a beautiful, so it is a formulation of a question that is the open problem.
2
SPEAKER_00
Why is one teacher better than another?
2
SPEAKER_01
Right.
2
SPEAKER_01
What he does better.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
Why at every level?
2
SPEAKER_00
How do they get better?
2
SPEAKER_00
What does it mean to be better?
2
SPEAKER_01
The whole... Yeah.
2
SPEAKER_01
From whatever model I have, one teacher can give a very good predicate.
2
SPEAKER_01
One teacher can say, swims like a duck, and another can say, jump like a duck.
2
SPEAKER_01
and jump like a duck, carries zero information.
2
SPEAKER_00
So what is the most exciting problem in statistical learning you ve ever worked on or are working on now?
2
SPEAKER_01
I just finished this invariance story.
2
SPEAKER_01
And I m happy that I believe that it is ultimate learning story.
2
SPEAKER_01
At least I can show that there are no another mechanism, only two mechanisms.
2
SPEAKER_01
But they separate statistical part from intelligent part.
2
SPEAKER_01
And I know nothing about intelligent part.
2
SPEAKER_01
And if we will know this intelligent part,
2
SPEAKER_01
So it will help us a lot in teaching, in learning.
2
SPEAKER_00
Do you know we ll know it when we see it?
2
SPEAKER_01
So, for example, in my talk, the last slide was a challenge.
2
SPEAKER_01
So you have, say, NIST digital recognition problem.
2
SPEAKER_01
And deep learning claims that they did it very well, say, 99.5% of correct answers.
2
SPEAKER_01
But they use 60,000 observations.
2
SPEAKER_01
Can you do the same using 100 times less?
2
SPEAKER_01
But incorporating invariants, what it means, you know, digit 1, 2, 3.
2
SPEAKER_01
Just looking at that, explain to me which invariant I should keep to use 100 examples or say 100 times less examples to do the same job.
2
SPEAKER_00
Yeah, that last slide, unfortunately your talk ended quickly, but that last slide was a powerful open challenge and a formulation of the essence.
2
SPEAKER_01
That is the exact problem of intelligence.
2
SPEAKER_01
Because
2
SPEAKER_01
Everybody, when machine learning started and it was developed by mathematicians, they immediately recognized that we use much more training data than humans needed.
2
SPEAKER_01
But now again we came to the same story.
2
SPEAKER_01
We have to decrease.
2
SPEAKER_01
And that is the problem of learning.
2
SPEAKER_01
It is not like in deep learning they use zillions of training data.
2
SPEAKER_01
Because maybe zillions are not enough if you have a good invariance.
2
SPEAKER_01
You will never collect some number of observations.
2
SPEAKER_01
But now it is a question to intelligence.
2
SPEAKER_01
How to do that?
2
SPEAKER_01
Because the statistical part is ready.
2
SPEAKER_01
As soon as you supply us with a predicate, we can do a good job with a small amount of observations.
2
SPEAKER_01
And the very first challenge is well-known digit recognition.
2
SPEAKER_01
And you know digits.
2
SPEAKER_01
And please tell me invariants.
2
SPEAKER_01
I think about that.
2
SPEAKER_01
I can say for digit 3, I would introduce concept of horizontal symmetry.
2
SPEAKER_01
So the digit 3 has horizontal symmetry, say, more than, say, digit 2 or something like that.
2
SPEAKER_01
But as soon as I get the idea of horizontal symmetry, I can mathematically invent a lot of measure of horizontal symmetry, or vertical symmetry, or diagonal symmetry, whatever, if I have idea of symmetry.
2
SPEAKER_01
But what else?
2
SPEAKER_01
Looking on digit, I see that it is metapredicate, which is not shape.
2
SPEAKER_01
It is something like symmetry, like how dark is whole picture, something like that, which can self-rise a predicate.
2
SPEAKER_00
You think such a predicate could rise out of something that s not general, meaning
2
SPEAKER_00
It feels like for me to be able to understand the difference between a two and a three, I would need to have had a childhood of 10 to 15 years playing with kids, going to school, being yelled by parents.
2
SPEAKER_00
all of that walking, jumping, looking at ducks, and now then I would be able to generate the right predicate for telling the difference between two and a three.
2
SPEAKER_00
Or do you think there s a more efficient way?
2
SPEAKER_01
I know for sure that you must know something more than digits.
2
SPEAKER_01
Yes.
2
SPEAKER_01
And that s a powerful statement.
2
SPEAKER_01
Yeah.
2
SPEAKER_01
But maybe there are several languages of description of these elements of digits.
2
SPEAKER_01
So I m talking about symmetry, about some properties of geometry.
2
SPEAKER_01
I m talking about something abstract.
2
SPEAKER_01
I don t know that.
2
SPEAKER_01
But this is a problem of intelligence.
2
SPEAKER_01
So in one of our articles, it is trivial to show that every example can carry not more than one bit of information in real, because
2
SPEAKER_01
When you show example, and you say this is one, you can remove, say, a function which does not tell you one.
2
SPEAKER_01
Say, the best strategy, if you can do it perfectly, is to remove half of the functions.
2
SPEAKER_01
But when you use one predicate, which looks like a duck, you can remove much more functions than half.
2
SPEAKER_01
And that means that it contains a lot of bit of information from a formal point of view.
2
SPEAKER_01
But when you have a general picture of what you want to recognize, a general picture of the world, can you invent this predicate?
2
SPEAKER_01
And that predicate carries a lot of information.
2
SPEAKER_00
Beautifully put.
2
SPEAKER_00
Maybe just me, but in all the math you show, in your work, which is some of the most profound mathematical work in the field of learning AI and just math in general, I hear a lot of poetry and philosophy.
2
SPEAKER_00
You really kind of talk about philosophy of science.
2
SPEAKER_00
There s a poetry and music to a lot of the work you re doing and the way you re thinking about it.
2
SPEAKER_00
Where does that come from?
2
SPEAKER_00
Do you escape to poetry?
2
SPEAKER_00
Do you escape to music?
2
SPEAKER_00
Or not?
2
SPEAKER_01
I think that there exists ground truth.
2
SPEAKER_00
There exists ground truth?
2
SPEAKER_01
Yeah, and that can be seen everywhere.
2
SPEAKER_01
The smart guy, philosopher, sometimes I m surprised how they deep see.
2
SPEAKER_01
Sometimes I see that some of them are completely out of subject.
2
SPEAKER_01
But the ground truth I see in music.
2
SPEAKER_00
Music is the ground truth?
2
SPEAKER_01
Yeah.
2
SPEAKER_01
And in poetry, many poets, they believe that
2
SPEAKER_01
They take dictation.
2
SPEAKER_00
So what piece of music, as a piece of empirical evidence, gave you a sense that they are touching something in the ground truth?
2
SPEAKER_00
It is structure.
2
SPEAKER_00
the structure of it, the math of music.
2
SPEAKER_01
Yeah, because when you re listening to Bach, you see the structure.
2
SPEAKER_01
Very clear, very classic, very simple.
2
SPEAKER_01
And the same in Mozart, when you have axioms in geometry, you have the same feeling.
2
SPEAKER_01
And in poetry, sometimes you see the same.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
And if you look back at your childhood, you grew up in Russia, you maybe were born as a researcher in Russia, you ve developed as a researcher in Russia, you ve came to the United States and a few places.
2
SPEAKER_00
If you look back,
2
SPEAKER_00
What was some of your happiest moments as a researcher?
2
SPEAKER_00
Some of the most profound moments, not in terms of their impact on society, but in terms of their impact on how damn good you feel that day and you remember that moment.
2
SPEAKER_01
You know, every time when you found something,
2
SPEAKER_01
It is great.
2
SPEAKER_01
Every simple thing.
2
SPEAKER_01
But my general feeling is that most of my time was wrong.
2
SPEAKER_01
You should go again, and again, and again, and try to be honest in front of yourself.
2
SPEAKER_01
Not to make interpretation.
2
SPEAKER_01
But try to understand that it s related to ground truth.
2
SPEAKER_01
It is not my blah blah blah interpretation or something like that.
2
SPEAKER_00
But you re allowed to get excited at the possibility of discovery.
2
SPEAKER_00
Oh yeah.
2
SPEAKER_00
You have to double check it.
2
SPEAKER_01
No, but how is it related to another ground truth?
2
SPEAKER_01
Is it just temporary or is it forever?
2
SPEAKER_01
You know, you always have a feeling when you found something, how big is that?
2
SPEAKER_01
So, 20 years ago, when we discovered statistical learning theory, nobody believed, except for one guy, Dudley, from MIT.
2
SPEAKER_01
And then in 20 years, it became fashion.
2
SPEAKER_01
And the same with support vector machines.
2
SPEAKER_01
That s kernel machines.
2
SPEAKER_00
So with support vector machines and learning theory, when you were working on it, you had a sense that you had a sense of the profundity of it, how this seems to be right.
2
SPEAKER_00
This seems to be powerful.
2
SPEAKER_01
Right, absolutely, immediately.
2
SPEAKER_01
I recognized that it will last forever.
2
SPEAKER_01
And now, when I found this invariance story,
2
SPEAKER_01
So I have a feeling that it is completely wrong.
2
SPEAKER_01
Because I have proof that there are no different mechanisms.
2
SPEAKER_01
You can have some, say, cosmetic improvement you can do, but in terms of invariance,
2
SPEAKER_01
You need both variance and statistical learning, and they should work together.
2
SPEAKER_01
But also, I m happy that we can formulate what is intelligence from that, and to separate from technical part.
2
SPEAKER_01
That is completely different.
2
SPEAKER_00
Absolutely.
2
SPEAKER_00
Well, Vladimir, thank you so much for talking today.
2
SPEAKER_00
Thank you.
2
SPEAKER_00
It s an honor.