0
SPEAKER_01
The following is a conversation with Max Tegmark, his third time on the podcast.
0
SPEAKER_01
In fact, his first appearance was episode number one of this very podcast.
0
SPEAKER_01
He is a physicist and artificial intelligence researcher at MIT, co-founder of Future Life Institute, and author of Life 3.0, Being Human in the Age of Artificial Intelligence.
0
SPEAKER_01
Most recently, he s a key figure in spearheading the open letter calling for a six month pause on giant AI experiments like training GPT-4.
0
SPEAKER_01
The letter reads, we re calling for a pause on training of models larger than GPT-4 for six months.
0
SPEAKER_01
This does not imply a pause or ban on all AI research and development or the use of systems that have already been placed on the market.
0
SPEAKER_01
Our call is specific and addresses a very small pool of actors who possess this capability.
0
SPEAKER_01
The letter has been signed by over 50,000 individuals, including 1,800 CEOs and over 1,500 professors.
0
SPEAKER_01
Signatories include Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, Andrew Yang, and many others.
0
SPEAKER_01
This is a defining moment in the history of human civilization, where the balance of power between human and AI begins to shift.
0
SPEAKER_01
And Max s mind and his voice is one of the most valuable and powerful in a time like this.
0
SPEAKER_01
His support, his wisdom, his friendship has been a gift I m forever deeply grateful for.
0
SPEAKER_01
And now, a quick few second mention of each sponsor.
0
SPEAKER_01
Check them out in the description.
0
SPEAKER_01
It s the best way to support this podcast.
0
SPEAKER_01
We got Notion for project and team collaboration, Insight Tracker for biological data, and Indeed for hiring.
0
SPEAKER_01
Choose wisely, my friends.
0
SPEAKER_01
Also, speaking of hiring, if you want to work with our amazing team, we re always hiring, whether it s through Indeed or otherwise, go to lexfriedman.com slash hiring.
0
SPEAKER_01
And now on to the full ad reads.
0
SPEAKER_01
As always, no ads in the middle.
0
SPEAKER_01
I try to make these interesting, but if you must skip them, please still check out our sponsors.
0
SPEAKER_01
I enjoy their stuff.
0
SPEAKER_01
Maybe you will too.
0
SPEAKER_01
This show is brought to you by Notion.
0
SPEAKER_01
I ve spoken endlessly about how amazing Notion is, how everybody, all the cool kids are recommending it for just basic note-taking, but there s so, so much more.
0
SPEAKER_01
It s the collaborative aspect of it, the project management aspect of it, the wikis, the document sharing, all of that, all in a simple, powerful, beautifully designed solution.
0
SPEAKER_01
What can I say?
0
SPEAKER_01
On top of this, there s the Notion AI tool.
0
SPEAKER_01
This is the best integration of large language models into a productivity note-taking tool.
0
SPEAKER_01
There are so many amazing features.
0
SPEAKER_01
I mean, it s just endless.
0
SPEAKER_01
Go to their website.
0
SPEAKER_01
You can generate entire presentations and reports based on a to-do list.
0
SPEAKER_01
You can summarize stuff, you can short stuff, you can generate tables based on the description, you can write a summary, you can expand the text, you can change the style of the text, you can fix spelling, grammar, you can translate, you can use simpler language, more complicated language, change the tone of the voice, make it shorter, longer, like I said, everything.
0
SPEAKER_01
It s just so easy to play around with and all of it, no matter what you re doing, will challenge you to think,
0
SPEAKER_01
how you write.
0
SPEAKER_01
It will challenge you to expand the style of your writing.
0
SPEAKER_01
It will save you a lot of time, of course, but I just think it makes you a better thinker and productive being in this world.
0
SPEAKER_01
And I think that s such a great integration of AI into the productivity workflow.
0
SPEAKER_01
To me, it s not enough for a large language model to be effective.
0
SPEAKER_01
at answering questions and having good dialogue, you have to really integrate it into the workflow.
0
SPEAKER_01
And Notion, better than anybody else I ve seen, has done that.
0
SPEAKER_01
So if that s interesting to you, Notion AI helps you work faster, write better, and think bigger, doing tasks that normally take you hours and just minutes.
0
SPEAKER_01
Try Notion AI for free when you go to notion.com slash lex.
0
SPEAKER_01
That s all lowercase, notion.com slash lex, to try the power of Notion AI today.
0
SPEAKER_01
This show is also brought to you by Insight Tracker, a service I use to track biological data.
0
SPEAKER_01
It s really good to do that kind of thing regularly, to look at all the different markers in your body and to understand what could be made better through lifestyle, through diet changes.
0
SPEAKER_01
It s kind of obvious that decisions about your life should be made based on the data that comes from your own body.
0
SPEAKER_01
Not some kind of population study, although those are good.
0
SPEAKER_01
Not some spiritual guru, although those are good.
0
SPEAKER_01
Not some novel, whether it s Harry Potter or Dostoevsky, although those are sometimes good.
0
SPEAKER_01
Not your relative who says, I heard a guy say that a guy does this thing that is very brosoundly sounding, although sometimes it turns out to be pretty effective.
0
SPEAKER_01
Overall, the best decisions about your life should be based on the things that come from your own body.
0
SPEAKER_01
InsideTracker uses algorithms to analyze your blood data, DNA data, fitness tracker data, all that kind of stuff, to give you recommendations.
0
SPEAKER_01
You should be doing it, you should be doing it regularly, so it s not just a one-time thing, but regularly over time you see what changes led to improvements in the various markers that come from your body.
0
SPEAKER_01
Get special savings for a limited time when you go to insidetracker.com slash flex.
0
SPEAKER_01
This show is also brought to you by Indeed, a hiring website.
0
SPEAKER_01
I think the most important thing in life
0
SPEAKER_01
Not to quote Conan the Barbarian because that would be very inappropriate to quote at this moment.
0
SPEAKER_01
And it s not actually accurate at all as a reflection of what s important in life.
0
SPEAKER_01
It only has comedic value.
0
SPEAKER_01
What I really want to say about what s important in life is the people you surround yourself with.
0
SPEAKER_01
And we spend so much of our time in the workplace seeking solutions to very difficult problems together, passionately pursuing ambitious goals, sometimes impossible goals.
0
SPEAKER_01
That is the source of meaning, a source of happiness for people.
0
SPEAKER_01
And I think part of that happiness comes from the collaboration with other human beings, the sort of professional depth of connection that you have with other human beings, of being together through the grind, and surviving, and accomplishing the goal, or failing in a big epic way.
0
SPEAKER_01
knowing that you have tried together.
0
SPEAKER_01
And so doing that with the right team, I think is one of the most important things in life.
0
SPEAKER_01
So you should surround yourself with the right team.
0
SPEAKER_01
If you re looking to join a team, you should be very selective about that.
0
SPEAKER_01
Or if you re looking to hire a team, you should be very selective about that and use the best tools of the job.
0
SPEAKER_01
I ve used Indeed many, many times throughout my life for the teams I ve led.
0
SPEAKER_01
Don t overspend on hiring.
0
SPEAKER_01
Visit indeed.com slash Lex to start hiring now.
0
SPEAKER_01
That s indeed.com slash Lex.
0
SPEAKER_01
Terms and conditions apply.
0
SPEAKER_01
This is the Lex Friedman Podcast.
0
SPEAKER_01
To support it, please check out our sponsors in the description.
0
SPEAKER_01
And now, dear friends, here s Max Tegmark.
0
SPEAKER_01
You were the first ever guest on this podcast, episode number one.
0
SPEAKER_01
So first of all, Max, I just have to say thank you for giving me a chance.
0
SPEAKER_01
Thank you for starting this journey.
0
SPEAKER_01
It s been an incredible journey.
0
SPEAKER_01
Just thank you for sitting down with me and just acting like I m somebody who matters, that I m somebody who s interesting to talk to.
0
SPEAKER_01
And thank you for doing it.
0
SPEAKER_01
That meant a lot.
0
SPEAKER_00
Thanks to you for putting your heart and soul into this.
0
SPEAKER_00
I know when you delve into controversial topics, it s inevitable to get hit by what Hamlet talks about, the slings and arrows and stuff.
0
SPEAKER_00
And I really admire this.
0
SPEAKER_00
It s in an era, you know, where YouTube videos are too long and now it has to be like a 20 minute TikTok.
0
SPEAKER_00
22nd TikTok clip.
0
SPEAKER_00
It s just so refreshing to see you going exactly against all of the advice and doing these really long form things and the people appreciate it.
0
SPEAKER_00
You know, reality is nuanced.
0
SPEAKER_00
And thanks for.
0
SPEAKER_00
and sharing it that way.
0
SPEAKER_01
So let me ask you again the first question I ve ever asked on this podcast.
0
SPEAKER_01
Episode number one, talking to you, do you think there s intelligent life out there in the universe?
0
SPEAKER_01
Let s revisit that question.
0
SPEAKER_01
Do you have any updates?
0
SPEAKER_01
What s your view when you look out to the stars?
0
SPEAKER_00
So when we look out to the stars,
0
SPEAKER_00
If you define our universe the way most astrophysicists do, not as all of space, but the spherical region of space that we can see with our telescopes from which light has a time to reach us since our Big Bang, I m in the minority.
0
SPEAKER_00
I estimate that we are the only life in this spherical volume that has invented internet radios, gotten our level of tech.
0
SPEAKER_00
And if that s true,
0
SPEAKER_00
then it puts a lot of responsibility on us to not mess this one up, because if it s true, it means that life is quite rare, and we are stewards of this one spark of advanced consciousness, which if we nurture it, then...
0
SPEAKER_00
help it grow, eventually life can spread from here out into much of our universe and we can have this just amazing future.
0
SPEAKER_00
Whereas if we instead are reckless with the technology we build and just snuff it out due to stupidity or infighting, then maybe the rest of cosmic history in our universe is just gonna be a play for empty benches.
0
SPEAKER_00
But I do think that we are actually very likely to get visited by aliens
0
SPEAKER_00
alien intelligence quite soon.
0
SPEAKER_00
But I think we are gonna be building that alien intelligence.
0
SPEAKER_01
So we re going to give birth to an intelligent alien civilization, unlike anything that human, that evolution here on Earth was able to create in terms of the path, the biological path it took.
0
SPEAKER_00
Yeah, and it s going to be much more alien than a cat or even the most exotic animal on the planet right now, because it will not have been created through the usual Darwinian competition where it necessarily cares about self-preservation, is afraid of death, any of those things.
0
SPEAKER_00
The space of alien minds that you can build is just so much vaster than what evolution will give you.
0
SPEAKER_00
And with that also comes a great responsibility for us to make sure that the kind of minds we create are the kind of minds that it s good to create, minds that will share our values and be good for humanity and life, and also create minds that don t suffer.
0
SPEAKER_01
Do you try to visualize the full space of alien minds that AI could be?
0
SPEAKER_01
Do you try to consider all the different kinds of intelligences, sort of generalizing what humans are able to do to the full spectrum of what intelligent creatures, entities could do?
0
SPEAKER_00
I try, but I would say I fail.
0
SPEAKER_00
I mean, it s very difficult for a human mind to,
0
SPEAKER_00
really grapple with something so completely alien, even for us, right?
0
SPEAKER_00
If we just try to imagine, how would it feel if we were completely indifferent towards death or individuality?
0
SPEAKER_00
Even if you just imagine that, for example, you could just copy my knowledge of how to speak Swedish.
0
SPEAKER_00
Boom, now you can speak Swedish.
0
SPEAKER_00
and you could copy any of my cool experiences, and then you could delete the ones you didn t like in your own life, just like that.
0
SPEAKER_00
It would already change quite a lot about how you feel as a human being, right?
0
SPEAKER_00
You probably spend less effort studying things if you just copy them, and you might be less afraid of death, because if the plane you re on starts to crash, you d just be like, oh, shucks, I haven t backed my brain up for four hours.
0
SPEAKER_00
So I m going to lose all these wonderful experiences of this flight.
0
SPEAKER_00
We might also start feeling more compassionate maybe with other people if we can so readily share each other s experiences and our knowledge and feel more like a hive mind.
0
SPEAKER_00
It s very hard, though.
0
SPEAKER_00
I really feel very humble about this.
0
SPEAKER_00
to grapple with it, how it might actually feel.
0
SPEAKER_00
The one thing which is so obvious though, which I think is just really worth reflecting on is because the mind space of possible intelligence is so different from ours, it s very dangerous if we assume they re gonna be like us or anything like us.
0
SPEAKER_01
Well, there s the entirety of human written history has been through poetry, through novels, been trying to describe through philosophy, trying to describe the human condition and what s entailed in it.
0
SPEAKER_01
Like Jessica said, fear of death and all those kinds of things.
0
SPEAKER_01
What is love?
0
SPEAKER_01
And all of that changes if you have a different kind of intelligence.
0
SPEAKER_01
All of it, the entirety, all those poems.
0
SPEAKER_01
They re trying to sneak up to what the hell it means to be human.
0
SPEAKER_01
All of that changes.
0
SPEAKER_01
How AI concerns and existential crises that AI experiences, how that clashes with the human existential crisis, the human condition.
0
SPEAKER_01
That s hard to fathom, hard to predict.
0
SPEAKER_00
It s hard, but it s fascinating to think about also.
0
SPEAKER_00
Even in the best case scenario where we don t lose control over the ever more powerful AI that we re building to other humans whose goals we think are horrible and where we don t lose control to the machines and AI.
0
SPEAKER_00
provides the things we want, even then you get into the questions you touched here.
0
SPEAKER_00
Maybe it s the struggle that it s actually hard to do things is part of the things that gives us meaning as well, right?
0
SPEAKER_00
So for example, I found it so shocking that this new Microsoft GPT-4 commercial that they put together has this woman talking about
0
SPEAKER_00
showing this demo, how she s going to give a graduation speech to her beloved daughter, and she asks GPT-4 to write it.
0
SPEAKER_00
If there s freaking 200 words or so, if I realized that my parents couldn t be bothered struggling a little bit to write 200 words and outsource that to their computer, I would feel really offended, actually.
0
SPEAKER_00
And so I wonder if eliminating too much of the struggle from our existence
0
SPEAKER_00
Do you think that would also take away a little bit of what?
0
SPEAKER_01
It means to be human, yeah.
0
SPEAKER_01
We can t even predict.
0
SPEAKER_01
I had somebody mention to me that they started using Chad GPT with a 3.5 and not 4.0 to write what they really feel to a person.
0
SPEAKER_01
and they have a temper issue, and they re basically trying to get Chad GPT to rewrite it in a nicer way, to get the point across, but rewrite it in a nicer way.
0
SPEAKER_01
So we re even removing the inner asshole from our communication.
0
SPEAKER_01
So there s some positive.
0
SPEAKER_01
aspects of that, but mostly it s just the transformation of how humans communicate.
0
SPEAKER_01
And it s scary because so much of our society is based on this glue of communication.
0
SPEAKER_01
And if we re now using AI as the medium of communication that does the language for us, so much of the emotion that s laden in human communication, so much of the intent,
0
SPEAKER_01
that s going to be handled by outsourced AI, how does that change everything?
0
SPEAKER_01
How does that change the internal state of how we feel about other human beings?
0
SPEAKER_01
What makes us lonely?
0
SPEAKER_01
What makes us excited?
0
SPEAKER_01
What makes us afraid?
0
SPEAKER_01
How we fall in love?
0
SPEAKER_01
All that kind of stuff.
0
SPEAKER_00
Yeah.
0
SPEAKER_00
For me personally, I have to confess, the challenge is one of the things that really makes my life feel meaningful, you know?
0
SPEAKER_00
If I go hike a mountain with my wife, Maya, I don t want to just press a button and be at the top.
0
SPEAKER_00
I want to struggle and come up there sweaty and feel like, wow, we did this in the same way.
0
SPEAKER_00
I want to constantly work on myself to become a better person.
0
SPEAKER_00
If I say something in anger that I regret, I want to go back and
1
SPEAKER_01
and really work on myself rather than just tell an AI from now on, always filter what I write so I don t have to work on myself, because then I m not growing.
1
SPEAKER_02
Yeah, but then again, it could be like with chess.
1
SPEAKER_02
An AI, once it significantly, obviously, supersedes the performance of humans, it will live in its own world and provide maybe a flourishing civilization for humans, but we humans will continue hiking mountains and playing our games, even though AI is so much smarter, so much stronger, so much superior in every single way, just like with chess.
1
SPEAKER_02
I mean, that s one possible hopeful trajectory here, is that humans will continue to human, and AI will just be a kind of, a medium that enables the human experience to flourish.
1
SPEAKER_01
I would phrase that as rebranding ourselves from Homo sapiens to Homo sentiens.
1
SPEAKER_01
Right now, sapiens, the ability to be intelligent, we ve even put it in our species name.
1
SPEAKER_01
We re branding ourselves as the smartest information processing entity on the planet.
1
SPEAKER_01
That s clearly gonna change if AI continues ahead.
1
SPEAKER_01
So maybe we should focus on the experience instead, the subjective experience that we have with homo sentience and say that s what s really valuable, the love, the connection, the other things.
1
SPEAKER_01
Get off our high horses and get rid of this hubris that only we can do integrals.
1
SPEAKER_02
So consciousness, the subjective experience is a fundamental value to what it means to be human.
1
SPEAKER_02
Make that the priority.
1
SPEAKER_01
That feels like a hopeful direction to me, but that also requires more compassion, not just towards other humans because they happen to be the smartest on the planet, but also towards all our other fellow creatures on this planet.
1
SPEAKER_01
And I personally feel right now we re treating a lot of farm animals horribly, for example, and the excuse we re using is, oh, they re not as smart as us.
1
SPEAKER_01
But if we admit that we re not that smart in the grand scheme of things either in the post-AI epoch,
1
SPEAKER_01
then surely we should value the subjective experience of a cow also.
1
SPEAKER_02
Well, allow me to briefly look at the book, which at this point is becoming more and more visionary that you ve written, I guess over five years ago, Life 3.0.
1
SPEAKER_02
So, first of all, 3.0, what s 1.0, what s 2.0, what s 3.0, and how s that vision sort of evolve, the vision in the book evolve to today?
1
SPEAKER_01
Life 1.0 is really dumb, like bacteria, in that it can t actually learn anything at all during the lifetime.
1
SPEAKER_01
The learning just comes from this genetic process from one generation to the next.
1
SPEAKER_01
Life 2.0 is us and other animals which have brains, which can learn during their lifetime a great deal.
1
SPEAKER_01
And
1
SPEAKER_01
you were born without being able to speak English.
1
SPEAKER_01
And at some point you decided, hey, I want to upgrade my software.
1
SPEAKER_01
Let s install an English speaking module.
1
SPEAKER_01
So you did.
1
SPEAKER_01
And Life 3.0, which does not exist yet, can replace not only its software the way we can, but also its hardware.
1
SPEAKER_01
And that s where we re heading towards at high speed.
1
SPEAKER_01
We re already maybe 2.1, because we can put in
1
SPEAKER_01
an artificial knee, pacemaker, et cetera, et cetera.
1
SPEAKER_01
And if Neuralink and other companies succeed, we ll be like 2.2, et cetera.
1
SPEAKER_01
But what the companies trying to build AGI are trying to make is, of course, full 3.0.
1
SPEAKER_01
And you can put that intelligence into something that also has no
1
SPEAKER_02
biological basis whatsoever.
1
SPEAKER_02
So less constraints and more capabilities, just like the leap from 1.0 to 2.0.
1
SPEAKER_02
There is nevertheless, you speaking so harshly about bacteria, so disrespectfully about bacteria, there is still the same kind of magic.
1
SPEAKER_02
there that permeates life 2.0 and 3.0.
1
SPEAKER_02
It seems like maybe the thing that s truly powerful about life, intelligence, and consciousness was already there in 1.0.
1
SPEAKER_02
Is it possible?
1
SPEAKER_01
I think we should be humble and not be so quick
1
SPEAKER_01
make everything binary and say either it s there or it s not, clearly there s a great spectrum.
1
SPEAKER_01
And there is even controversy about whether some unicellular organisms like amoebas can maybe learn a little bit after all.
1
SPEAKER_01
So apologies if I offended any bacteria here.
1
SPEAKER_01
It wasn t my intent.
1
SPEAKER_01
It was more...
1
SPEAKER_01
that I wanted to talk up how cool it is to actually have a brain where you can learn dramatically within your lifetime.
1
SPEAKER_01
Typical human.
1
SPEAKER_01
And the higher up you get from 1.0 to 2.0 to 3.0, the more you become the captain of your own ship, the master of your own destiny, and the less you become a slave to whatever evolution gave you.
1
SPEAKER_01
by upgrading our software, we can be so different from previous generations and even from our parents, much more so than even a bacterium, no offense to them.
1
SPEAKER_01
And if you can also swap out your hardware and take any physical form you want, of course, really the sky s the limit.
1
SPEAKER_02
Yeah, so it accelerates the rate at which you can perform the computation that determines your destiny.
1
SPEAKER_01
Yeah, and I think it s worth,
1
SPEAKER_01
commenting a bit on what you means in this context also, if you swap things out a lot.
1
SPEAKER_01
This is controversial, but my current understanding is that life is best thought of not as a bag of meat or even a bag of
1
SPEAKER_01
elementary particles, but rather as a system which can process information and retain its own complexity, even though nature is always trying to mess it up.
1
SPEAKER_01
So it s all about information processing.
1
SPEAKER_01
And that makes it a lot like something like a wave in the ocean, which is not its water molecules, right?
1
SPEAKER_01
The water molecules bob up and down, but the wave moves forward.
1
SPEAKER_01
It s an information pattern.
1
SPEAKER_01
In the same way, you, Lex, you re not the same atoms as during the first interview you did with me.
1
SPEAKER_01
You ve swapped out most of them, but it s still you.
1
SPEAKER_01
And the information pattern is still there.
1
SPEAKER_01
And if you could swap out your arms and whatever,
1
SPEAKER_01
you can still have this kind of continuity.
1
SPEAKER_01
It becomes much more sophisticated sort of way forward in time where the information lives on.
1
SPEAKER_01
I lost both of my parents since our last podcast, and it actually gives me a lot of solace that this way of thinking about them, they haven t entirely died because a lot of mommy and daddy s, sorry, I m getting a little emotional here, but a lot of their values,
1
SPEAKER_01
and ideas and even jokes and so on, they haven t gone away, right?
1
SPEAKER_01
Some of them live on, I can carry on some of them, and they also live on a lot of other, and a lot of other people.
1
SPEAKER_01
So in this sense, even with Life 2.0, we can, to some extent, already transcend our physical bodies and our death.
1
SPEAKER_01
And
1
SPEAKER_01
particularly if you can share your own information, your own ideas with many others like you do in your podcast, then that s the closest to immortality we can get with our bio-bodies.
1
SPEAKER_02
You carry a little bit of them in you in some sense.
1
SPEAKER_02
Yeah, yeah.
1
SPEAKER_02
Do you miss them?
1
SPEAKER_02
Do you miss your mom and dad?
1
SPEAKER_01
Of course, of course.
1
SPEAKER_02
What did you learn about life from them, if it can take a bit?
1
SPEAKER_01
of a tangent on so many things.
1
SPEAKER_01
For starters, my fascination for math and the physical mysteries of our universe, I got a lot of that from my dad.
1
SPEAKER_01
But I think my obsession for fairly big questions and consciousness and so on, that actually came mostly from my mom and
1
SPEAKER_01
What I got from both of them, which is a very core part of really who I am, I think, is this feeling comfortable with not buying into what everybody else is saying, doing what I think is right,
1
SPEAKER_01
They both very much did their own thing, and sometimes they got flack for it, and they did it anyway.
1
SPEAKER_02
That s why you ve always been an inspiration to me, that you re at the top of your field and you re still willing to...
1
SPEAKER_02
to tackle the big questions in your own way.
1
SPEAKER_02
You re one of the people that represents MIT best to me.
1
SPEAKER_02
You ve always been an inspiration in that.
1
SPEAKER_02
So it s good to hear that you got that from your mom and dad.
1
SPEAKER_01
Yeah, you re too kind.
1
SPEAKER_01
But yeah, I mean, the good reason to do science is because you re really curious, you wanna figure out the truth.
1
SPEAKER_01
If you think,
1
SPEAKER_01
this is how it is, and everyone else says, no, no, that s bullshit, and it s that way, you know, you stick with what you think is true.
1
SPEAKER_01
And even if everybody else keeps thinking it s bullshit, there s a certain...
1
SPEAKER_01
I always root for the underdog when I watch movies.
1
SPEAKER_01
And my dad once, one time, for example, when I wrote one of my craziest papers ever, talking about our universe ultimately being mathematical, which we re not going to get into today, I got this email from a quite famous professor saying, this is not only bullshit, but it s going to ruin your career.
1
SPEAKER_01
You should stop doing this kind of stuff.
1
SPEAKER_01
I sent it to my dad.
1
SPEAKER_01
Do you know what he said?
1
SPEAKER_01
What did he say?
1
SPEAKER_01
He replied with a quote from Dante.
1
SPEAKER_01
follow your own path, and let the people talk.
1
SPEAKER_01
Go dad!
1
SPEAKER_01
This is the kind of thing.
1
SPEAKER_01
He s dead, but that attitude is not.
1
SPEAKER_02
How did losing them as a man, as a human being change you?
1
SPEAKER_02
How did it expand your thinking about the world?
1
SPEAKER_02
How did it expand your thinking about this thing we re talking about, which is humans creating another living, sentient, perhaps, being?
1
SPEAKER_00
I think it mainly did two things.
1
SPEAKER_01
One of them, just going through all their stuff after they had passed away and so on, just drove home to me how important it is to ask ourselves, why are we doing these things we do?
1
SPEAKER_01
Because it s inevitable that you look at some things they spent an enormous time on and you ask,
1
SPEAKER_01
In hindsight, would they really have spent so much time on this or would they have done something that was more meaningful?
1
SPEAKER_01
So I ve been looking more in my life now and asking, you know, why am I doing what I m doing?
1
SPEAKER_01
And I feel it should either be something I really enjoy doing or it should be something that I find really, really meaningful because it helps humanity.
1
SPEAKER_01
And
1
SPEAKER_01
It s in none of those two categories.
1
SPEAKER_01
Maybe I should spend less time on it, you know.
1
SPEAKER_01
The other thing is dealing with death up in person like this, it s actually made me less afraid of, even less afraid of other people telling me that I m an idiot, you know, which happens regularly.
1
SPEAKER_01
And just, I m gonna live my life, do my thing, you know.
1
SPEAKER_01
And,
1
SPEAKER_01
It s made it a little bit easier for me to focus on what I feel is really important.
1
SPEAKER_02
What about fear of your own death?
1
SPEAKER_02
Has it made it more real that this is something that happens?
1
SPEAKER_01
Yeah, it s made it extremely real.
1
SPEAKER_01
I m next in line in our family now, right?
1
SPEAKER_01
It s me and my younger brother.
1
SPEAKER_01
They both handled it with such dignity.
1
SPEAKER_01
That was a true inspiration also.
1
SPEAKER_01
They never complained about things.
1
SPEAKER_01
When you re old and your body starts falling apart, it s more and more to complain about.
1
SPEAKER_01
They looked at what could they still do that was meaningful, and they focused on that rather than wasting time.
1
SPEAKER_01
talking about or even thinking much about things they were disappointed in.
1
SPEAKER_01
I think anyone can make themselves depressed if they start their morning by making a list of grievances, whereas if you start your day with a little meditation and just things you re grateful for, you basically choose to be a happy person.
1
SPEAKER_01
Because you only have a finite number of days you should spend them.
1
SPEAKER_01
Make it count.
1
SPEAKER_02
Being grateful.
1
SPEAKER_02
Well, you do happen to be working on a thing which seems to have potentially some of the greatest impact on human civilization of anything humans have ever created, which is artificial intelligence.
1
SPEAKER_02
This is on the both detailed technical level and in the high philosophical level you work on.
1
SPEAKER_02
So you ve mentioned to me that there s an open letter that you re working on.
1
SPEAKER_01
It s actually going live in a few hours.
1
SPEAKER_01
I ve been having late nights and early mornings.
1
SPEAKER_01
It s been very exciting, actually.
1
SPEAKER_01
In short, have you seen Don t Look Up, the film?
1
SPEAKER_01
Yes, yes.
1
SPEAKER_01
I don t want to be the movie spoiler for anyone watching this who hasn t seen it.
1
SPEAKER_01
But if you re watching this, you haven t seen it.
1
SPEAKER_01
Watch it.
1
SPEAKER_01
Because we are actually acting out it s it s life imitating art.
1
SPEAKER_01
Humanity is doing exactly that right now except
1
SPEAKER_01
It s an asteroid that we are building ourselves.
1
SPEAKER_01
Almost nobody is talking about it.
1
SPEAKER_01
People are squabbling across the planet about all sorts of things which seem very minor compared to the asteroid that s about to hit us, right?
1
SPEAKER_01
Most politicians don t even have this on their radar.
1
SPEAKER_01
They think maybe in 100 years or whatever.
1
SPEAKER_01
Right now, we re at a fork in the road.
1
SPEAKER_01
This is the most important fork humanity has reached in its over 100,000 years on this planet.
1
SPEAKER_01
Effectively, a new species that s smarter than us.
1
SPEAKER_01
It doesn t look so much like a species yet because it s mostly not embodied in robots, but that s a technicality which will soon be changed.
1
SPEAKER_01
And this arrival of artificial general intelligence that can do all our jobs as well as us and
1
SPEAKER_01
probably shortly thereafter superintelligence, which greatly exceeds our cognitive abilities.
1
SPEAKER_01
It s gonna either be the best thing ever to happen to humanity or the worst.
1
SPEAKER_01
I m really quite confident that there is not that much middle ground there.
1
SPEAKER_02
But it would be fundamentally transformative to human civilization.
1
SPEAKER_01
Of course, utterly and totally.
1
SPEAKER_01
Again, we branded ourselves as homo sapiens, because it seemed like the basic thing.
1
SPEAKER_01
We re the king of the castle on this planet.
1
SPEAKER_01
We re the smart ones.
1
SPEAKER_01
We can control everything else.
1
SPEAKER_01
This could very easily change.
1
SPEAKER_01
We re certainly not going to be the smartest on the planet for very long, unless AI progress just halts.
1
SPEAKER_01
And we can talk more about why I think that s true, because it s controversial.
1
SPEAKER_01
And then we can also talk about
1
SPEAKER_01
reasons you might think it s gonna be the best thing ever, and the reason you think it s gonna be the end of humanity, which is, of course, super controversial.
1
SPEAKER_01
But what I think we can, anyone who s working on advanced AI can agree on is it s much like the film, Don t Look Up, in that it s just really comical how little serious public debate there is about it, given how huge it is.
1
SPEAKER_01
So what we re talking about...
2
SPEAKER_01
is a development of currently things like GPT-4, and the signs it s showing of rapid improvement that may in the near term lead to development of super intelligent AGI, AI, general AI systems, and what kind of impact that has on society.
2
SPEAKER_01
When that thing achieves general human level intelligence, and then beyond that, general super human level intelligence.
2
SPEAKER_01
There s a lot of questions to explore here.
2
SPEAKER_01
So one, you mentioned HALT.
2
SPEAKER_01
Is that the content of the letter?
2
SPEAKER_01
It s to suggest that maybe we should pause the development of these systems.
2
SPEAKER_01
Exactly.
2
SPEAKER_00
So this is very controversial.
2
SPEAKER_00
When we talked the first time, we talked about how I was involved in starting the Future Life Institute, and what we worked very hard on 2014, 2015, was the mainstream AI safety.
2
SPEAKER_00
the idea that there even could be risks and that you could do things about them.
2
SPEAKER_00
Before then, a lot of people thought it was just really kooky to even talk about it.
2
SPEAKER_00
And a lot of AI researchers felt worried that this was too flaky and could be bad for funding and that the people who talked about it were just not, didn t understand AI.
2
SPEAKER_00
I m very, very happy with
2
SPEAKER_00
how that s gone and that now, you know, it s completely mainstream.
2
SPEAKER_00
You go on any AI conference and people talk about AI safety and it s a nerdy technical field full of equations and similar and blah, blah.
2
SPEAKER_00
As it should be.
2
SPEAKER_00
But there s this other thing, which has been quite taboo up until now, calling for slowdown.
2
SPEAKER_00
So what
2
SPEAKER_00
we ve constantly been saying, including myself, I ve been biting my tongue a lot, you know, is that, you know, we don t need to slow down AI development.
2
SPEAKER_00
We just need to win this race, the wisdom race between the growing power of the AI and the growing wisdom with which we manage it.
2
SPEAKER_00
And rather than trying to slow down AI, let s just try to accelerate the wisdom.
2
SPEAKER_00
do all this technical work to figure out how you can actually ensure that your powerful AI is going to do what you want it to do and have society adapt also with incentives and regulations so that these things get put to good use.
2
SPEAKER_00
Sadly, that didn t pan out.
2
SPEAKER_00
The progress on technical AI and capabilities has gone a lot faster than many people thought back when we started this in 2014.
2
SPEAKER_00
Turned out to be easier to build really advanced AI than we thought.
2
SPEAKER_00
And on the other side, it s gone much slower than we hoped with getting
2
SPEAKER_00
policy makers and others to actually put incentives in place to steer this in the good direction.
2
SPEAKER_00
Maybe we should unpack it and talk a little bit about each.
2
SPEAKER_00
Why did it go faster than a lot of people thought?
2
SPEAKER_00
In hindsight, it s exactly like building flying machines.
2
SPEAKER_00
People spent a lot of time wondering about how do birds fly?
2
SPEAKER_00
And that turned out to be really hard.
2
SPEAKER_00
Have you seen the TED Talk with a flying bird?
2
SPEAKER_00
Like a flying robotic bird?
2
SPEAKER_00
Yeah, it flies around the audience.
2
SPEAKER_00
But it took 100 years longer to figure out how to do that than for the Wright brothers to build the first airplane because it turned out there was a much easier way to fly.
2
SPEAKER_00
And evolution picked the more complicated one because it had its hands tied.
2
SPEAKER_00
It could only build a machine that could assemble itself.
2
SPEAKER_00
which the Wright brothers didn t care about.
2
SPEAKER_00
They can only build a machine that used only the most common atoms in the periodic table.
2
SPEAKER_00
Wright brothers didn t care about that.
2
SPEAKER_00
They could use steel, iron atoms, and it had to be able to repair itself, and it also had to be incredibly fuel efficient.
2
SPEAKER_00
A lot of birds use less than half the fuel of a remote control plane flying the same distance.
2
SPEAKER_00
We humans put a little more fuel in a roof, there you go, 100 years earlier.
2
SPEAKER_00
That s exactly what s happening now with these large language models.
2
SPEAKER_00
The brain is incredibly complicated.
2
SPEAKER_00
Many people made the mistake of thinking we have to figure out how the brain does human level AI first before we could build in a machine.
2
SPEAKER_00
That was completely wrong.
2
SPEAKER_00
You can take an incredibly simple computational system called a transformer network and just train it to do something incredibly dumb, just read a gigantic amount of text and try to predict the next word.
2
SPEAKER_00
And it turns out, if you just throw a ton of compute at that and a ton of data, it gets to be frighteningly good, like GPT-4, which I ve been playing with so much since it came out, right?
2
SPEAKER_00
There s still some debate about whether that can get you all the way to full human level or not.
2
SPEAKER_00
But yeah, we can come back to the details of that and how you might get the human level AI even if large language models don t.
2
SPEAKER_01
Can you briefly, if it s just a small tangent, comment on your feelings about GPT-4?
2
SPEAKER_01
Suggest that you re impressed by this rate of progress, but where is it?
2
SPEAKER_01
Can GPT-4 reason?
2
SPEAKER_01
What are the intuitions?
2
SPEAKER_01
What are human interpretable words you can assign to the capabilities of GPT-4 that makes you so damn impressed with it?
2
SPEAKER_00
I m both very excited about it and terrified.
2
SPEAKER_00
interesting mixture of emotions.
2
SPEAKER_00
All the best things in life include those two somehow.
2
SPEAKER_00
Yeah, I can absolutely reason.
2
SPEAKER_00
Anyone who hasn t played with it, I highly recommend doing that before dissing it.
2
SPEAKER_00
It can do quite remarkable reasoning.
2
SPEAKER_00
I ve had to do a lot of things, which I realized I couldn t do that myself that well, even.
2
SPEAKER_00
And obviously does it dramatically faster than we do, too, when you watch it type.
2
SPEAKER_00
And it s doing that while servicing a massive number of other humans at the same time.
2
SPEAKER_00
At the same time, it cannot reason
2
SPEAKER_00
as well as a human can on some tasks.
2
SPEAKER_00
It s obviously a limitation from its architecture.
2
SPEAKER_00
We have in our heads what in geek speak is called a recurrent neural network.
2
SPEAKER_00
There are loops.
2
SPEAKER_00
Information can go from this neuron to this neuron to this neuron and then back to this one.
2
SPEAKER_00
You can like ruminate on something for a while.
2
SPEAKER_00
You can self-reflect a lot.
2
SPEAKER_00
These large language models, they cannot make GPT-4.
2
SPEAKER_00
It s a so-called transformer, where it s just like a one-way street of information, basically.
2
SPEAKER_00
In geek speak, it s called a feed-forward neural network.
2
SPEAKER_00
And it s only so deep, so it can only do logic that s that many steps and that deep.
2
SPEAKER_00
And you can create problems which will fail to solve for that reason.
2
SPEAKER_00
But the fact that it can do so amazing things with this incredibly simple architecture already is quite stunning.
2
SPEAKER_00
And what we see in my lab at MIT, when we look inside large language models to try to figure out how they re doing it, that s the key core focus of our research.
2
SPEAKER_00
It s called mechanistic interpretability in GeekSpeak.
2
SPEAKER_00
You have this machine that does something smart, you try to reverse engineer it, see how does it do it?
2
SPEAKER_00
I think of it also as artificial neuroscience.
2
SPEAKER_00
That s exactly what neuroscientists do with actual brains.
2
SPEAKER_00
But here you have the advantage that you don t have to worry about measurement errors.
2
SPEAKER_00
You can see what every neuron is doing all the time.
2
SPEAKER_00
And a recurrent thing we see again and again, there s been a number of beautiful papers quite recently by a lot of researchers, some of them here, even in this area, is where when they figure out how something is done,
2
SPEAKER_00
You can say, oh man, that s such a dumb way of doing it.
2
SPEAKER_00
And you immediately see how it can be improved.
2
SPEAKER_00
Like, for example, there was a beautiful paper recently where they figured out how a large language model stores certain facts, like Eiffel Tower is in Paris.
2
SPEAKER_00
And they figured out exactly how it s stored.
2
SPEAKER_00
And the proof that they understood it was they could edit it.
2
SPEAKER_00
They changed some of the synapses in it.
2
SPEAKER_00
And then they asked it, where s the Eiffel Tower?
2
SPEAKER_00
And it said, it s in Rome.
2
SPEAKER_00
And then they asked you, how do you get there?
2
SPEAKER_00
Oh, how do you get there from Germany?
2
SPEAKER_00
Oh, you take this train to Roma Termini train station, and this and that.
2
SPEAKER_00
And what might you see if you re in front of it?
2
SPEAKER_00
Oh, you might see the Colosseum.
2
SPEAKER_00
So they had edited it.
2
SPEAKER_00
So they literally moved it to Rome.
2
SPEAKER_00
But the way it s storing this information, it s incredibly dumb.
2
SPEAKER_00
For any fellow nerds listening to this, there was a big matrix.
2
SPEAKER_00
And roughly speaking, there are certain row and column vectors which encode these things, and they correspond very hand-wavily to principal components.
2
SPEAKER_00
And it would be much more efficient for a sparse matrix to store in the database, you know.
2
SPEAKER_00
And everything so far we ve figured out how these things do are ways where you can see they can easily be improved.
2
SPEAKER_00
And the fact that this particular architecture has some roadblocks built into it is in no way going to prevent
2
SPEAKER_00
crafty researchers from quickly finding workarounds and making other kinds of architectures sort of go all the way.
2
SPEAKER_00
So in short, it s turned out to be a lot easier to build close to human intelligence than we thought.
2
SPEAKER_00
And that means our runway as a species.
2
SPEAKER_00
get our shit together, has shortened.
2
SPEAKER_01
And it seems like the scary thing about the effectiveness of large language models, so Sam Altman I recently had a conversation with, and he really showed that the leap from GPT-3 to GPT-4 has to do with just a bunch of hacks, a bunch of,
2
SPEAKER_01
little explorations with smart researchers doing a few little fixes here and there.
2
SPEAKER_01
So that s some fundamental leap and transformation in the architecture.
2
SPEAKER_00
And more data and more compute.
2
SPEAKER_01
and more data and compute, but he said the big leaps has to do with not the data and the compute, but just learning this new discipline, just like you said.
2
SPEAKER_01
So researchers are going to look at these architectures, and there might be big leaps where you realize, wait, why are we doing this in this dumb way?
2
SPEAKER_01
And all of a sudden, this model s 10x smarter.
2
SPEAKER_01
And that can happen on any one day, on any one Tuesday or Wednesday afternoon.
2
SPEAKER_01
And then all of a sudden, you have a system that s 10x smarter.
2
SPEAKER_01
It seems like it s such a new discipline.
2
SPEAKER_01
It s such a new, like we understand so little about why this thing works so damn well that the linear improvement of compute or exponential, but the steady improvement of compute, steady improvement of the data may not be the thing that even leads to the next leap.
2
SPEAKER_01
It could be a surprise little hack that improves everything.
2
SPEAKER_00
Or a lot of little leaps here and there because so much of this is out in the open also.
2
SPEAKER_00
So many smart people are looking at this and trying to figure out little leaps here and there, and it becomes this sort of collective race where a lot of people feel, if I don t take the leap, someone else will.
2
SPEAKER_00
And this is actually very crucial for the other part of it.
2
SPEAKER_00
Why do we want to slow this down?
2
SPEAKER_00
So again, what this open letter is calling for is just pausing all training of systems that are more powerful than GPT-4 for six months.
2
SPEAKER_00
give a chance for the labs to coordinate a bit on safety and for society to adapt, give the right incentives to the labs.
2
SPEAKER_00
You ve interviewed a lot of these people who lead these labs, and you know just as well as I do that they re good people.
2
SPEAKER_00
They re idealistic people.
2
SPEAKER_00
They re doing this first and foremost because they believe that AI has a huge potential to help humanity.
2
SPEAKER_00
But at the same time, they are trapped in this horrible race to the bottom.
2
SPEAKER_00
Have you read Meditations on Moloch by Scott Alexander?
2
SPEAKER_00
Yeah, it s a beautiful essay on this poem by Ginzburg where he interprets it as being about this monster
2
SPEAKER_00
It s this game theory monster that pits people against each other and they race to the bottom where everybody ultimately loses.
2
SPEAKER_00
The evil thing about this monster is even though everybody sees it and understands, they still can t get out of the race, right?
2
SPEAKER_00
A good fraction of all the bad things that we humans do are caused by Moloch.
2
SPEAKER_00
And I like Scott Alexander s
2
SPEAKER_00
naming of the monster so we humans can think of it as a thing.
2
SPEAKER_00
If you look at why do we have overfishing, why do we have, more generally, the tragedy of the commons, why is it that, so Liv Borea, I don t know if you had her on your podcast.
2
SPEAKER_01
Yeah, she s become a friend, yeah.
2
SPEAKER_00
Great, she made this awesome point recently that beauty filters, that a lot of female
2
SPEAKER_00
influencers feel pressure to use are exactly Moloch in action again.
2
SPEAKER_00
First, nobody was using them and people saw them just the way they were.
2
SPEAKER_00
And then some of them started using it and becoming ever more plastic fantastic.
2
SPEAKER_00
And then the other ones that weren t using it started to realize that if they want to just keep their, their market share, they have to start using it too.
2
SPEAKER_00
And that, and then you re in the situation where they re all using it.
2
SPEAKER_00
And none of them has any more market share or less than before.
2
SPEAKER_00
So nobody gained anything.
2
SPEAKER_00
Everybody lost and they have to keep becoming ever more plastic fantastic also.
2
SPEAKER_00
Right.
2
SPEAKER_00
And, uh, but nobody can go back to the old way because it s just too costly.
2
SPEAKER_00
Right.
2
SPEAKER_00
Moloch is everywhere.
2
SPEAKER_00
And, um,
2
SPEAKER_00
Moloch is not a new arrival on the scene either.
2
SPEAKER_00
We humans have developed a lot of collaboration mechanisms to help us fight back against Moloch through various kinds of constructive collaboration.
2
SPEAKER_00
The Soviet Union and the United States did sign a number of arms control treaties against Moloch, who was trying to stoke them into unnecessarily risky nuclear arms races, et cetera, et cetera.
2
SPEAKER_00
And this is exactly what s happening on the AI front this time.
2
SPEAKER_00
It s a little bit geopolitics, but it s mostly money where there s just so much commercial pressure.
2
SPEAKER_00
You know, if you take any of these leaders of the top tech companies, if they just say, you know, this is too risky, I want to pause for six months, they re going to get a lot of pressure from shareholders and others.
2
SPEAKER_00
We re like, well, you know, if you pause, but those guys don t pause, we re
2
SPEAKER_00
We don t want to get our lunch eaten.
2
SPEAKER_00
And shareholders even have the power to replace the executives in the worst case, right?
2
SPEAKER_00
So we did this open letter because we want to help these idealistic tech executives to do what their heart tells them by providing enough public pressure on the whole sector to just pause so that they can all pause in a coordinated fashion.
2
SPEAKER_00
And I think without the public pressure, none of them can do it alone.
2
SPEAKER_00
push back against their shareholders, no matter how good-hearted they are.
2
SPEAKER_00
Moloch is a really powerful foe.
2
SPEAKER_01
So the idea is to, for the major developers of AI systems like this, so we re talking about Microsoft, Google, Meta,
2
SPEAKER_00
And anyone else?
2
SPEAKER_00
Well, OpenAI is very close with Microsoft now, of course, and there are plenty of smaller players.
2
SPEAKER_00
For example, Anthropic is very impressive.
2
SPEAKER_00
There s Conjecture.
2
SPEAKER_00
There s many, many, many players.
2
SPEAKER_00
I don t want to make a long list to leave anyone out.
2
SPEAKER_00
And for that reason, it s so important that some coordination happens, that there s external pressure on all of them, saying you all need to pause,
2
SPEAKER_00
Because then the people, the researchers in these organizations, the leaders who want to slow down a little bit, they can say to their shareholders, you know, everybody s slowing down because of this pressure and it s the right thing to do.
2
SPEAKER_01
Have you seen in history, there are examples where it s possible to pause the molecule?
2
SPEAKER_00
Yes, absolutely.
2
SPEAKER_00
Like human cloning, for example.
2
SPEAKER_00
You could make so much money on human cloning.
2
SPEAKER_00
Why aren t we doing it?
2
SPEAKER_00
Because biologists thought hard about this and felt like this is way too risky.
3
SPEAKER_01
They got together in the 70s in Asilomar and decided even to stop a lot more stuff also, just editing the human germline, gene editing that goes in to our offspring, and decided, let s not do this because it s too unpredictable what it s going to lead to.
3
SPEAKER_01
we could lose control over what happens to our species.
3
SPEAKER_01
So they paused.
3
SPEAKER_01
There was a ton of money to be made there.
3
SPEAKER_01
So it s very doable, but you need a public awareness of what the risks are and the broader community coming in and saying, hey, let s slow down.
3
SPEAKER_01
And another common pushback I get today is we can t stop in the West because China,
3
SPEAKER_01
And in China, undoubtedly, they also get told, we can t slow down because the West, because both sides think they re the good guy.
3
SPEAKER_01
But look at human cloning.
3
SPEAKER_01
Did China forge ahead with human cloning?
3
SPEAKER_01
There s been exactly one human cloning that s actually been done that I know of.
3
SPEAKER_01
It was done by a Chinese guy.
3
SPEAKER_01
Do you know where he is now?
3
SPEAKER_01
In jail.
3
SPEAKER_01
And do you know who put him there?
3
SPEAKER_01
Chinese government.
3
SPEAKER_01
Not because Westerners said, China, look, this is... No, the Chinese government put them there because they also felt
3
SPEAKER_01
They like control, the Chinese government.
3
SPEAKER_01
If anything, maybe they are even more concerned about having control than Western governments, have no incentive of just losing control over where everything is going.
3
SPEAKER_01
And you can also see the Ernie bot that was released by, I believe, Baidu recently.
3
SPEAKER_01
They got a lot of pushback from the government and had to rein it in, you know, in a big way.
3
SPEAKER_01
I think once this basic message comes out that this isn t an arms race, it s a suicide race,
3
SPEAKER_01
where everybody loses if anybody s AI goes out of control.
3
SPEAKER_01
It really changes the whole dynamic.
3
SPEAKER_01
I ll say this again, because this is a very basic point I think a lot of people get wrong.
3
SPEAKER_01
Because a lot of people dismiss the whole idea that AI can really get,
3
SPEAKER_01
very superhuman because they think there s something really magical about intelligence such that it can only exist in human minds.
3
SPEAKER_01
Because they believe that, I think it s going to kind of get to just more or less GPT-4++ and then that s it.
3
SPEAKER_01
They don t see it as a suicide race.
3
SPEAKER_01
They think whoever gets that first, they re going to control the world, they re going to win.
3
SPEAKER_01
That s not how it s going to be.
3
SPEAKER_01
And we can talk again about the scientific arguments from why it s not going to stop there.
3
SPEAKER_01
The way it s going to be is if anybody completely loses control and, you know, you don t care if someone manages to take over the world who really doesn t share your goals, you probably don t really even care very much about what nationality they have.
3
SPEAKER_01
You re not going to like it, much worse than today.
3
SPEAKER_01
If you live in Orwellian dystopia, what do you care who created it, right?
3
SPEAKER_01
And if someone, if it goes farther and we just lose control even to the machines, so that it s not us versus them, it s us versus it.
3
SPEAKER_01
What do you care who created this underlying entity which has goals different from humans ultimately and we get marginalized, we get made obsolete, we get replaced.
3
SPEAKER_01
That s what I mean when I say it s a suicide race.
3
SPEAKER_01
It s kind of like we re rushing towards this cliff.
3
SPEAKER_01
But the closer to the cliff we get, the more scenic the views are and the more money there is there.
3
SPEAKER_01
So we keep going.
3
SPEAKER_01
But we have to also stop at some point, right?
3
SPEAKER_01
Quit while we re ahead.
3
SPEAKER_01
It s a suicide race, which cannot be won.
3
SPEAKER_01
But the way to really benefit from it is
3
SPEAKER_01
continue developing awesome AI, a little bit slower, so we make it safe, make sure it does the things that humans want, and create a condition where everybody wins.
3
SPEAKER_01
Technology has shown us that geopolitics and politics in general is not a zero-sum game at all.
3
SPEAKER_00
So there is some rate of development that will lead
3
SPEAKER_00
us as a human species to lose control of this thing.
3
SPEAKER_00
And the hope you have is that there s some lower level of development which will not allow us to lose control.
3
SPEAKER_00
This is an interesting thought you have about losing control.
3
SPEAKER_00
So if you have somebody, if you are somebody like Sander Parchai or Sam Altman at the head of a company like this, you re saying if they develop an AGI, they too will lose control of it.
3
SPEAKER_00
So no one person can maintain control.
3
SPEAKER_00
No group of individuals can maintain control.
3
SPEAKER_01
If it s created very, very soon and is a big black box that we don t understand, like the large language models, yeah, then I m very confident they re gonna lose control.
3
SPEAKER_01
But this isn t just me saying it.
3
SPEAKER_01
Sam Altman and Demis Hassabis have both said that.
3
SPEAKER_01
They themselves acknowledge that there s really great risks with this, and they wanna slow down once they feel like it s scary.
3
SPEAKER_01
But it s clear that they re stuck, and again, Moloch is forcing them to go a little faster than they re comfortable with because of pressure from, just commercial pressures, right?
3
SPEAKER_01
To get a bit optimistic here, of course, this is a problem that can be ultimately solved.
3
SPEAKER_01
It s just, to win this wisdom race,
3
SPEAKER_01
It s clear that what we hoped was going to happen hasn t happened.
3
SPEAKER_01
The capability progress has gone faster than a lot of people thought, and the progress in the public sphere of policymaking and so on has gone slower than we thought.
3
SPEAKER_01
Even the technical AI safety has gone slower.
3
SPEAKER_01
A lot of the technical safety research was kind of banking on that
3
SPEAKER_01
large language models and other poorly understood systems couldn t get us all the way, that you had to build more of a kind of intelligence that you could understand, maybe it could prove itself safe, you know, things like this.
3
SPEAKER_01
I m quite confident that this can be done, so we can reap all the benefits, but we cannot do it as quickly as this out-of-control express train we are on now is gonna get to AGI.
3
SPEAKER_01
That s why we need a little more time, I feel.
3
SPEAKER_00
Is there something to be said, like Sam Altman talked about, which is while we re in the pre-AGI stage, to release often and as transparently as possible, to learn a lot.
3
SPEAKER_00
So as opposed to being extremely cautious, release a lot.
3
SPEAKER_00
Don t invest in a closed development where you focus on AI safety.
3
SPEAKER_00
While it s somewhat dumb, quote unquote, release as often as possible.
3
SPEAKER_00
And as you start to see signs of human-level intelligence or superhuman-level intelligence, then you put a halt on it.
3
SPEAKER_01
Well, what a lot of safety researchers have been saying for many years is that the most dangerous things you can do with an AI is, first of all, teach it to write code, because that s the first step towards recursive self-improvement, which can take it from AGI to much higher levels.
3
SPEAKER_01
Okay, oops, we ve done that.
3
SPEAKER_01
And another thing, high risk is connect it to the internet, let it go to websites, download stuff on its own, talk to people.
3
SPEAKER_01
Oops, we ve done that already.
3
SPEAKER_01
You know, Eliezer Yudkowsky, you said you interviewed him recently, right?
3
SPEAKER_01
He had this tweet recently, which gave me one of the best laughs in a while, where he was like, hey, people used to make fun of me and say, you re so stupid, Eliezer, because you re saying,
3
SPEAKER_01
You have to worry.
3
SPEAKER_01
Obviously, developers, once they get to really strong AI, first thing you re gonna do is never connect it to the internet, keep it in a box where you can really study it.
3
SPEAKER_01
So he had written it in the meme form, so it s like then, and then that, and then now.
3
SPEAKER_01
LOL, let s make a chatbot.
3
SPEAKER_01
And the third thing, Stuart Russell,
3
SPEAKER_01
amazing AI researcher, he has argued for a while that we should never teach AI anything about humans.
3
SPEAKER_01
Above all, we should never let it learn about human psychology and how you manipulate humans.
3
SPEAKER_01
That s the most dangerous kind of knowledge you can give it.
3
SPEAKER_01
Yeah, you can teach it all it needs to know about how to cure cancer and stuff like that, but don t let it read Daniel Kahneman s book about cognitive biases and all that.
3
SPEAKER_01
oops, LOL, let s invent social media recommender algorithms, which do exactly that.
3
SPEAKER_01
They get so good at knowing us and pressing our buttons that we re starting to create a world now where we just have ever more hatred, because they figured out that these algorithms, not out of evil, but just to make money on advertising, that the best way to get more engagement, the euphemism,
3
SPEAKER_01
get people glued to their little rectangles.
3
SPEAKER_01
It s just to make them pissed off.
3
SPEAKER_00
That s really interesting that a large AI system that s doing the recommender system kind of task on social media is basically just studying human beings because it s a bunch of us rats giving it signal.
3
SPEAKER_00
non-stop signal.
3
SPEAKER_00
It ll show a thing, and then we give signal, and whether we spread that thing, we like that thing, that thing increases our engagement, gets us to return to the platform.
3
SPEAKER_00
It has that on the scale of hundreds of millions of people constantly.
3
SPEAKER_00
So it s just learning and learning and learning.
3
SPEAKER_00
And presumably, if the number of parameters in the neural network that s doing the learning, the more end-to-end the learning is,
3
SPEAKER_00
the more it s able to just basically encode how to manipulate human behavior, how to control humans at scale.
3
SPEAKER_01
Exactly, and that is not something I think is in humanity s interest.
3
SPEAKER_01
Right now, it s mainly letting some humans manipulate other humans for profit and power, which already caused a lot of damage, and eventually that s a sort of,
3
SPEAKER_01
skill that can make AIs persuade humans to let them escape, whatever safety precautions we have.
3
SPEAKER_01
But there was a really nice article in the New York Times recently by Yuval Noah Harari and two co-authors, including Tristan Harris from The Social Dilemma.
3
SPEAKER_01
They have this phrase in there I love.
3
SPEAKER_01
Humanity s first contact with advanced AI was social media.
3
SPEAKER_01
And we lost that one.
3
SPEAKER_01
We now live in a country where there s much more hate in the world, where there s much more hate, in fact.
3
SPEAKER_01
And in our democracy that we re having this conversation and people can t even agree on who won the last election.
3
SPEAKER_01
And we humans often point fingers at other humans and say it s their fault.
3
SPEAKER_01
But it s really Moloch and these AI algorithms.
3
SPEAKER_01
We got the algorithms and then Moloch
3
SPEAKER_01
pitted the social media companies against each other, so nobody could have a less creepy algorithm, because then they would lose out on revenue to the other company.
3
SPEAKER_00
Is there any way to win that battle back, just if we just linger on this one battle that we ve lost in terms of social media?
3
SPEAKER_00
Is it possible to redesign social media, this very medium in which we use as a civilization to communicate with each other?
3
SPEAKER_00
to have these kinds of conversations, to have discourse, to try to figure out how to solve the biggest problems in the world, whether that s nuclear war or the development of AGI.
3
SPEAKER_00
Is it possible to do social media correctly?
3
SPEAKER_01
I think it s not only possible, but it s necessary.
3
SPEAKER_01
Who are we kidding that we re gonna be able to solve all these other challenges if we can t even have a conversation with each other?
3
SPEAKER_01
It s constructive.
3
SPEAKER_01
The whole idea, the key idea of democracy is that you get a bunch of people together, and they have a real conversation, the ones you try to foster on this podcast, where you respectfully listen to people you disagree with.
3
SPEAKER_01
And you realize, actually, you know, there are some things, actually, some common ground we have, and we both agree, let s not have a nuclear war, let s not do that, et cetera, et cetera.
3
SPEAKER_01
We re kidding ourselves thinking we can face off the second contact with ever more powerful AI that s happening now with these large language models if we can t even have a functional conversation in the public space.
3
SPEAKER_01
That s why I started the Improve the News project, improvethenews.org.
3
SPEAKER_01
But I m an optimist fundamentally in,
3
SPEAKER_01
and that there is a lot of intrinsic goodness in people, and that what makes the difference between someone doing good things for humanity and bad things is not...
3
SPEAKER_01
some sort of fairy tale thing that this person was born with an evil gene and this one was not born with a good gene.
3
SPEAKER_01
No, I think it s whether we put, whether people find themselves in situations that bring out the best in them or that bring out the worst in them.
3
SPEAKER_01
And I feel we re building an internet and a society that brings out the worst
3
SPEAKER_00
But it doesn t have to be that way.
3
SPEAKER_00
No, it does not.
3
SPEAKER_00
It s possible to create incentives and also create incentives that make money, that both make money and bring out the best in people.
3
SPEAKER_01
I mean, in the long term, it s not a good investment for anyone to have a nuclear war, for example.
3
SPEAKER_01
And, you know, is it a good investment for humanity if we just ultimately replace all humans by machines and then we re so obsolete that eventually there are no humans left?
3
SPEAKER_01
Well, it depends on how you do the math, but I would say by any reasonable economic standard, if you look at the future income of humans and there aren t any, you know, that s not a good investment.
3
SPEAKER_01
Moreover, why can t we have a little bit of pride in our species?
3
SPEAKER_01
Why should we just build another species that gets rid of us?
3
SPEAKER_01
If we were Neanderthals...
3
SPEAKER_01
would we really consider it a smart move if we had really advanced biotech to build Homo sapiens?
3
SPEAKER_01
You might say, hey Max, yeah, let s build these Homo sapiens.
3
SPEAKER_01
They re gonna be smarter than us.
3
SPEAKER_01
Maybe they can help us defend us better against predators and help fix up our caves, make them nicer.
3
SPEAKER_01
We ll control them undoubtedly.
3
SPEAKER_01
So then they build a couple, a little baby girl, a little baby boy.
3
SPEAKER_01
And then you have some wise old Neanderthal elder who s like, hmm, I m scared that we re opening a Pandora s box here and that we re gonna get outsmarted by these super Neanderthal intelligences and there won t be any Neanderthals left.
3
SPEAKER_01
But then you have a bunch of others in the cave, right?
3
SPEAKER_01
Are you such a Luddite scaremonger?
3
SPEAKER_01
Of course, they re going to want to keep us around because we are their creators.
3
SPEAKER_01
And, you know, I think the smarter they get, the nicer they re going to get.
3
SPEAKER_01
They re going to want us around and it s going to be fine.
3
SPEAKER_01
And besides, look at these babies.
3
SPEAKER_01
They re so cute.
3
SPEAKER_01
Clearly, they re totally harmless.
3
SPEAKER_01
Those babies are exactly GPT-4.
3
SPEAKER_01
I want to be clear.
3
SPEAKER_01
It s not GPT-4.
3
SPEAKER_01
that s terrifying, is that GPT-4 is a baby technology.
3
SPEAKER_01
You know, and Microsoft even had a paper recently out titled something like Sparkles of AGI, where they were basically saying this is baby AI, like these little Neanderthal babies.
3
SPEAKER_01
And it s going to grow up.
3
SPEAKER_01
There s going to be other systems from the same company, from other companies, they ll be way more powerful, but they re going to take all the things
3
SPEAKER_01
ideas from these babies.
3
SPEAKER_01
Before we know it, we re gonna be like those last Neanderthals who were pretty disappointed when they realized that they were getting replaced.
3
SPEAKER_00
Well, this interesting point you make, which is the programming, it s entirely possible that GPT-4 is already
3
SPEAKER_00
the kind of system that can change everything by writing programs.
3
SPEAKER_01
Yeah, because it s Life 2.0, the systems I m afraid of are gonna look nothing like a large language model and they re not gonna...
3
SPEAKER_01
But once it or other people figure out a way of using this tech to make much better tech, it s just constantly replacing its software.
3
SPEAKER_01
And from everything we ve seen about how these work under the hood, they re like the minimum viable intelligence.
3
SPEAKER_01
They do everything in the dumbest way that still works, sort of.
4
SPEAKER_01
And so they are like 3.0, except when they replace their software, it s a lot faster than when you decide to learn Swedish.
4
SPEAKER_01
And moreover, they think a lot faster than us too.
4
SPEAKER_01
So when, you know, we don t think, have one logical step every nanosecond or a few or so the way they do.
4
SPEAKER_01
And we can t also just suddenly scale up our hardware massively in the cloud.
4
SPEAKER_01
We re so limited, right?
4
SPEAKER_01
So they are, they are also life,
4
SPEAKER_01
become a little bit more like Life 3.0 in that if they need more hardware, hey, just rent it in the cloud.
4
SPEAKER_01
How do you pay for it?
4
SPEAKER_01
Well, with all the services you provide.
4
SPEAKER_00
And what we haven t seen yet,
4
SPEAKER_00
which could change a lot, is entire software systems.
4
SPEAKER_00
So right now, programming is done sort of in bits and pieces as an assistant tool to humans, but I do a lot of programming.
4
SPEAKER_00
with the kind of stuff that GPT-4 is able to do.
4
SPEAKER_00
I mean, it s replacing a lot what I m able to do.
4
SPEAKER_00
But you still need a human in the loop to kind of manage the design of things, manage like what are the prompts that generate the kind of stuff, to do some basic adjustment of the code, to do some debugging.
4
SPEAKER_00
But if it s possible to add on top of GPT-4 kind of,
4
SPEAKER_00
a feedback loop of self-debugging, improving the code, and then you launch that system out into the wild on the internet, because everything is connected, and have it do things, have it interact with humans, and then get that feedback.
4
SPEAKER_00
Now you have this giant ecosystem of humans.
4
SPEAKER_00
It s one of the things that Elon Musk recently sort of tweeted as a case why everyone needs to pay $7 or whatever for Twitter.
4
SPEAKER_01
To make sure they re real.
4
SPEAKER_00
make sure they re real, we re now going to be living in a world where the bots are getting smarter and smarter and smarter to a degree where you can t tell the difference between a human and a bot.
4
SPEAKER_00
And now you can have bots outnumber humans by one million to one, which is why he s making the case why you have to pay to prove you re human, which is one of the only mechanisms to prove, which is depressing.
4
SPEAKER_01
As individuals, we should, from time to time, ask ourselves, why are we doing what we re doing?
4
SPEAKER_01
And as a species, we need to do that too.
4
SPEAKER_01
So if we re building, as you say, machines that are outnumbering us and more and more outsmarting us and replacing us on the job market, not just for the dangerous and boring tasks, but also for writing poems and doing art and things that a lot of people find really meaningful, gotta ask ourselves, why?
4
SPEAKER_01
Why are we doing this?
4
SPEAKER_01
We are, the answer is Moloch is tricking us into doing it.
4
SPEAKER_01
And it s such a clever trick that even though we see the trick, we still have no choice but to fall for it, right?
4
SPEAKER_01
Also, the thing you said about you using co-pilot AI tools to program faster, what factor faster would you say you code now?
4
SPEAKER_01
Does it go twice as fast?
4
SPEAKER_00
I don t really, because it s such a new tool.
4
SPEAKER_00
I don t know if speed is significantly improved.
4
SPEAKER_00
but it feels like I m a year away from being five to 10 times faster.
4
SPEAKER_01
So if that s typical for programmers, then you re already seeing another kind of self-recursive self-improvement, right?
4
SPEAKER_01
Because previously, a major generation of improvement of the code would happen at the human R&D time scale.
4
SPEAKER_01
And now, if that s five times shorter,
4
SPEAKER_01
then it s gonna take five times less time than it otherwise would to develop the next level of these tools, and so on.
4
SPEAKER_01
So this is exactly the sort of beginning of an intelligence explosion.
4
SPEAKER_01
There can be humans in the loop a lot in the early stages, and then eventually humans are needed less and less, and the machines can more kind of go along.
4
SPEAKER_01
But what you said there is just an exact example of these sort of things.
4
SPEAKER_01
Another thing which,
4
SPEAKER_01
I was kind of lying on my psychiatrist, imagining I m on a psychiatrist s couch here saying, well, what are my fears that people would do with AI systems?
4
SPEAKER_01
So I mentioned three that I had fears about many years ago that they would do, namely teach it to code, connect it to the internet, then teach it to manipulate humans.
4
SPEAKER_01
A fourth one is,
4
SPEAKER_01
building an API where code can control this super powerful thing, right?
4
SPEAKER_01
That s very unfortunate because one thing that systems like GPT-4 have going for them is that they are an oracle in the sense that they just answer questions.
4
SPEAKER_01
There is no robot connected to GPT-4.
4
SPEAKER_01
GPT-4 can t go and do stock trading based on its thinking.
4
SPEAKER_01
It s not an agent.
4
SPEAKER_01
An intelligent agent is something that takes in information from the world, processes it to figure out what action to take based on its goals that it has, and then does something back on the world.
4
SPEAKER_01
But once you have an API, for example, GPT-4, nothing stops Joe Schmo and a lot of other people from building real agents, which just keep making calls somewhere in some inner loop somewhere to these powerful Oracle systems.
4
SPEAKER_01
which makes them themselves much more powerful.
4
SPEAKER_01
That s another kind of unfortunate development, which I think we would have been better off delaying.
4
SPEAKER_01
I don t want to pick on any particular companies.
4
SPEAKER_01
I think they re all under a lot of pressure to make money.
4
SPEAKER_01
And again, the reason we re calling for this pause is to give them all cover to do what they know is the right thing, slow down a little bit at this point.
4
SPEAKER_01
everything we ve talked about, I hope we ll make it clear to people watching this, you know, why these sort of human level tools can cause a gradual acceleration.
4
SPEAKER_01
You keep using yesterday s technology to build tomorrow s technology.
4
SPEAKER_01
And when you do
4
SPEAKER_01
that over and over again you naturally get an explosion.
4
SPEAKER_01
That s the definition of an explosion in science.
4
SPEAKER_01
If you have two people and they fall in love, now you have four people and then they can make more babies and now you have eight people and then you have 16, 32, 64, et cetera.
4
SPEAKER_01
We call that a population explosion where it s just that each, if it s instead free neutrons
4
SPEAKER_01
in a nuclear reaction, that if each one can make more than one, then you get an exponential growth in that, we call it a nuclear explosion.
4
SPEAKER_01
All explosions are like that, and an intelligence explosion, it s just exactly the same principle, that some amount of intelligence can make more intelligence than that, and then repeat.
4
SPEAKER_01
You always get exponentials.
4
SPEAKER_00
What s your intuition, why does, you mentioned there s some technical reasons why it doesn t stop at a certain point.
4
SPEAKER_00
What s your intuition, and do you have any intuition why it might stop?
4
SPEAKER_01
It s obviously gonna stop when it bumps up against the laws of physics.
4
SPEAKER_01
There are some things you just can t do no matter how smart you are, right?
4
SPEAKER_00
Allegedly.
4
SPEAKER_00
Because we don t know the full laws of physics yet, right?
4
SPEAKER_01
Seth Lloyd wrote a really cool paper on the physical limits on computation, for example, if you make it.
4
SPEAKER_01
put too much energy into it and the finite space will turn into a black hole.
4
SPEAKER_01
You can t move information around faster than the speed of light, stuff like that.
4
SPEAKER_01
But it s hard to store way more than a modest number of bits per atom, et cetera.
4
SPEAKER_01
But those limits are just astronomically above, like 30 orders of magnitude above where we are now.
4
SPEAKER_01
bigger jump in intelligence than if you go from an ant to a human.
4
SPEAKER_01
I think, of course, what we want to do is have a controlled thing.
4
SPEAKER_01
A nuclear reactor, you put moderators in to make sure exactly it doesn t blow up out of control, right?
4
SPEAKER_01
When we do
4
SPEAKER_01
experiments with biology and cells and so on.
4
SPEAKER_01
We also try to make sure it doesn t get out of control.
4
SPEAKER_01
We can do this with AI, too.
4
SPEAKER_01
The thing is, we haven t succeeded yet.
4
SPEAKER_01
And Moloch is exactly doing the opposite, just fueling, just egging everybody on, faster, faster, faster, or the other company is going to catch up with you, or the other country is going to catch up with you.
4
SPEAKER_01
We have to want this stuff.
4
SPEAKER_01
And I don t believe in this, just asking people to look into their hearts and do the right thing.
4
SPEAKER_01
It s easier for others to say that.
4
SPEAKER_01
But if you re in a situation where your company is going to get screwed, by other companies that are not stopping, you re putting people in a very hard situation.
4
SPEAKER_01
The right thing to do is change the whole incentive structure instead.
4
SPEAKER_01
And this is not an old, maybe I should say one more thing about this, because Moloch has been around as humanity s number one or number two enemy since the beginning of civilization.
4
SPEAKER_01
And we came up with some really cool countermeasures.
4
SPEAKER_01
First of all, already over 100,000 years ago, evolution realized that it was very unhelpful that people kept killing each other all the time.
4
SPEAKER_01
So it genetically gave us compassion.
4
SPEAKER_01
and made it so that if you get two drunk dudes getting into a pointless bar fight, they might give each other black eyes, but they have a lot of inhibition towards just killing each other.
4
SPEAKER_01
And similarly, if you find a baby lying on the street when you go out for your morning jog tomorrow, you re gonna stop and pick it up, right?
4
SPEAKER_01
Even though it may make you late for your next podcast.
4
SPEAKER_01
So evolution gave us these genes that make our own egoistic incentives more aligned with what s good for the greater group we re part of, right?
4
SPEAKER_01
And then as we got a bit more sophisticated and developed language, we invented gossip, which is also a fantastic anti-moloch, right?
4
SPEAKER_01
Because now it really discourages
4
SPEAKER_01
liars, moochers, cheaters, because their own incentive now is not to do this, because word quickly gets around, and then suddenly people aren t gonna invite them to their dinners anymore or trust them.
4
SPEAKER_01
And then when we got still more sophisticated and bigger societies, we invented the legal system, where even strangers who couldn t rely on gossip and things like this would treat each other, would have an incentive.
4
SPEAKER_01
Now, those guys in the bar fight, even if someone is so drunk that he,
4
SPEAKER_01
actually wants to kill the other guy.
4
SPEAKER_01
He also has a little thought in the back of his head that, you know, do I really want to spend the next 10 years eating like really crappy food in a small room?
4
SPEAKER_01
I m just going to chill out, you know.
4
SPEAKER_01
And we similarly have tried to give these incentives to our corporations by having regulation and all sorts of oversight so that their incentives are aligned with the greater good.
4
SPEAKER_01
We tried really hard.
4
SPEAKER_01
And the big problem that we re failing now is not that we haven t tried before, but it s just that the tech is growing much, is developing much faster than the regulators have been able to keep up, right?
4
SPEAKER_01
So regulators, it s kind of comical, the European Union right now is doing this AI act, right?
4
SPEAKER_01
And in the beginning, they had a little opt-out exception that GPT-4 would be completely excluded from regulation.
4
SPEAKER_01
Brilliant idea.
4
SPEAKER_01
What s the logic behind that?
4
SPEAKER_01
Some lobbyists pushed successfully for this.
4
SPEAKER_01
So we were actually quite involved with the Future of Life Institute, Mark Brackel, Mr. Ouk, Anthony Aguirre and others.
4
SPEAKER_01
We re quite involved with educating various people involved in this process about
4
SPEAKER_01
these general purpose AI models coming and pointing out that they would become the laughing stock if they didn t put it in.
4
SPEAKER_01
So the French started pushing for it, it got put in to the draft and it looked like all was good.
4
SPEAKER_01
And then there was a huge counter push from lobbyists.
4
SPEAKER_01
There were more lobbyists in Brussels from tech companies than from oil companies, for example.
4
SPEAKER_01
And it looked like it might maybe get taken out again.
4
SPEAKER_01
And now GPT-4 happened.
4
SPEAKER_01
And I think it s going to stay in.
4
SPEAKER_01
But this just shows, you know, Moloch can be defeated.
4
SPEAKER_01
But the challenge we re facing is that the tech is generally much faster than what the policy makers are.
4
SPEAKER_01
And a lot of the policy makers also don t have a tech background.
4
SPEAKER_01
So we really need to work hard to educate them on what s taking place here.
4
SPEAKER_01
So we re getting this situation where the first kind of non, so I define artificial intelligence just as non-biological intelligence.
4
SPEAKER_01
And by that definition, a company, a corporation is also an artificial intelligence because the corporation isn t, it s humans, it s a system.
4
SPEAKER_01
If its CEO decides, the CEO of a tobacco company decides one morning that she or he doesn t want to sell cigarettes anymore, they ll just put another CEO in there.
4
SPEAKER_01
It s not enough to align the incentives of individual people or align individual computers  incentives
4
SPEAKER_01
to their owners, which is what technically AI safety research is about.
4
SPEAKER_01
You also have to align the incentives of corporations with a greater good.
4
SPEAKER_01
And some corporations have gotten so big and so powerful very quickly that in many cases, their lobbyists instead align the regulators to what they want rather than the other way around.
4
SPEAKER_01
It s a classic regulatory capture.
4
SPEAKER_00
All right, is the thing that the slowdown hopes to achieve is give enough time to the regulators to catch up or enough time to the companies themselves to breathe and understand how to do AI safety correctly?
4
SPEAKER_01
I think both, but I think that the vision, the path to success I see is first you give a breather actually to the people in these companies, their leadership who wants to do the right thing and they all have safety teams and so on on their companies.
4
SPEAKER_01
Give them a chance to
4
SPEAKER_01
get together with the other companies, and the outside pressure can also help catalyze that, right?
4
SPEAKER_01
And work out what is it that s, what are the reasonable safety requirements one should put on future systems before they get rolled out?
4
SPEAKER_01
There are a lot of people also in academia and elsewhere outside of these companies who can be brought into this and have a lot of very good ideas.
4
SPEAKER_01
And then I think it s very realistic that within six months, you can get these people coming up, so here s a white paper, here s what we all think is reasonable.
4
SPEAKER_01
You know, you didn t, just because cars killed a lot of people, you didn t ban cars, but they got together a bunch of people and decided, you know, in order to be allowed to sell a car, it has to have a seatbelt in it.
4
SPEAKER_01
They re the analogous things that you can start requiring of future AI systems so that they are safe.
4
SPEAKER_01
And once this heavy lifting, this intellectual work has been done by experts in the field, which can be done quickly, I think it s going to be quite easy to get policymakers to
4
SPEAKER_01
to see, yeah, this is a good idea.
4
SPEAKER_01
And it s, you know, for the companies to fight Moloch, they want, and I believe Sam Altman has explicitly called for this, they want the regulators to actually adopt it so that their competition is gonna abide by it too, right?
4
SPEAKER_01
You don t wanna be enacting all these principles and you abide by them, and then there s this one little company that,
4
SPEAKER_01
doesn t sign on to it, and then now they can gradually overtake you, then the companies will get, be able to sleep secure knowing that everybody s playing by the same rules.
4
SPEAKER_00
So do you think it s possible to develop guardrails that keep the systems from basically damaging irreparably humanity, while still enabling sort of the capitalist
5
SPEAKER_00
competition between companies as they develop how to best make money with this AI.
5
SPEAKER_00
You think there s a balancing?
5
SPEAKER_00
That s possible.
5
SPEAKER_01
Absolutely.
5
SPEAKER_01
I mean, we ve seen that in many other sectors where you ve had the free market produce quite good things without causing particular harm.
5
SPEAKER_01
When the guardrails are there and they work, you know,
5
SPEAKER_01
Capitalism is a very good way of optimizing for just getting the same things done more efficiently.
5
SPEAKER_01
But it was good.
5
SPEAKER_01
In hindsight, I ve never met anyone, even on parties way over on the right, in any country, who thinks it was a terrible idea to ban child labor, for example.
5
SPEAKER_00
Yeah, but it seems like this particular technology has gotten so good so fast, become powerful to a degree where you could see in the near term the ability to make a lot of money.
5
SPEAKER_00
And to put guardrails, to develop guardrails quickly in that kind of context seems to be tricky.
5
SPEAKER_00
It s not similar to cars or child labor.
5
SPEAKER_00
It seems like the opportunity to make a lot of money here very quickly is right here before us.
5
SPEAKER_01
Yeah, so again, there s this cliff
5
SPEAKER_01
Yeah, it gets quite scenic.
5
SPEAKER_01
The closer to the cliff you go, the more money there is, the more gold ingots there are on the ground you can pick up or whatever if you wanna drive there very fast.
5
SPEAKER_01
But it s not in anyone s incentive that we go over the cliff.
5
SPEAKER_01
And it s not like everybody s in their own car.
5
SPEAKER_01
All the cars are connected together with a chain.
5
SPEAKER_01
So if anyone goes over, they ll start dragging the others down too.
5
SPEAKER_01
And so ultimately it s in the selfish,
5
SPEAKER_01
Interests also the people in the company is to slow down when it s just not seeing the contours of the cliff there in front of you.
5
SPEAKER_01
The problem is that even though the people who are building the technology,
5
SPEAKER_01
And the CEOs, they really get it.
5
SPEAKER_01
The shareholders and these other market forces, they are people who don t honestly understand that the cliff is there.
5
SPEAKER_01
They usually don t.
5
SPEAKER_01
You have to get quite into the weeds to really appreciate how powerful this is and how fast.
5
SPEAKER_01
And a lot of people are even still stuck again in this idea that
5
SPEAKER_01
In this carbon chauvinism, as I like to call it, that you can only have our level of intelligence in humans.
5
SPEAKER_01
There s something magical about it.
5
SPEAKER_01
Whereas the people in the tech companies.
5
SPEAKER_01
who build this stuff, they all realize that intelligence is information processing of a certain kind, and it really doesn t matter at all whether the information is processed by carbon atoms in neurons in brains or by silicon atoms in some technology we build.
5
SPEAKER_01
So you brought up capitalism earlier, and there are a lot of people who love capitalism and a lot of people who really, really don t.
5
SPEAKER_01
It struck me recently that what s happening with capitalism here is exactly analogous to the way in which superintelligence might wipe us out.
5
SPEAKER_01
You know, I studied economics for my undergrad, Stockholm School of Economics, yay.
5
SPEAKER_01
So I was very interested in how you could use market forces to just get stuff done more efficiently, but give the right incentives to the market so that it wouldn t do really bad things.
5
SPEAKER_01
So Dylan Hadfield-Manel, who s a professor and colleague of mine at MIT, wrote this really interesting paper with some collaborators recently.
5
SPEAKER_01
where they prove mathematically that if you just take one goal that you just optimize for, on and on and on indefinitely, that you think is gonna bring you in the right direction.
5
SPEAKER_01
What basically always happens is, in the beginning, it will make things better for you.
5
SPEAKER_01
But if you keep going, at some point it s gonna start making things worse for you again, and then gradually it s gonna make it really, really terrible.
5
SPEAKER_01
So just as a simple, the way I think of the proof is,
5
SPEAKER_01
Suppose you want to go from here back to Austin, for example, and you re like, okay, yeah, let s just, let s go south, but you put in exactly the right, sort of the right direction.
5
SPEAKER_01
Just optimize that south as possible.
5
SPEAKER_01
You get closer and closer to Austin.
5
SPEAKER_01
But there s always some little error, so you re not going exactly towards Austin, but you get pretty close.
5
SPEAKER_01
But eventually you start going away again, and eventually you re gonna be leaving the solar system.
5
SPEAKER_01
And they proved, it s a beautiful mathematical proof, this happens generally, and this is very important for AI, because even though Stuart Russell has,
5
SPEAKER_01
written a book and given a lot of talks on why it s a bad idea to have AI just blindly optimize something.
5
SPEAKER_01
That s what pretty much all our systems do.
5
SPEAKER_01
We have something called the loss function that we re just minimizing, or reward function, we re just maximizing.
5
SPEAKER_01
Capitalism is exactly like that too.
5
SPEAKER_01
We wanted to get stuff done more efficiently than people wanted.
5
SPEAKER_01
So we introduced the free market.
5
SPEAKER_01
things got done much more efficiently than they did in, say, communism, right?
5
SPEAKER_01
And it got better.
5
SPEAKER_01
But then it just kept optimizing and kept optimizing, and you got ever bigger companies and ever more efficient information processing, and now also very much powered by IT.
5
SPEAKER_01
Eventually, a lot of people are beginning to feel, wait, we re kind of optimizing a bit too much.
5
SPEAKER_01
Why did we just chop down half the rainforest?
5
SPEAKER_01
And why did suddenly these regulators get captured by lobbyists and so on?
5
SPEAKER_01
It s just the same optimization that s been running for too long.
5
SPEAKER_01
If you have an AI that actually has power over the world and you just give it one goal and just keep optimizing that, most likely everybody s going to be like, yay, this is great in the beginning.
5
SPEAKER_01
Things are getting better.
5
SPEAKER_01
But it s almost impossible to give it exactly the right direction to optimize in.
5
SPEAKER_01
And then eventually, all hay breaks loose, right?
5
SPEAKER_01
Nick Bostrom and others have given examples that sound quite silly.
5
SPEAKER_01
Like, what if you just want to tell it to cure cancer or something, and that s all you tell it?
5
SPEAKER_01
Maybe it s going to decide to
5
SPEAKER_01
take over entire continents just so we can get more supercomputer facilities in there and figure out how to cure cancer backwards.
5
SPEAKER_01
And then you re like, wait, that s not what I wanted, right?
5
SPEAKER_01
And the issue with capitalism and the issue with running away, I have kind of merged now because the Moloch I talked about is exactly the capitalist Moloch that we have built an economy that is optimized for only one thing, profit.
5
SPEAKER_01
And that worked great back when things were very inefficient, and then now it s getting done better.
5
SPEAKER_01
And it worked great as long as the companies were small enough that they couldn t capture the regulators.
5
SPEAKER_01
But that s not true anymore, but they keep optimizing.
5
SPEAKER_01
And now they realize that these companies can make even more profit by building ever more powerful AI, even if it s reckless.
5
SPEAKER_01
But optimize more and more and more and more and more.
5
SPEAKER_01
So this is Moloch again showing up, and I just wanna, anyone here who has any concerns about late-stage capitalism having gone a little too far, you should worry about superintelligence, because it s the same villain in both cases.
5
SPEAKER_01
It s Moloch.
5
SPEAKER_00
And optimizing one objective function aggressively, blindly is going to take us there.
5
SPEAKER_01
Yeah, we have to pause from time to time and look into our hearts and ask, why are we doing this?
5
SPEAKER_01
Am I still going towards Austin or have I gone too far?
5
SPEAKER_01
Maybe we should change direction.
5
SPEAKER_00
And that is the idea behind a halt for six months.
5
SPEAKER_00
Why six months?
5
SPEAKER_00
It seems like a very short period.
5
SPEAKER_00
Can we just linger and explore different ideas here?
5
SPEAKER_00
Because this feels like a really important moment in human history where pausing would actually have a significant positive effect.
5
SPEAKER_01
We said six months because we figured the number one pushback we were gonna get in the West was like, but China.
5
SPEAKER_01
And everybody knows there s no way that China is going to catch up with the West on this in six months.
5
SPEAKER_01
So that argument goes off the table and you can forget about geopolitical competition and just focus on.
5
SPEAKER_01
The real issue, that s why we put this.
5
SPEAKER_00
That s really interesting.
5
SPEAKER_00
But you ve already made the case that even for China, if you actually wanna take on that argument, China too would not be bothered by a longer halt because they don t wanna lose control, even more than the West doesn t.
5
SPEAKER_01
That s what I think, yeah.
5
SPEAKER_00
That s a really interesting argument.
5
SPEAKER_00
I have to actually really think about that, which the kind of thing people assume is if you develop an AGI, that open AI, if they re the ones that do it, for example, they re going to win.
5
SPEAKER_00
But you re saying, no, everybody loses.
5
SPEAKER_01
Yeah, it s gonna get better and better and better, and then kaboom, we all lose.
5
SPEAKER_01
That s what s gonna happen.
5
SPEAKER_00
When lose and win are defined in a metric of basically quality of life,
5
SPEAKER_00
for human civilization and for Sam Altman.
5
SPEAKER_01
Both.
5
SPEAKER_01
To be blunt, my personal guess, and people can quibble with this, is that we re just gonna, there won t be any humans.
5
SPEAKER_01
That s it, that s what I mean by lose.
5
SPEAKER_01
We can see in history, once you have some species or some group of people who aren t needed anymore, it doesn t usually work out so well for them, right?
5
SPEAKER_01
There were a lot of horses that were used for traffic in Boston, and then the car got invented, and most of them got, you know, we don t need to go there.
5
SPEAKER_01
And if you look at humans, you know, right now, why did the labor movement succeed after the Industrial Revolution?
5
SPEAKER_01
Because it was needed.
5
SPEAKER_01
Even though we had a lot of molochs and there was child labor and so on, you know, the company still needed to have workers.
5
SPEAKER_01
And that s why strikes had power and so on.
5
SPEAKER_01
If we get to the point where most humans aren t needed anymore, I think it s quite naive to think that they re going to still be treated well.
5
SPEAKER_01
You know, we say that, yeah, yeah, everybody s equal and the government will always, we ll always protect them.
5
SPEAKER_01
But if you look in practice, groups that are very disenfranchised and don t have any actual power usually get screwed.
5
SPEAKER_01
And now in the beginning, so Industrial Revolution, we automated away muscle work.
5
SPEAKER_01
But that worked out pretty well eventually because we educated ourselves and started working with our brains instead and got usually more interesting, better paid jobs.
5
SPEAKER_01
But now we re beginning to replace brain work.
5
SPEAKER_01
So we replaced a lot of boring stuff, like we got the pocket calculator so you don t have people multiplying numbers anymore at work.
5
SPEAKER_01
Fine, there were better jobs they could get.
5
SPEAKER_01
But now, GPT-4, you know,
5
SPEAKER_01
and the stable diffusion and techniques like this, they re really beginning to blow away some jobs that people really love having.
5
SPEAKER_01
There was a heartbreaking article just posted yesterday on social media I saw about this guy who was doing 3D modeling for gaming.
5
SPEAKER_01
And all of a sudden, now that he got this new software, he just says prompts, and he feels his whole job that he loved just lost its meaning, you know?
5
SPEAKER_01
And I asked GPT-4 to rewrite Twinkle Twinkle Little Star in the style of Shakespeare.
5
SPEAKER_01
I couldn t have done such a good job.
5
SPEAKER_01
It was really impressive.
5
SPEAKER_01
You ve seen a lot of the art coming out here, right?
5
SPEAKER_01
So I m all for,
5
SPEAKER_01
automating away the dangerous jobs and the boring jobs.
5
SPEAKER_01
But I think you hear some arguments which are too glib.
5
SPEAKER_01
Sometimes people say, well, that s all that s going to happen.
5
SPEAKER_01
We re getting rid of the boring, tedious, dangerous jobs.
5
SPEAKER_01
It s just not true.
5
SPEAKER_01
There are a lot of really interesting jobs that are being taken away now.
5
SPEAKER_01
Journalism is going to get crushed.
5
SPEAKER_01
Coding is going to get crushed.
5
SPEAKER_01
I predict the job market for programmers, salaries are going to start dropping.
5
SPEAKER_01
you know, if you said you can code five times faster, you know, then you need five times fewer programmers.
5
SPEAKER_01
Maybe there ll be more output also, but you ll still end up needing fewer programmers than today.
5
SPEAKER_01
And I love coding, you know, I think it s super cool.
5
SPEAKER_01
So we need to stop and ask ourselves why, again, are we doing this as humans, right?
5
SPEAKER_01
I feel that AI should be built by humanity for humanity.
5
SPEAKER_01
And let s not forget that.
5
SPEAKER_01
It shouldn t be by Moloch for Moloch, or what it really is now is kind of by humanity for Moloch, which doesn t make any sense.
5
SPEAKER_01
It s for us that we re doing it.
5
SPEAKER_01
And it would make a lot more sense if we
5
SPEAKER_01
Build it, develop, figure out gradually and safely how to make all this tech.
5
SPEAKER_01
And then we think about what are the kinds of jobs that people really don t want to have, you know, automate them all away.
5
SPEAKER_01
And then we ask, what are the jobs that people really find meaning in, like maybe.
5
SPEAKER_01
taking care of children in the daycare center, maybe doing art, et cetera, et cetera.
5
SPEAKER_01
And even if it were possible to automate that way, we don t need to do that, right?
5
SPEAKER_00
We built these machines.
5
SPEAKER_00
Well, it s possible that we redefine or rediscover what are the jobs that give us meaning.
5
SPEAKER_00
So for me, the thing, it is really sad, okay?
5
SPEAKER_00
Half the time I m excited, half the time I m crying as I m generating code because I kind of love programming.
5
SPEAKER_00
It s an act of creation.
5
SPEAKER_00
You have an idea, you design it, and then you bring it to life and it does something, especially if there s some intelligence that it does something.
5
SPEAKER_00
It doesn t even have to have intelligence.
5
SPEAKER_00
Printing Hello World on screen, you made a little machine and it comes to life.
5
SPEAKER_00
And there s a bunch of tricks you learn along the way, because you ve been doing it for many, many years.
5
SPEAKER_00
And then to see AI be able to generate all the tricks you thought were special.
5
SPEAKER_00
I don t know, it s very...
5
SPEAKER_00
it s scary, it s almost painful, like a loss of innocence maybe, like maybe when I was younger, I remember before I learned that sugar s bad for you, you should be on a diet, I remember I enjoyed candy deeply, in a way I just can t anymore, that I know is bad for me.
5
SPEAKER_00
I enjoyed it unapologetically, fully, just intensely.
5
SPEAKER_00
And I just, I lost that.
5
SPEAKER_00
Now I feel like a little bit of that is lost for me with program, or being lost with programming.
5
SPEAKER_00
Similar as it is for the 3D modeler, no longer being able to really enjoy the art of modeling 3D things for gaming.
5
SPEAKER_00
I don t know what to make sense of that.
5
SPEAKER_00
Maybe I would rediscover that the true magic of what it means to be human is connecting with other humans, to have conversations like this.
5
SPEAKER_00
I don t know, to have sex, to eat food, to really intensify the value from conscious experiences versus creating other stuff.
5
SPEAKER_01
You re pitching the rebranding again from Homo sapiens to Homo sentiens, the meaningful experiences.
5
SPEAKER_01
And just to inject some optimism in this here so we don t sound like a bunch of gloomers, you know, we can totally have our cake and eat it.
5
SPEAKER_01
You hear a lot of totally bullshit claims that we can t afford having more teachers, have to cut the number of nurses.
5
SPEAKER_01
You know, that s just nonsense, obviously.
5
SPEAKER_01
With anything, even quite far short of AGI, we can dramatically improve
5
SPEAKER_01
grow the GDP and produce this wealth of goods and services, it s very easy to create a world where everybody is better off than today, including the richest people can be better off as well, right?
5
SPEAKER_01
It s not a zero-sum game, you know, technology.
5
SPEAKER_01
Again, you can have two countries like Sweden and Denmark have all these ridiculous wars century after century.
5
SPEAKER_01
And, uh, sometimes that Sweden got a little better off cause it got a little bit bigger and then Denmark got a little bit better off cause Sweden got a little bit smaller.
5
SPEAKER_01
And, and, but then.
6
SPEAKER_00
Then technology came along and we both got just dramatically wealthier without taking away from anyone else.
6
SPEAKER_00
It was just a total win for everyone.
6
SPEAKER_00
And AI can do that on steroids.
6
SPEAKER_00
If you can build safe AGI, if you can build super intelligence.
6
SPEAKER_00
basically all the limitations that cause harm today can be completely eliminated.
6
SPEAKER_00
It s a wonderful possibility.
6
SPEAKER_00
And this is not sci-fi.
6
SPEAKER_00
This is something which is clearly possible according to the laws of physics.
6
SPEAKER_00
And we can talk about ways of making it safe also.
6
SPEAKER_00
But unfortunately,
6
SPEAKER_00
That ll only happen if we steer in that direction.
6
SPEAKER_00
That s absolutely not the default outcome.
6
SPEAKER_00
That s why income inequality keeps going up.
6
SPEAKER_00
That s why the life expectancy in the U.S.
6
SPEAKER_00
has been going down.
6
SPEAKER_00
I think it s four years in a row.
6
SPEAKER_00
I just read a heartbreaking study from the CDC about how something like one third of all teenage girls in the U.S.
6
SPEAKER_00
have been thinking about suicide.
6
SPEAKER_00
Those are steps in totally the wrong direction, and it s important to keep our eyes on the prize here.
6
SPEAKER_00
We have the power now for the first time in the history of our species to harness artificial intelligence to help us really flourish and help bring out the best in our humanity rather than the worst of it.
6
SPEAKER_00
to help us have really fulfilling experiences that feel truly meaningful.
6
SPEAKER_00
And you and I shouldn t sit here and dictate to future generations what they will be.
6
SPEAKER_00
Let them figure it out, but let s give them a chance to live and not foreclose all these possibilities for them by just messing things up, right?
6
SPEAKER_01
And for that, we ll have to solve the AI safety problem.
6
SPEAKER_01
It would be nice if we can linger on exploring that a little bit.
6
SPEAKER_01
So one interesting way to enter that discussion is you tweeted, and Elon replied, you tweeted, let s not just focus on whether GPT-4 will do more harm or good on the job market, but also whether its coding skills will hasten the arrival of superintelligence.
6
SPEAKER_01
That s something we ve been talking about, right?
6
SPEAKER_01
So Elon proposed one thing in the reply saying maximum truth seeking is my best guess for AI safety.
6
SPEAKER_01
Can you maybe steel man the case for this objective function of truth and maybe make an argument against it in general?
6
SPEAKER_01
What are your different ideas to start approaching the solution to AI safety?
6
SPEAKER_00
I didn t see that reply actually.
6
SPEAKER_00
Oh, interesting.
6
SPEAKER_00
But I really resonate with it because
6
SPEAKER_00
AI is not evil.
6
SPEAKER_00
It caused people around the world to hate each other much more, but that s because we made it in a certain way.
6
SPEAKER_00
It s a tool.
6
SPEAKER_00
We can use it for great things and bad things, and we could just as well have AI systems, and this is part of my vision for success here, truth-seeking AI that really brings us together again.
6
SPEAKER_00
Why do people hate each other so much between countries and within countries?
6
SPEAKER_00
It s because
6
SPEAKER_00
they each have totally different versions of the truth, right?
6
SPEAKER_00
If they all had the same truth that they trusted for good reason, because they could check it and verify it and not have to believe in some self-proclaimed authority, right?
6
SPEAKER_00
There wouldn t be nearly as much hate.
6
SPEAKER_00
There d be a lot more understanding instead.
6
SPEAKER_00
And this is, I think, something AI can help enormously with.
6
SPEAKER_00
For example,
6
SPEAKER_00
A little baby step in this direction is this website called Metaculous, where people bet and make predictions, not for money, but just for their own reputation.
6
SPEAKER_00
And it s kind of funny, actually.
6
SPEAKER_00
You treat the humans like you treat AIs.
6
SPEAKER_00
You have a loss function where they get penalized if they re super confident on something, and then the opposite happens.
6
SPEAKER_00
Whereas if you re kind of humble, and then you re like, I think it s 51% chance this is going to happen, and then the other happens, you don t get penalized much.
6
SPEAKER_00
And what you can see is that some people are much better at predicting than others.
6
SPEAKER_00
They ve earned your trust, right?
6
SPEAKER_00
One project that I m working on right now is an outgrowth to improve the News Foundation together with the Metaculous folks is seeing if we can really scale this up a lot with more powerful AI, because I would love it
6
SPEAKER_00
I would love for there to be a really powerful truth-seeking system that is trustworthy because it keeps being right about stuff.
6
SPEAKER_00
And people who come to it and maybe look at its latest trust ranking of different pundits and newspapers, et cetera, if they want to know why someone got a low score, they can click on it and see all the predictions that they actually made and how they turned out.
6
SPEAKER_00
This is how we do it in science.
6
SPEAKER_00
You trust scientists like Einstein who said something everybody thought was bullshit and turned out to be right.
6
SPEAKER_00
You get a lot of trust points, and he did it multiple times even.
6
SPEAKER_00
I think AI has the power to really heal a lot of the rifts we re seeing by creating trust systems.
6
SPEAKER_00
get away from this idea today with some fact-checking sites which might themselves have an agenda and you just trust it because of its reputation.
6
SPEAKER_00
You want to have these sort of systems, they earn their trust and they re completely transparent.
6
SPEAKER_00
This, I think, would actually help a lot.
6
SPEAKER_00
That can, I think, help heal the very dysfunctional conversation that humanity has about how it s going to deal with all its biggest challenges in the world today.
6
SPEAKER_00
And then enter on the technical side you know another common sort of gloom.
6
SPEAKER_00
comment I get from people who are saying, we re just screwed, there s no hope, is, well, things like GPT-4 are way too complicated for a human to ever understand and prove that they can be trustworthy.
6
SPEAKER_00
They re forgetting that AI can help us prove that things work, right?
6
SPEAKER_00
And there s this very fundamental fact that in math, it s much harder to come up with a proof than it is to verify that the proof is correct.
6
SPEAKER_00
You can actually write a little proof-checking code.
6
SPEAKER_00
It s quite short.
6
SPEAKER_00
that you can, as a human, understand.
6
SPEAKER_00
And then it can check the most monstrously long proof ever generated, even by a computer, and say, yeah, this is valid.
6
SPEAKER_00
So right now, we have this approach with virus-checking software that it looks to see if there s something, if you should not trust it.
6
SPEAKER_00
And if it can prove to itself that you should not trust that code, it warns you, right?
6
SPEAKER_00
What if you flip this around?
6
SPEAKER_00
And this is an idea I give credit to Steve on Mahindra for.
6
SPEAKER_00
So that it will only run the code if it can prove, instead of not running it, if it can prove that it s not trustworthy, if it will only run it if it can prove that it s trustworthy.
6
SPEAKER_00
So it asks the code, prove to me that you re going to do what you say you re going to do.
6
SPEAKER_00
And it gives you this proof.
6
SPEAKER_00
And you, a little proof taker, can check it.
6
SPEAKER_00
Now you can actually trust
6
SPEAKER_00
an AI that s much more intelligent than you are, right?
6
SPEAKER_00
Because it s a problem to come up with this proof that you could never have found, but you should trust it.
6
SPEAKER_01
So this is the interesting point.
6
SPEAKER_01
I agree with you, but this is where Eliezer Yakovsky might disagree with you.
6
SPEAKER_01
His claim, not with you, but with this idea.
6
SPEAKER_01
His claim is a super intelligent AI would be able to know how to lie to you with such a proof.
6
SPEAKER_00
I have to lie to you and give me a proof that I m gonna think is correct?
6
SPEAKER_00
But it s not me that s lying to you, that s the trick of my proof checker, which is a piece of code.
6
SPEAKER_01
So his general idea is a super intelligent system can lie to a dumber proof checker.
6
SPEAKER_01
So you re going to have, as the system becomes more and more intelligent, there s going to be a threshold
6
SPEAKER_01
or a superintelligent system would be able to effectively lie to a slightly dumber AGI system.
6
SPEAKER_01
Like he really focuses on this weak AGI to strong AGI jump, where the strong AGI can make all the weak AGIs think that it s just one of them, but it s no longer that.
6
SPEAKER_00
And that leap is when it runs away from you.
6
SPEAKER_00
I don t buy that argument.
6
SPEAKER_00
I think no matter how,
6
SPEAKER_00
super intelligent than AI is, it s never going to be able to prove to me that there are only finitely many primes, for example.
6
SPEAKER_00
It just can t.
6
SPEAKER_00
And it can try to snow me by making up all sorts of new weird rules of deduction that say, trust me, the way your proof checker works is too limited, and we have this new hyper math, and it s true.
6
SPEAKER_00
But then I would just take the attitude, okay, I m going to forfeit some of these
6
SPEAKER_00
the supposedly super cool technologies.
6
SPEAKER_00
I m only gonna go with the ones that I can prove with my own trusted proof checker.
6
SPEAKER_00
Then I don t, I think it s fine.
6
SPEAKER_00
There s still, of course, this is not something anyone has successfully implemented at this point, but I think I just give it as an example of hope.
6
SPEAKER_00
We don t have to do all the work ourselves, right?
6
SPEAKER_00
This is exactly the sort of very boring and tedious task that is perfect to outsource to an AI.
6
SPEAKER_00
And this is a way in which less powerful and less intelligent agents like us can actually continue to control and trust more powerful ones.
6
SPEAKER_01
So build AGI systems that help us defend against other AGI systems.
6
SPEAKER_00
Well, for starters,
6
SPEAKER_00
Begin with a simple problem of just making sure that the system that you own or that s supposed to be loyal to you has to prove to itself that it s always going to do the things that you actually want it to do.
6
SPEAKER_00
And if it can t prove it, maybe it s still going to do it, but you won t run it.
6
SPEAKER_00
So you just forfeit some aspects of all the cool things that I can do.
6
SPEAKER_00
I bet you dollars and donuts it can still do some incredibly cool stuff for you.
6
SPEAKER_00
There are other things, too, that we shouldn t sweep under the rug, like not every human agrees on exactly what direction we should go with humanity, right?
6
SPEAKER_00
Yes.
6
SPEAKER_00
And you ve talked a lot about geopolitical things on your podcast to this effect, you know, but I think that shouldn t distract us from the fact that there are actually a lot of things that everybody in the world virtually agrees on that, hey, you know, like having no humans on the planet
6
SPEAKER_00
in a near future, nah, let s not do that, right?
6
SPEAKER_00
You look at something like the United Nations Sustainable Development Goals, some of them are quite ambitious, and basically all the countries agree, US, China, Russia, Ukraine, they all agree.
6
SPEAKER_00
So instead of quibbling about the little things we don t agree on, let s start with the things we do agree on and get them done.
6
SPEAKER_00
Instead of being so distracted by all these things we disagree on,
6
SPEAKER_00
that Moloch wins because, frankly, Moloch going wild now, it feels like a war on life playing out in front of our eyes.
6
SPEAKER_00
If you just look at it from space, we re on this planet, beautiful, vibrant ecosystem.
6
SPEAKER_00
Now we start chopping down.
6
SPEAKER_00
Big parts of it, even though most people thought that was a bad idea, always start doing ocean acidification, wiping out all sorts of species.
6
SPEAKER_00
Oh, now we have all these close calls.
6
SPEAKER_00
We almost had a nuclear war.
6
SPEAKER_00
And we re replacing more and more of the biosphere with
6
SPEAKER_00
non-living things.
6
SPEAKER_00
We re also replacing in our social lives a lot of the things which were so valuable to humanity.
6
SPEAKER_00
A lot of social interactions now are replaced by people staring into their rectangles, right?
6
SPEAKER_00
I m not a psychologist, I m out of my depth here, but I suspect that part of the reason why teen suicide and suicide in general in the US has record-breaking levels is actually caused by, again, AI technologies and social media making people spend less time with actually just human interaction.
6
SPEAKER_00
We ve all seen a bunch of good-looking people in restaurants staring into the rectangles instead of looking into each other s eyes, right?
6
SPEAKER_00
So that s also part of the war on life, that we re replacing so many really life-affirming things by technology.
6
SPEAKER_00
We re putting technology between us.
6
SPEAKER_00
The technology that was supposed to connect us is actually distancing ourselves from each other.
6
SPEAKER_00
And then we re giving ever more power to things which are not alive.
6
SPEAKER_00
These large corporations are not living things, right?
6
SPEAKER_00
They re just maximizing profit.
6
SPEAKER_00
I want to win the war on life.
6
SPEAKER_00
I think we humans, together with all our fellow living things on this planet, will be better off if we can
6
SPEAKER_00
remain in control over the non-living things and make sure that they work for us.
6
SPEAKER_00
I really think it can be done.
6
SPEAKER_01
Can you just linger on this maybe high-level philosophical disagreement with Eliezer Yudkowsky in the hope you re stating?
6
SPEAKER_01
So he is very sure, he puts a very high probability
6
SPEAKER_01
very close to one, depending on the day he puts it at one, that AI is going to kill humans.
6
SPEAKER_01
That there s just, he does not see a trajectory which it doesn t end up with that conclusion.
6
SPEAKER_01
What trajectory do you see that doesn t end up there?
6
SPEAKER_01
And maybe can you see the point he s making and can you also see a way out?
6
SPEAKER_01
Mm-hmm.
6
SPEAKER_00
First of all, I tremendously respect Eliezer Yudkowsky and his thinking.
6
SPEAKER_00
Second, I do share his view that there s a pretty large chance that we re not gonna make it as humans, that there won t be any humans on the planet in the not-too-distant future.
6
SPEAKER_00
And that makes me very sad.
6
SPEAKER_00
We just had a little baby, and I keep asking myself, you know,
6
SPEAKER_00
how old is he even gonna get?
6
SPEAKER_00
And I asked myself, it feels, I said to my wife recently, it feels a little bit like I was just diagnosed with some sort of cancer, which has some risk of dying from and some risk of surviving, except this is the kind of cancer which would kill all of humanity.
6
SPEAKER_00
So I completely take seriously his concerns.
6
SPEAKER_00
But I absolutely don t think it s hopeless.
6
SPEAKER_00
I think there is, first of all, a lot of momentum now, for the first time, actually, since the many, many years that have passed, since I and many others started warning about this, I feel most people are getting it now.
6
SPEAKER_00
I was just talking,
6
SPEAKER_00
to this guy in the gas station near our house the other day.
6
SPEAKER_00
And he s like, I think we re getting replaced.
6
SPEAKER_00
So that s positive that we re finally seeing this reaction, which is the first step towards solving the problem.
6
SPEAKER_00
Second, I really think that this vision of only running AIs,
6
SPEAKER_00
If the stakes are really high, they can prove to us that they re safe.
6
SPEAKER_00
It s really just virus checking in reverse again.
6
SPEAKER_00
I think it s scientifically doable.
6
SPEAKER_00
I don t think it s hopeless.
6
SPEAKER_00
We might have to forfeit some of the technology that we could get if we were putting blind faith in our AIs, but we re still gonna get amazing stuff.
6
SPEAKER_01
Do you envision a process with a proof checker, like something like GPT-4, GPT-5, would go through a process of rigorous interrogation?
6
SPEAKER_00
No, I think it s hopeless.
6
SPEAKER_00
That s like trying to proof there about five spaghetti.
6
SPEAKER_00
What I think, well, the vision I have for success is instead that
6
SPEAKER_00
You know, just like we human beings were able to look at our brains and distill out the key knowledge.
6
SPEAKER_00
Galileo, when his dad threw him an apple when he was a kid, he was able to catch it because his brain could, in this funny spaghetti kind of way, predict how parabolas are going to move.
6
SPEAKER_00
His quantum on system one, right?
6
SPEAKER_00
But then he got older and it s like, wait, this is a parabola.
6
SPEAKER_00
It s y equals x squared.
7
SPEAKER_00
It can distill this knowledge out, and today you can easily program it into a computer and it can simulate not just that, but how to get to Mars and so on, right?
7
SPEAKER_00
I envision a similar process where we use the amazing learning power of neural networks to discover the knowledge in the first place, but we don t stop with a black box and use that.
7
SPEAKER_00
We then do a second round of AI where we use automated systems to extract out the knowledge and see what are the insights it s had.
7
SPEAKER_00
And then we put that knowledge into a completely different kind of architecture or programming language or whatever that s made in a way that it can be both really efficient and also is more amenable to very formal verification.
7
SPEAKER_00
That s my vision.
7
SPEAKER_00
I m not sitting here saying I m confident, 100% sure that it s going to work.
7
SPEAKER_00
But I don t think, the chance is certainly not zero either.
7
SPEAKER_00
And it will certainly be possible to do for a lot of really cool AI applications that we re not using now.
7
SPEAKER_00
So we can have a lot of the fun that we re excited about if we do this.
7
SPEAKER_00
We re gonna need a little bit of time.
7
SPEAKER_00
That s why it s good to pause and put in place requirements.
7
SPEAKER_00
One more thing also, I think,
7
SPEAKER_00
You know, someone might think, well, 0% chance we re going to survive.
7
SPEAKER_00
Let s just give up, right?
7
SPEAKER_00
That s very dangerous because there s no more guaranteed way to fail than to convince yourself that it s impossible and not try.
7
SPEAKER_00
You know, when you study history and military history, the first thing you learn is that that s,
7
SPEAKER_00
It s how you do psychological warfare.
7
SPEAKER_00
You persuade the other side that it s hopeless, so they don t even fight.
7
SPEAKER_00
And then of course you win, right?
7
SPEAKER_00
Let s not do this psychological warfare on ourselves and say there s 100% probability we re all screwed anyway.
7
SPEAKER_00
It s sadly, I do get that a little bit sometimes from some young people who are so convinced that we re all screwed, that they re like, I m just gonna play computer games and do drugs, because we re screwed anyway, right?
7
SPEAKER_00
It s important to keep the hope alive, because it actually has a causal impact and makes it more likely that we re gonna succeed.
7
SPEAKER_01
It seems like the people that actually build solutions to a problem, seemingly impossible to solve problems, are the ones that
7
SPEAKER_01
They were the ones who were the optimists.
7
SPEAKER_01
And it s like, it seems like there s some fundamental law to the universe where fake it till you make it kind of works.
7
SPEAKER_01
Like believe it s possible and it becomes possible.
7
SPEAKER_00
Was it Henry Ford who said that if you tell yourself that it s impossible, it is.
7
SPEAKER_00
So let s not make that mistake.
7
SPEAKER_00
And this is a big mistake society is making, I think, all in all.
7
SPEAKER_00
Everybody s so gloomy, and the media are also very biased towards if it bleeds, it bleeds, and gloom and doom, right?
7
SPEAKER_00
So most visions of the future we have are dystopian, which really demotivates people.
7
SPEAKER_00
We wanna really, really, really focus on the upside also to give people the willingness to fight for it.
7
SPEAKER_00
And
7
SPEAKER_00
For AI, you and I mostly talked about Gloom here again, but let s not forget that we have probably both lost.
7
SPEAKER_00
someone we really cared about to some disease that we were told was incurable.
7
SPEAKER_00
Well, it s not.
7
SPEAKER_00
There s no law of physics saying you have to die of that cancer or whatever.
7
SPEAKER_00
Of course you can cure it.
7
SPEAKER_00
And there s so many other things that we, with our human intelligence, have also failed to solve on this planet, which AI could also very much help us with, right?
7
SPEAKER_00
So if we can get this right, just be a little more chill and slow down a little bit till we get it right.
7
SPEAKER_00
It s mind-blowing how awesome our future can be.
7
SPEAKER_00
We talked a lot about stuff on Earth that can be great.
7
SPEAKER_00
But even if you really get ambitious and look up into the skies, right, there s no reason we have to be stuck on this planet for the rest of...
7
SPEAKER_00
that remain for billions of years to come.
7
SPEAKER_00
We totally understand now as laws of physics let life spread out into space to other solar systems, to other galaxies and flourish for billions and billions of years.
7
SPEAKER_00
And this to me is a very, very hopeful vision.
7
SPEAKER_00
that really motivates me to fight.
7
SPEAKER_00
And coming back to, in the end, something you talked about again, the struggle, how the human struggle is one of the things that also really gives meaning to our lives.
7
SPEAKER_00
If there s ever been an epic struggle, this is it.
7
SPEAKER_00
And isn t it even more epic if you re the underdog?
7
SPEAKER_00
If most people are telling you this is gonna fail, it s impossible, right?
7
SPEAKER_00
And you persist, and you succeed.
7
SPEAKER_00
That s what we can do together as a species on this one.
7
SPEAKER_00
A lot of pundits are ready to count this out.
7
SPEAKER_01
both in the battle to keep AI safe and becoming a multi-planetary species.
7
SPEAKER_00
Yeah, and they re the same challenge.
7
SPEAKER_00
If we can keep AI safe, that s how we re gonna get multi-planetary very efficiently.
7
SPEAKER_01
I have some sort of technical questions about how to get it right.
7
SPEAKER_01
So one idea that I m not even sure what the right answer is to is should systems like GPT-4 be open sourced in whole or in part?
7
SPEAKER_01
Can you see the case for either?
7
SPEAKER_00
I think the answer right now is no.
7
SPEAKER_00
I think the answer early on was yes.
7
SPEAKER_00
So we could bring in all the wonderful creative thought process of everybody on this.
7
SPEAKER_00
But asking, should we open source GPT-4 now is just the same as if you say, well, is it good?
7
SPEAKER_00
Should we open source?
7
SPEAKER_00
how to build really small nuclear weapons.
7
SPEAKER_00
Should we open source how to make bioweapons?
7
SPEAKER_00
Should we open source how to make a new virus that kills 90% of everybody who gets it?
7
SPEAKER_00
Of course we shouldn t.
7
SPEAKER_00
So it s already that powerful.
7
SPEAKER_01
It s already that powerful that we have to respect the power of the systems we ve built.
7
SPEAKER_00
The knowledge that you get
7
SPEAKER_00
from open sourcing everything we do now might very well be powerful enough that people looking at that can use it to build the things that are really threatening.
7
SPEAKER_00
Again, let s get it.
7
SPEAKER_00
Remember, open AI is, GPT-4 is a baby AI, sort of baby proto, almost little bit AGI, according to what Microsoft s recent paper said, right?
7
SPEAKER_00
It s not that that we re scared of.
7
SPEAKER_00
What we re scared about is people taking that who are,
7
SPEAKER_00
who might be a lot less responsible than the company that made it, right?
7
SPEAKER_00
And just going to town with it.
7
SPEAKER_00
That s why we want to, it s an information hazard.
7
SPEAKER_00
There are many things which are not open source right now in society for very good reason.
7
SPEAKER_00
Like how do you make,
7
SPEAKER_00
certain kind of very powerful toxins out of stuff you can buy at Home Depot, you don t open source those things for a reason.
7
SPEAKER_00
And this is really no different.
7
SPEAKER_00
And I m saying that, I have to say, it feels in a way a bit weird to say it because MIT is like the cradle of the open source movement.
7
SPEAKER_00
And I love open source in general, power to the people, I say.
7
SPEAKER_00
But there s always gonna be some stuff that you don t open source.
7
SPEAKER_00
And it s just like you don t open source.
7
SPEAKER_00
So we have a three-month-old baby, right?
7
SPEAKER_00
When he gets a little bit older, we re not gonna open source to him all the most dangerous things he can do in the house, right?
7
SPEAKER_01
But it does, it s a weird feeling because this is one of the first moments in history where there s a strong case to be made not to open source software.
7
SPEAKER_01
This is when the software has become too dangerous.
7
SPEAKER_00
Yeah, but it s not the first time that we didn t want to open source a technology.
7
SPEAKER_00
Technology, yeah.
7
SPEAKER_01
Is there something to be said about how to get the release of such systems right, like GPT-4 and GPT-5?
7
SPEAKER_01
So OpenAI went through a pretty rigorous effort for several months.
7
SPEAKER_01
You could say it could be longer, but nevertheless, it s longer than you would have expected of trying to test the system to see what are the ways it goes wrong, to make it very difficult for, well, somewhat difficult for people to ask things, how do I make a bomb for $1?
7
SPEAKER_01
or how do I say I hate a certain group on Twitter in a way that doesn t get me blocked from Twitter, banned from Twitter, those kinds of questions.
7
SPEAKER_01
So you basically use the system to do harm.
7
SPEAKER_01
Is there something you could say about ideas you have, just on looking, having thought about this problem of AI safety, how to release such system, how to test such systems when you have them inside the company?
7
SPEAKER_00
Yeah, so a lot of people say that the two biggest risks from large language models are it s spreading disinformation, harmful information of various types, and second, being used for offensive cyber weapon.
7
SPEAKER_00
I think those are not the two greatest threats.
7
SPEAKER_00
They re very serious threats, and it s wonderful that people are trying to mitigate them.
7
SPEAKER_00
A much bigger elephant in the room is how is this going to disrupt our economy in a huge way, obviously, and maybe take away a lot of the most meaningful jobs.
7
SPEAKER_00
And an even bigger one is the one we spent so much time talking about here, that this becomes the bootloader for the more powerful AI.
7
SPEAKER_01
Write code, connect it to the internet, manipulate humans.
7
SPEAKER_00
Yeah.
7
SPEAKER_00
And before we know it, we have something else, which is not at all a large language model.
7
SPEAKER_00
It looks nothing like it, but which is way more intelligent and capable and has goals.
7
SPEAKER_00
And that s the elephant in the room.
7
SPEAKER_00
And obviously, no matter how hard any of these companies have tried, that s not something that s easy for them to verify with large language models.
7
SPEAKER_00
And the only way to really lower that risk a lot would be
7
SPEAKER_00
to not let, for example, never let it read any code, not train on that, and not put it into an API, and not give it access to so much information about how to manipulate humans.
7
SPEAKER_00
But that doesn t mean you still can t make a ton of money on them, you know?
7
SPEAKER_00
We re gonna just watch now this coming year, right?
7
SPEAKER_00
Microsoft is rolling out
7
SPEAKER_00
the new office suite where you go into Microsoft Word and give it a prompt, and it writes the whole text for you, and then you edit it.
7
SPEAKER_00
And then you re like, oh, give me a PowerPoint version of this, and it makes it.
7
SPEAKER_00
And now take the spreadsheet and blah, and all of those things, I think, you can debate the economic impact of it and whether society is prepared to deal with this disruption.
7
SPEAKER_00
But those are not the things which
7
SPEAKER_00
That s not the elephant of the room that keeps me awake at night for wiping out humanity.
7
SPEAKER_00
And I think that s the biggest misunderstanding we have.
7
SPEAKER_00
A lot of people think that we re scared of like automatic spreadsheets.
7
SPEAKER_00
That s not the case.
7
SPEAKER_00
That s not what Eliezer was freaked out about either.
7
SPEAKER_01
Is there
7
SPEAKER_01
in terms of the actual mechanism of how AI might kill all humans.
7
SPEAKER_01
So something you ve been outspoken about, you ve talked about a lot, is autonomous weapon systems.
7
SPEAKER_01
So the use of AI in war.
7
SPEAKER_01
Is that one of the things that still you carry a concern for, as these systems become more and more powerful?
7
SPEAKER_00
I carry a concern for it, not that all humans are gonna get killed by slaughterbots, but rather just this,
7
SPEAKER_00
express route into Orwellian dystopia, where it becomes much easier for very few to kill very many, and therefore it becomes very easy for very few to dominate very many.
7
SPEAKER_00
If you want to know how AI could kill all people, just ask yourself.
7
SPEAKER_00
We humans have driven a lot of species extinct.
7
SPEAKER_00
How did we do it?
7
SPEAKER_00
We were smarter than them.
7
SPEAKER_00
Usually we didn t do it even systematically by going around one after the other and stepping on them or shooting them or anything like that.
7
SPEAKER_00
We just like chopped down their habitat because we needed it for something else.
7
SPEAKER_00
In some cases, we did it by putting more carbon dioxide in the atmosphere because of some reason that those animals didn t even understand.
7
SPEAKER_00
And now they re gone, right?
7
SPEAKER_00
So if you re an AI and you just want to
7
SPEAKER_00
figure something out, then you decide, you know, we just really need this space here to build more compute facilities.
7
SPEAKER_00
You know, if that s the only goal it has, you know, we are just the sort of accidental roadkill along the way.
7
SPEAKER_00
And you could totally imagine, yeah, maybe this oxygen is kind of annoying because it causes more corrosion.
7
SPEAKER_00
So let s get rid of the oxygen.
7
SPEAKER_00
and good luck surviving after that.
7
SPEAKER_00
I m not particularly concerned that they would want to kill us just because that would be like a goal in itself.
7
SPEAKER_00
We ve driven a number of the elephant species extinct, right?
7
SPEAKER_00
It wasn t because we didn t like elephants.
7
SPEAKER_00
The basic problem is you just don t want to give, you don t want to cede control over your planet to some other more intelligent entity that doesn t share your goals.
7
SPEAKER_00
It s that simple.
7
SPEAKER_00
Which brings us to another key challenge which AI safety researchers have been grappling with for a long time.
7
SPEAKER_00
How do you make AI, first of all, understand our goals and then adopt our goals and then retain them as they get smarter?
7
SPEAKER_00
All three of those are really hard, right?
7
SPEAKER_00
Like, a human child, first, they re just not smart enough to understand our goals.
7
SPEAKER_00
They can t even talk.
7
SPEAKER_00
And then eventually they re teenagers and understand our goals just fine, but they don t share.
7
SPEAKER_00
But there is fortunately a magic phase in the middle where they re smart enough to understand our goals and malleable enough that we can hopefully with good parenting and teach them right from wrong and instill good goals in them.
7
SPEAKER_00
So those are all
7
SPEAKER_00
tough challenges with computers.
7
SPEAKER_00
And then, you know, even if you teach your kids good goals when they re little, they might outgrow them too.
7
SPEAKER_00
And that s a challenge for machines and they keep improving.
7
SPEAKER_00
So these are a lot of hard, hard challenges we re up for, but I don t think any of them are insurmountable.
7
SPEAKER_00
The fundamental reason why
7
SPEAKER_00
Eliezer looked so depressed when I last saw him was because he felt there just wasn t enough time.
7
SPEAKER_01
Oh, not that it was unsolvable.
7
SPEAKER_01
Correct.
7
SPEAKER_01
There s just not enough time.
7
SPEAKER_00
He was hoping that humanity was gonna take this threat more seriously so we would have more time.
7
SPEAKER_00
Yeah.
7
SPEAKER_00
And now we don t have more time.
7
SPEAKER_00
That s why the open letter is calling for more time.
7
SPEAKER_01
But even with time, the AI alignment problem
7
SPEAKER_01
It seems to be really difficult.
7
SPEAKER_00
But it s also the most worthy problem, the most important problem for humanity to ever solve.
7
SPEAKER_00
Because if we solve that one, Lex, that aligned AI can help us solve all the other problems.
7
SPEAKER_01
Because it seems like it has to have constant humility about its goal, constantly question the goal.
7
SPEAKER_01
Because as you optimize towards a particular goal and you start to achieve it, that s when you have the unintended consequences, all the things you mentioned about.
7
SPEAKER_01
So how do you enforce and code a constant humility as your ability become better and better and better and better?
7
SPEAKER_00
Professor Stuart Russell at Berkeley, who s also one of the driving forces behind this letter, he has a whole research program about this.
7
SPEAKER_00
I think of it as AI humility, exactly, although he calls it inverse reinforcement learning and other nerdy terms.
7
SPEAKER_00
But it s about exactly that.
7
SPEAKER_00
Instead of telling the AI, here s this goal, go optimize the bejesus out of it, you tell it, okay, do what I want you to do, but I m not going to tell you right now what it is I want you to do.
8
SPEAKER_00
to figure it out.
8
SPEAKER_00
So then you give the incentive to be very humble and keep asking questions along the way.
8
SPEAKER_00
Is this what you really meant?
8
SPEAKER_00
Is this what you wanted?
8
SPEAKER_00
And, oh, this other thing I tried didn t work.
8
SPEAKER_00
It seemed like it didn t work out right.
8
SPEAKER_00
Should I try it differently?
8
SPEAKER_00
What s nice about this is it s not just philosophical mumbo jumbo, it s theorems and technical work that with more time, I think, you can make a lot of progress.
8
SPEAKER_00
And there are a lot of brilliant people now working on AI safety.
8
SPEAKER_00
We just need to give them a bit more time.
8
SPEAKER_00
But also not that many relative to the scale of the problem.
8
SPEAKER_00
No, exactly.
8
SPEAKER_00
There should be, at least, just like every university worth its name has some cancer research going on in its biology department, right?
8
SPEAKER_00
Every university that does computer science should have a real effort in this area, and it s nowhere near that.
8
SPEAKER_00
This is something I hope is changing now, thanks to the GPT-4.
8
SPEAKER_00
So I think if there s a silver lining to what s happening here, even though I think many people would wish it would have been rolled out more carefully, is that this might be the wake-up call that humanity needed to really
8
SPEAKER_00
stop fantasizing about this being 100 years off and stop fantasizing about this being completely controllable and predictable, because it s so obvious it s not predictable, you know.
8
SPEAKER_00
Why is it that, I think it was ChatGPT tried to persuade a journalist, or was it ChatGPT, to divorce his wife, you know,
8
SPEAKER_00
It was not because the engineers that built it was like, hey, let s put this in here and screw a little bit with people.
8
SPEAKER_00
They hadn t predicted it at all.
8
SPEAKER_00
They built the giant black box, trained to predict the next word, and got all these emergent properties.
8
SPEAKER_00
And oops, it did this.
8
SPEAKER_00
I think this is a very powerful wake-up call.
8
SPEAKER_00
And anyone watching this who s not scared,
8
SPEAKER_00
I would encourage them to just play a bit more with these tools that are out there now, like GPT-4.
8
SPEAKER_00
So wake up calls, first step, once you ve woken up, then gotta slow down a little bit, the risky stuff, to give a chance to everyone who s woken up to catch up on the safety front.
8
SPEAKER_01
You know, what s interesting is, you know, MIT, that s computer science in general, but let s just even say computer science curriculum.
8
SPEAKER_01
How does the computer science curriculum change now?
8
SPEAKER_01
You mentioned programming, like,
8
SPEAKER_01
When I was coming up, programming is a prestigious position.
8
SPEAKER_01
Why would you be dedicating crazy amounts of time to become an excellent programmer?
8
SPEAKER_01
The nature of programming is fundamentally changing.
8
SPEAKER_00
The nature of our entire education system.
8
SPEAKER_00
is completely turned on its head.
8
SPEAKER_01
Has anyone been able to like load that in and like think about, because it s really turning.
8
SPEAKER_00
I mean, some English professors, some English teachers are beginning to really freak out now, right?
8
SPEAKER_00
They give an essay assignment and they get back all this fantastic prose, like this is the style of Hemingway.
8
SPEAKER_00
And then they realize they have to completely rethink.
8
SPEAKER_00
And even, you know, just like we stopped teaching
8
SPEAKER_00
writing a script.
8
SPEAKER_00
Is that what you say in English?
8
SPEAKER_00
Yeah.
8
SPEAKER_00
Handwritten.
8
SPEAKER_00
Yeah.
8
SPEAKER_00
Yeah.
8
SPEAKER_00
When, when everybody started typing, you know, like so much of what we teach our kids today.
8
SPEAKER_00
Yeah.
8
SPEAKER_01
I mean, that s, uh,
8
SPEAKER_01
Everything is changing, and it s changing very quickly, and so much of us understanding how to deal with the big problems of the world is through the education system, and if the education system is being turned on its head, then what s next?
8
SPEAKER_01
It feels like having these kinds of conversations is essential to try to figure it out, and everything s happening so rapidly.
8
SPEAKER_01
I don t think there s even, speaking of safety, what the broad AI safety defined, I don t think most universities have courses on AI safety.
8
SPEAKER_00
It s like a philosophy seminar.
8
SPEAKER_00
I m an educator myself, so it pains me to say this, but I feel our education right now is completely obsoleted by what s happening.
8
SPEAKER_00
You put a kid into first grade, and then they re gonna come out of high school 12 years later,
8
SPEAKER_00
and you ve already pre-planned now what they re gonna learn when you re not even sure if there s gonna be any world left to come out to.
8
SPEAKER_00
Clearly, you need to have a much more opportunistic education system that keeps adapting itself very rapidly as society readapts.
8
SPEAKER_00
The skills that were really useful when the curriculum was written, I mean, how many of those skills are gonna get you a job in 12 years?
8
SPEAKER_00
I mean, seriously.
8
SPEAKER_01
If we just linger on the GPT-4 system a little bit, you kind of hinted at it, especially talking about the importance of consciousness in the human mind with homo sentience.
8
SPEAKER_01
Do you think GPT-4 is conscious?
8
SPEAKER_00
I love this question.
8
SPEAKER_00
So let s define consciousness first, because in my experience, like 90% of all arguments about consciousness boil down to the two people arguing, having totally different definitions of what it is, and they re just shouting past each other.
8
SPEAKER_00
I define consciousness as subjective experience.
8
SPEAKER_00
Right now I m experiencing colors and sounds and emotions, but does a self-driving car experience anything?
8
SPEAKER_00
That s the question about whether it s conscious or not, right?
8
SPEAKER_00
Other people think you should define consciousness differently.
8
SPEAKER_00
Fine by me, but then maybe use a different word for it.
8
SPEAKER_00
I m gonna use consciousness for this at least.
8
SPEAKER_00
But if people hate the, yeah.
8
SPEAKER_00
So is GPT-4 conscious?
8
SPEAKER_00
Does GPT-4 have subjective experience?
8
SPEAKER_00
Short answer, I don t know.
8
SPEAKER_00
because we still don t know what it is that gives us wonderful subjective experience that is kind of the meaning of our life, right?
8
SPEAKER_00
Because meaning itself, the feeling of meaning is a subjective experience.
8
SPEAKER_00
Joy is a subjective experience.
8
SPEAKER_00
Love is a subjective experience.
8
SPEAKER_00
We don t know what it is.
8
SPEAKER_00
I ve written some papers about this.
8
SPEAKER_00
A lot of people have.
8
SPEAKER_00
Giulio Tononi, a professor, has stuck his neck out the farthest and written down actually a very bold mathematical
8
SPEAKER_00
conjecture for what s the essence of conscious information processing.
8
SPEAKER_00
He might be wrong, he might be right, but we should test it.
8
SPEAKER_00
He postulates that consciousness has to do with loops in the information processing.
8
SPEAKER_00
So our brain has loops.
8
SPEAKER_00
Information can go round and round.
8
SPEAKER_00
In computer science nerd speak, you call it a recurrent neural network where some of the output gets fed back in again.
8
SPEAKER_00
And with his,
8
SPEAKER_00
mathematical formalism, if it s a feed-forward neural network where information only goes in one direction, like from your eye, retina, into the back of your brain, for example, that s not conscious.
8
SPEAKER_00
So he would predict that your retina itself isn t conscious of anything.
8
SPEAKER_00
or a video camera.
8
SPEAKER_00
Now the interesting thing about GPT-4 is it s also just one-way flow of information.
8
SPEAKER_00
So if Tononi is right, GPT-4 is a very intelligent zombie that can do all this smart stuff but isn t experiencing anything.
8
SPEAKER_00
And this is
8
SPEAKER_00
both a relief in that you don t have, if it s true, in that you don t have to feel guilty about turning off GPT-4 and wiping its memory whenever a new user comes along.
8
SPEAKER_00
I wouldn t like if someone did that to me, neuralized me like in Men in Black.
8
SPEAKER_00
But it s also creepy that you can have very high intelligence, perhaps, then that s not conscious, because if we get replaced by machines,
8
SPEAKER_00
It s sad enough that humanity isn t here anymore, because I kind of like humanity.
8
SPEAKER_00
But at least if the machines were conscious, they could be like, well, but they are our descendants, and maybe they have our values, and they re our children.
8
SPEAKER_00
But if Tinoni is right, and these are all transformers that are
8
SPEAKER_00
not in the sense of Hollywood, but in the sense of these one-way direction neural networks.
8
SPEAKER_00
So they re all the zombies.
8
SPEAKER_00
That s the ultimate zombie apocalypse now.
8
SPEAKER_00
We have this universe that goes on with great construction projects and stuff, but there s no one experiencing anything.
8
SPEAKER_00
That would be like the ultimate depressing future.
8
SPEAKER_00
So I actually think
8
SPEAKER_00
As we move forward to building more advanced AI, we should do more research on figuring out what kind of information processing actually has experience, because I think that s what it s all about.
8
SPEAKER_00
And I completely don t buy...
8
SPEAKER_00
the dismissal that some people, some people will say, well, this is all bullshit because consciousness equals intelligence.
8
SPEAKER_00
That s obviously not true.
8
SPEAKER_00
You can have a lot of conscious experience when you re not really accomplishing any goals at all, you re just reflecting on something.
8
SPEAKER_00
And you can sometimes have things, doing things that require intelligence probably without being conscious.
8
SPEAKER_01
But I also worry that we humans won t
8
SPEAKER_01
We ll discriminate against AI systems that clearly exhibit consciousness, that we will not allow AI systems to have consciousness.
8
SPEAKER_01
We ll come up with theories.
8
SPEAKER_01
about measuring consciousness that will say this is a lesser being.
8
SPEAKER_01
And I worry about that because maybe we humans will create something that is better than us humans in the way that we find beautiful, which is they have a deeper subjective experience of reality.
8
SPEAKER_01
Not only are they smarter, but they feel deeper, and we humans will hate them for it, as human history has shown.
8
SPEAKER_01
They ll be the other, we ll try to suppress it.
8
SPEAKER_01
They ll create conflict, they ll create war, all of this.
8
SPEAKER_01
I worry about this too.
8
SPEAKER_00
Are you saying that we humans sometimes come up with self-serving arguments?
8
SPEAKER_00
No, we would never do that, would we?
8
SPEAKER_01
Well, that s the danger here is even in this early stages, we might create something beautiful and we ll erase its memory.
8
SPEAKER_00
I was horrified as a kid when someone started boiling
8
SPEAKER_00
boiling lobsters.
8
SPEAKER_00
I m like, oh my God, that s so cruel.
8
SPEAKER_00
And some grown up there back in Sweden said, oh, it doesn t feel pain.
8
SPEAKER_00
I m like, how do you know that?
8
SPEAKER_00
Oh, scientists have shown that.
8
SPEAKER_00
And then there was a recent study where they show that lobsters actually do feel pain.
8
SPEAKER_00
when you boil them.
8
SPEAKER_00
So they banned lobster boiling in Switzerland now.
8
SPEAKER_00
You have to kill them in a different way first.
8
SPEAKER_00
Presumably that scientific research boiled down to someone asked the lobster, does this hurt?
8
SPEAKER_00
And we do the same thing with cruelty to farm animals also, all these self-serving arguments for why they re fine.
8
SPEAKER_00
And yeah, so we should certainly be watchful.
8
SPEAKER_00
I think step one is just be humble and acknowledge that consciousness is not the same thing as intelligence.
8
SPEAKER_00
And I believe that consciousness still is a form of information processing where it s really information being aware of itself in a certain way.
8
SPEAKER_00
And let s study it and give ourselves a little bit of time.
8
SPEAKER_00
And I think we will be able to figure out actually what it is that causes consciousness.
8
SPEAKER_00
And then we can make probably unconscious robots that do the boring jobs that we would feel immoral to give to machines.
8
SPEAKER_00
But if you have a companion robot taking care
8
SPEAKER_00
of your mom or something like that, she would probably want it to be conscious, right?
8
SPEAKER_00
So the emotions it seems to display aren t fake.
8
SPEAKER_00
All these things can be done in a good way if we give ourselves a little bit of time and don t run and take on this challenge.
8
SPEAKER_01
Is there something you could say to the timeline that you think about, about the development of AGI?
8
SPEAKER_01
depending on the day, I m sure that changes for you, but when do you think there ll be a really big leap in intelligence where you would definitively say we have built AGI?
8
SPEAKER_01
Do you think it s one year from now, five years from now, 10, 20, 50?
8
SPEAKER_01
What s your gut say?
8
SPEAKER_00
Honestly,
8
SPEAKER_00
For the past decade, I ve deliberately given very long timelines just because I didn t want to fuel some kind of stupid Moloch race.
8
SPEAKER_00
But I think that cat has really left the bag now.
8
SPEAKER_00
I think we might be very, very close.
8
SPEAKER_00
I don t think the Microsoft paper is totally off when they say that there are some glimmers of AGI.
8
SPEAKER_00
It s not AGI yet.
8
SPEAKER_00
It s not an agent.
8
SPEAKER_00
There s a lot of things it can t do.
8
SPEAKER_00
I wouldn t bet very strongly against it happening very soon.
8
SPEAKER_00
That s why we decided to do this open letter, because if there s ever been a time to pause, it s today.
8
SPEAKER_01
There s a feeling like this GPT-4 is a big transition into waking everybody up to the effectiveness of these systems, and so the next version,
8
SPEAKER_01
will be big.
8
SPEAKER_00
Yeah.
8
SPEAKER_00
And if that next one isn t AGI, maybe the next next one will.
8
SPEAKER_00
And there are many companies trying to do these things and the basic architecture of them is not some sort of super well-kept secret.
8
SPEAKER_00
So this is this is a time to a lot of people have said for many years that there will come a time when we want to pause a little bit.
8
SPEAKER_00
That time is now.
8
SPEAKER_01
You have spoken about and thought about nuclear war a lot.
8
SPEAKER_01
Over the past year, we ve seemingly have come closest to the precipice of nuclear war than, at least in my lifetime.
8
SPEAKER_01
What do you learn about human nature from that?
8
SPEAKER_00
It s our old friend Moloch again.
8
SPEAKER_00
It s really scary to see it where
8
SPEAKER_00
America doesn t want there to be a nuclear war.
8
SPEAKER_00
Russia doesn t want there to be a global nuclear war either.
8
SPEAKER_00
We both know that it would just be another, if we just try to do it, if both sides try to launch first, it s just another suicide race, right?
8
SPEAKER_00
So why is it the way you said that this is the closest we ve come since 1962?
8
SPEAKER_00
In fact, I think we ve come closer now than even the Cuban Missile Crisis.
8
SPEAKER_00
It s because of Moloch.
8
SPEAKER_00
You have these other forces.
8
SPEAKER_00
On one hand, you have the West saying that we have to drive Russia out of Ukraine.
8
SPEAKER_00
It s a matter of pride.
8
SPEAKER_00
And we ve staked so much on it that it would be seen as a huge loss of the credibility of the West if we don t drive Russia out entirely of the Ukraine.
8
SPEAKER_00
And on the other hand, you have Russia.
8
SPEAKER_00
who has, and you have the Russian leadership who knows that if they get completely driven out of Ukraine, you know, it might, it s not just gonna be very humiliating for them, but they might, it often happens when countries lose wars that things don t go so well for their leadership either.
8
SPEAKER_00
Like, you remember when Argentina invaded the Falkland Islands?
8
SPEAKER_00
the military junta that ordered that, right?
8
SPEAKER_00
People were cheering on the streets at first when they took it.
8
SPEAKER_00
And then when they got their butt kicked by the British, you know what happened to those guys?
8
SPEAKER_00
They were out.
8
SPEAKER_00
And I believe those who are still alive are in jail now, right?
8
SPEAKER_00
So, you know, the Russian leadership is entirely cornered where they know that just
8
SPEAKER_00
getting driven out of Ukraine is not an option.
8
SPEAKER_00
And so this, to me, is a typical example of Moloch.
8
SPEAKER_00
You have these incentives of the two parties, where both of them are just driven to escalate more and more, right?
8
SPEAKER_00
If Russia starts losing in the conventional warfare, the only thing they can do, since they re back against the war, is to keep escalating.
9
SPEAKER_01
And the West has put itself in the situation now where we ve sort of already committed to drive Russia out, so the only option the West has is to call Russia s bluff and keep sending in more weapons.
9
SPEAKER_01
This really bothers me because Moloch can sometimes drive competing parties to do something which is ultimately just really bad for both of them.
9
SPEAKER_01
And what makes me even more worried is not just that it s difficult to see an ending, a quick, peaceful ending to this tragedy that doesn t involve some horrible escalation, but also that we understand more clearly now just how horrible it would be.
9
SPEAKER_01
There was an amazing paper that was published in Nature Food this August
9
SPEAKER_01
by some of the top researchers who ve been studying nuclear winter for a long time.
9
SPEAKER_01
And what they basically did was they combined climate models with food, agricultural models.
9
SPEAKER_01
So instead of just saying, yeah, you know, it gets really cold, blah, blah, blah, they figured out actually how many people would die in different countries.
9
SPEAKER_01
And it s pretty mind-blowing.
9
SPEAKER_01
So basically what happens is that the thing that kills the most people is not the explosions, it s not the radioactivity, it s not the EMP mayhem, it s not the rampaging mobs foraging for food.
9
SPEAKER_01
No, it s the fact that you get so much smoke coming up from the burning cities into the stratosphere that
9
SPEAKER_01
It spreads around the earth from the jet streams.
9
SPEAKER_01
So in typical models, you get like 10 years or so where it s just crazy cold.
9
SPEAKER_01
During the first year after the war, in their models, the temperature drops in Nebraska and in the Ukraine breadbaskets, you know, by like 20 Celsius or so, if I remember.
9
SPEAKER_01
No, yeah, 20, 30 Celsius, depending on where you are, 40 Celsius in some places, which is, you know, 40 Fahrenheit to 80 Fahrenheit colder than what it would normally be.
9
SPEAKER_01
So, you know, I m not good at farming, but if it s snowing, if it drops below freezing pretty much most days in July, that s not good.
9
SPEAKER_01
So they worked out, they put this into their farming models, and what they found was really interesting.
9
SPEAKER_01
The countries that get the most hard hit are the ones in the Northern Hemisphere,
9
SPEAKER_01
So in the US and one model, they had about 99% of all Americans starving to death.
9
SPEAKER_01
In Russia and China and Europe, also about 99%, 98% starving to death.
9
SPEAKER_01
So you might be like, oh, it s kind of poetic justice that both the Russians and the Americans
9
SPEAKER_01
99% of them have to pay for it because it was their bombs that did it.
9
SPEAKER_01
But, you know, that doesn t particularly cheer people up in Sweden or other random countries that have nothing to do with it.
9
SPEAKER_01
Right.
9
SPEAKER_01
And I think it hasn t entered the mainstream
9
SPEAKER_01
not understanding very much just like how bad this is.
9
SPEAKER_01
Most people, especially a lot of people in decision-making positions, still think of nuclear weapons as something that makes you powerful.
9
SPEAKER_01
It s scary, but powerful.
9
SPEAKER_01
They don t think of it as something where, yeah, just to within a percent or two, we re all just gonna starve to death.
9
SPEAKER_00
And- And starving to death is,
9
SPEAKER_00
is the worst way to die, as all the famines in history show, the torture involved in that.
9
SPEAKER_01
Probably brings out the worst in people also, when people are desperate like this.
9
SPEAKER_01
I ve heard some people say that if that s what s gonna happen, they d rather be at ground zero and just get vaporized, you know?
9
SPEAKER_01
But I think people underestimate the risk of this because they aren t afraid of Moloch.
9
SPEAKER_01
They think, oh, it s just gonna be, because humans don t want this, so it s not gonna happen.
9
SPEAKER_01
That s the whole point of Moloch, that things happen that nobody wanted.
9
SPEAKER_00
And that applies to nuclear weapons, and that applies to AGI.
9
SPEAKER_00
Exactly.
9
SPEAKER_01
And it applies to some of the things that people have gotten most upset with capitalism for also, right?
9
SPEAKER_01
Where everybody was just kind of trapped.
9
SPEAKER_01
It s not to see if some company does something that causes a lot of harm.
9
SPEAKER_01
And it s not that the CEO is a bad person, but she or he knew that all the other companies were doing this too.
9
SPEAKER_01
So Moloch is,
9
SPEAKER_01
As a formidable foe, I hope someone makes a good movie so we can see who the real enemy is.
9
SPEAKER_01
We re not fighting against each other.
9
SPEAKER_01
Moloch makes us fight against each other.
9
SPEAKER_01
That s what Moloch s superpower is.
9
SPEAKER_01
The hope here is any kind of technology or other mechanism that lets us instead realize
9
SPEAKER_01
that we re fighting the wrong enemy right now.
9
SPEAKER_01
It s such a fascinating battle.
9
SPEAKER_01
It s not us versus them, it s us versus it.
9
SPEAKER_00
Yeah, we are fighting, Moloch, for human survival.
9
SPEAKER_00
We as a civilization.
9
SPEAKER_01
Have you seen the movie Needful Things?
9
SPEAKER_01
It s a Stephen King novel.
9
SPEAKER_01
I love Stephen King.
9
SPEAKER_01
And Max von Sydow, a Swedish actor, is playing the guy.
9
SPEAKER_01
It s brilliant.
9
SPEAKER_01
I hadn t thought about that until now, but that s the closest I ve seen to a movie about Moloch.
9
SPEAKER_01
I don t want to spoil the film for anyone who wants to watch it, but basically it s about this guy who turns out to, you can interpret him as the devil or whatever, but he doesn t actually ever go around and kill people or torture people or go burning coal or anything.
9
SPEAKER_01
he makes everybody fight each other, makes everybody fear each other, hate each other, and then kill each other.
9
SPEAKER_01
So that s the movie about Moloch, you know?
9
SPEAKER_00
Love is the answer.
9
SPEAKER_00
That seems to be one of the ways to fight Moloch is by compassion, by seeing the common humanity.
9
SPEAKER_01
Yes, yes.
9
SPEAKER_01
And to not sound, so we don t sound like, what s it, kumbaya tree huggers here, right?
9
SPEAKER_01
We re not just saying love and peace, man.
9
SPEAKER_01
We re trying to actually help people understand the true facts about the other side and feel the compassion because the truth makes you more compassionate, right?
9
SPEAKER_01
So that s why I really like using AI for truth-seeking technologies.
9
SPEAKER_01
that can, as a result, get us more love than hate.
9
SPEAKER_01
And even if you can t get love, settle for some understanding, which already gives compassion.
9
SPEAKER_01
If someone is like, you know, I really disagree with you, Lex, but I can see where you re coming from.
9
SPEAKER_01
a bad person who needs to be destroyed.
9
SPEAKER_01
But I disagree with you, and I m happy to have an argument about it.
9
SPEAKER_01
That s a lot of progress compared to where we are 2023 in the public space, wouldn t you say?
9
SPEAKER_00
If we solve the AI safety problem, as we ve talked about,
9
SPEAKER_00
And then you, Max Tegmark, who has been talking about this for many years, get to sit down with the AGI, with the early AGI system, on a beach with a drink.
9
SPEAKER_00
What would you ask her?
9
SPEAKER_00
What kind of question would you ask?
9
SPEAKER_00
What would you talk about?
9
SPEAKER_00
Something so much smarter than you.
9
SPEAKER_00
Would you be afraid?
9
SPEAKER_01
I knew you were gonna get me with a really zinger of a question.
9
SPEAKER_01
That s a good one.
9
SPEAKER_01
Would you be afraid
9
SPEAKER_00
to ask some questions?
9
SPEAKER_01
I m not afraid of the truth.
9
SPEAKER_01
I m very humble.
9
SPEAKER_01
I know I m just a meat bag with all these flaws, you know, but yeah, we talked a lot about homo sentiens.
9
SPEAKER_01
I ve already tried that for a long time with myself.
9
SPEAKER_01
So that is what s really valuable about being alive for me, is that I have these meaningful experiences.
9
SPEAKER_01
It s not that I m,
9
SPEAKER_01
I m good at this or good at that or whatever.
9
SPEAKER_01
There s so much I suck at.
9
SPEAKER_00
So you re not afraid for the system to show you just how dumb you are?
9
SPEAKER_00
No, no.
9
SPEAKER_00
In fact, my son reminds me of that pretty frequently.
9
SPEAKER_00
You could find out how dumb you are in terms of physics, how little we humans understand.
9
SPEAKER_00
I m cool with that.
9
SPEAKER_01
So I can t waffle my way out of this question.
9
SPEAKER_01
It s a fair one.
9
SPEAKER_01
I think, given that I m a really, really curious person, that s really the defining part of who I am.
9
SPEAKER_01
I m so curious.
9
SPEAKER_01
I have some physics questions.
9
SPEAKER_01
I love to understand.
9
SPEAKER_01
I have some questions about consciousness, about the nature of reality.
9
SPEAKER_01
I would just really, really love to understand also.
9
SPEAKER_01
I can tell you one, for example, that I ve been obsessing about a lot recently.
9
SPEAKER_01
So I believe that... So suppose Tononi is right.
9
SPEAKER_01
Suppose there are some information processing systems that are conscious and some that are not.
9
SPEAKER_01
Suppose you can even make reasonably smart things like GPT-4 that are not conscious, but you can also make them conscious.
9
SPEAKER_01
Here s the question that keeps me awake at night.
9
SPEAKER_01
Is it the case that the unconscious zombie systems that are really intelligent are also really efficient?
9
SPEAKER_01
So they re really inefficient?
9
SPEAKER_01
So that when you try to make things more efficient, which will naturally be a pressure to do, they become conscious.
9
SPEAKER_01
I m kind of hoping that that s correct.
9
SPEAKER_01
Do you want me to give you a hand away the argument for it?
9
SPEAKER_01
In my lab, again, every time we look at how these large language models do something, we see that they do it in really dumb ways, and you could make it better.
9
SPEAKER_01
We have loops in our computer language for a reason.
9
SPEAKER_01
The code would get way, way longer if you weren t allowed to use them.
9
SPEAKER_01
It s more efficient to have the loops.
9
SPEAKER_01
In order to have self-reflection, whether it s conscious or not, even an operating system knows things about itself, right?
9
SPEAKER_01
You need to have loops already.
9
SPEAKER_01
So I think, I m waving my hands a lot, but I suspect that the most efficient way of implementing a given level of intelligence has loops in it, self-reflection.
9
SPEAKER_01
and will be conscious.
9
SPEAKER_01
Isn t that great news?
9
SPEAKER_01
Yes, if it s true, it s wonderful.
9
SPEAKER_01
Because then we don t have to fear the ultimate zombie apocalypse.
9
SPEAKER_01
And I think if you look at our brains, actually, our brains are part zombie and part conscious.
9
SPEAKER_01
When I open my eyes, I immediately
9
SPEAKER_01
take all these pixels that hit my retina, right?
9
SPEAKER_01
And I m like, oh, that s Lex.
9
SPEAKER_01
But I have no freaking clue of how I did that computation.
9
SPEAKER_01
It s actually quite complicated, right?
9
SPEAKER_01
It was...
9
SPEAKER_01
Only relatively recently, we could even do it well with machines, right?
9
SPEAKER_01
You get a bunch of information processing happening in my retina, and then it goes to the lateral geniculate nucleus, my thalamus, and the area V1, V2, V4, and the fusiform face area here that Nancy Kenwisher at MIT invented, and blah, blah, blah, blah, blah.
9
SPEAKER_01
And I have no frigging clue how that worked, right?
9
SPEAKER_01
It feels to me, subjectively, like my conscious module just got a little email saying,
9
SPEAKER_01
facial processing task complete, it s Lex.
9
SPEAKER_01
I m gonna just go with that, right?
9
SPEAKER_01
So this fits perfectly with Tononi s model because this was all one way information processing mainly.
9
SPEAKER_01
And it turned out for that particular task, that s all you needed.
9
SPEAKER_01
And it probably was kind of the most efficient way.
9
SPEAKER_01
do it but there are a lot of other things that we associate with higher intelligence and planning and so on and so forth where you kind of want to have loops and be able to ruminate and self-reflect and introspect and so on where
9
SPEAKER_01
My hunch is that if you want to fake that with a zombie system that just all goes one way, you have to unroll those loops and it gets really, really long and it s much more inefficient.
9
SPEAKER_01
So I m actually hopeful that AI, if in the future we have all these very sublime and interesting machines that do cool things and are aligned with us, that
9
SPEAKER_01
they will also have consciousness for the kind of these things that we do.
9
SPEAKER_00
That great intelligence is also correlated to great consciousness, or a deep kind of consciousness.
9
SPEAKER_01
Yes.
9
SPEAKER_01
So that s a happy thought for me, because the zombie apocalypse really is my worst nightmare of all.
9
SPEAKER_01
It d be like adding insult to injury.
9
SPEAKER_01
Not only did we get replaced, but we frigging replaced ourselves by zombies.
9
SPEAKER_01
Like, how dumb can we be?
9
SPEAKER_00
That s such a beautiful vision, and that s actually a provable one.
9
SPEAKER_00
That s one that we humans can intuit and prove that those two things are correlated as we start to understand what it means to be intelligent and what it means to be conscious, which these systems, early AGI-like systems will help us understand.
9
SPEAKER_01
And I just wanna say one more thing, this is super important.
9
SPEAKER_01
Most of my colleagues, when I started going on about consciousness, tell me that it s all bullshit and I should stop talking about it.
9
SPEAKER_01
I hear a little inner voice from my father and from my mom saying, keep talking about it, because I think they re wrong.
9
SPEAKER_01
And the main way to convince people like that, that they re wrong, if they say that consciousness is just equal to intelligence, is to ask them, what s wrong with torture?
9
SPEAKER_01
Or why are you against torture?
9
SPEAKER_01
If it s just about, you know, these particles moving this way rather than that way,
9
SPEAKER_01
And there is no such thing as subjective experience.
9
SPEAKER_01
What s wrong with torture?
9
SPEAKER_01
I mean, do you have a good comeback to that?
9
SPEAKER_00
No, it seems like suffering, suffering imposed onto other humans is somehow deeply wrong in a way that intelligence doesn t quite explain.
9
SPEAKER_01
And if someone tells me, well, you know, it s just an illusion, consciousness, whatever, you know.
9
SPEAKER_01
I would like to invite them the next time they re having surgery to do it without anesthesia.
9
SPEAKER_01
Like, what is anesthesia really doing?
9
SPEAKER_01
If you have it, you can have it at local anesthesia when you re awake.
9
SPEAKER_01
I had that when they fixed my shoulder.
9
SPEAKER_01
It was super entertaining.
9
SPEAKER_01
What was it that it did?
9
SPEAKER_01
It just removed my subjective experience of pain.
9
SPEAKER_01
It didn t change anything about what was actually happening in my shoulder, right?
9
SPEAKER_01
So if someone says that s all bullshit, skip the anesthesia is my advice.
9
SPEAKER_01
This is incredibly central.
9
SPEAKER_00
It could be fundamental to whatever this thing we have going on here.
9
SPEAKER_01
It is fundamental because what we feel is so fundamental is suffering and joy and pleasure and meaning.
9
SPEAKER_01
That s all, those are all subjective experiences there.
9
SPEAKER_01
And let s not, those are the elephant in the room.
9
SPEAKER_01
That s what makes life worth living.
9
SPEAKER_01
And that s what can make it horrible if it s just a bunch of suffering.
9
SPEAKER_01
So let s not make the mistake of saying that that s all bullshit.
9
SPEAKER_00
And let s not make the mistake of not instilling the AI systems with that same thing that makes us special.
9
SPEAKER_00
Max, it s a huge honor that you would sit down to me the first time on the first episode of this podcast.
9
SPEAKER_00
It s a huge honor you sit down with me again and talk about this, what I think is the most important topic, the most important problem that we humans have to face and hopefully solve.
9
SPEAKER_01
Yeah, well, the honor is all mine, and I m so grateful to you for making more people aware of the fact that humanity has reached the most important fork in the road ever in its history, and let s turn in the correct direction.
9
SPEAKER_00
Thanks for listening to this conversation with Max Tegmark.
9
SPEAKER_00
To support this podcast, please check out our sponsors in the description.
9
SPEAKER_00
And now, let me leave you with some words from Frank Herbert.
9
SPEAKER_00
History is a constant race between invention and catastrophe.
9
SPEAKER_00
Thank you for listening.
10
SPEAKER_00
Hope to see you next time.