friendly AI

15 results back to index

pages: 294 words: 81,292




One path leading to global catastrophe—to someone pressing the button with a mistaken idea of what the button does—is that Artificial Intelligence comes about through a similar accretion of working algorithms, with the researchers having no deep understanding of how the combined system works. [italics mine] Not knowing how to build a Friendly AI is not deadly, of itself.… It’s the mistaken belief that an AI will be friendly which implies an obvious path to global catastrophe. Assuming that human-level AIs (AGIs) will be friendly is wrong for a lot of reasons. The assumption becomes even more dangerous after the AGI’s intelligence rockets past ours, and it becomes ASI—artificial superintelligence. So how do you create friendly AI? Or could you impose friendliness on advanced AIs after they’re already built? Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. Friendly AI is a subject so dense yet important it exasperates its chief proponent himself, who says about it, “it only takes one error for a chain of reasoning to end up in Outer Mongolia.”

And not only what we would want, but what we would want if we “knew more, thought faster, and were more the people we thought we were.” CEV would be an oracular feature of friendly AI. It would have to derive from us our values as if we were better versions of ourselves, and be democratic about it so that humankind is not tyrannized by the norms of a few. Does this sound a little starry-eyed? Well, there are good reasons for that. First, I’m giving you a highly summarized account of Friendly AI and CEV, concepts you can read volumes about online. And second, the whole topic of Friendly AI is incomplete and optimistic. It’s unclear whether or not Friendly AI can be expressed in a formal, mathematical sense, and so there may be no way to build it or to integrate it into promising AI architectures. But if we could, what would the future look like?

In fact, we thrive. God bless you, Friendly AI! * * * Now that most (but not all) AI makers and theorists have recognized Asimov’s Three Laws of Robotics for what they were meant to be—tools for drama, not survival—Friendly AI may be the best concept humans have come up with for planning their survival. But besides not being ready yet, it’s got other big problems. First, there are too many players in the AGI sweepstakes. Too many organizations in too many countries are working on AGI and AGI-related technologies for them all to agree to mothball their projects until Friendly AI is created, or to include in their code a formal friendliness module, if one could be made. And few are even taking part in the public dialogue about the necessity for Friendly AI. Some of the AGI contestants include: IBM (with several AGI-related projects), Numenta, AGIRI, Vicarious, Carnegie Mellon’s NELL and ACT-R, SNERG, LIDA, CYC, and Google.

pages: 271 words: 52,814

Blockchain: Blueprint for a New Economy by Melanie Swan


23andMe, Airbnb, altcoin, Amazon Web Services, asset allocation, banking crisis, basic income, bioinformatics, bitcoin, blockchain, capital controls, cellular automata, central bank independence, clean water, cloud computing, collaborative editing, Conway's Game of Life, crowdsourcing, cryptocurrency, disintermediation, Edward Snowden,, ethereum blockchain, fault tolerance, fiat currency, financial innovation, Firefox, friendly AI, Hernando de Soto, intangible asset, Internet Archive, Internet of things, Khan Academy, Kickstarter, lifelogging, litecoin, Lyft, M-Pesa, microbiome, Network effects, new economy, peer-to-peer, peer-to-peer lending, peer-to-peer model, personalized medicine, post scarcity, prediction markets, QR code, ride hailing / ride sharing, Satoshi Nakamoto, Search for Extraterrestrial Intelligence, SETI@home, sharing economy, Skype, smart cities, smart contracts, smart grid, software as a service, technological singularity, Turing complete, unbanked and underbanked, underbanked, web application, WikiLeaks

To achieve their goals, digital intelligences will want to conduct certain transactions over the network, many of which could be managed by blockchain and other consensus mechanisms. Only Friendly AIs Are Able to Get Their Transactions Executed One of the unforeseen benefits of consensus models might be that they could possibly enforce friendly AI, which is to say cooperative, moral players within a society.196 In decentralized trust networks, an agentÕs reputation (where agents themselves remain pseudonymous) could be an important factor in whether his transactions will be executed, such that malicious players would not be able to get their transactions executed or recognized on the network. Any important transaction regarding resource access and use might require assent by consensus models. Thus, the way that friendly AI could be enforced is that even bad agents want to participate in the system to access resources and to do so, they need to look like good agents.

Advanced Concepts Terminology and Concepts Currency, Token, Tokenizing Communitycoin: Hayek’s Private Currencies Vie for Attention Campuscoin Coin Drops as a Strategy for Public Adoption Currency: New Meanings Currency Multiplicity: Monetary and Nonmonetary Currencies Demurrage Currencies: Potentially Incitory and Redistributable Extensibility of Demurrage Concept and Features 6. Limitations Technical Challenges Business Model Challenges Scandals and Public Perception Government Regulation Privacy Challenges for Personal Records Overall: Decentralization Trends Likely to Persist 7. Conclusion The Blockchain Is an Information Technology Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI Large Possibility Space for Intelligence Only Friendly AIs Are Able to Get Their Transactions Executed Smart Contract Advocates on Behalf of Digital Intelligence Blockchain Consensus Increases the Information Resolution of the Universe A. Cryptocurrency Basics Public/Private-Key Cryptography 101 B. Ledra Capital Mega Master Blockchain List Endnotes and References Index Blockchain Blueprint for a New Economy Melanie Swan Blockchain by Melanie Swan Copyright © 2015 Melanie Swan.

The blockchain is a consensus model at scale, and possibly the mechanism we have been waiting for that could help to usher in an era of friendly machine intelligence. Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI One forward-looking but important concern in the general future of technology is different ways in which artificial intelligence (AI) might arise and how to sponsor it such that it engenders a “friendly” or benevolent relationship with humans. There is the notion of a technological singularity, a moment when machine intelligence might supersede human intelligence. However, those in the field have not set forth any sort of robust plan for how to effect friendly AI, and many remain skeptical of this possibility.195 It is possible that blockchain technology could be a useful connector of humans and machines in a world of increasingly autonomous machine activity through Dapps, DAOs, and DACs that might eventually give way to AI.

pages: 144 words: 43,356




It also boasts some Hollywood glamour, with Alan Alda and Morgan Freeman on the advisory board, along with technology entrepreneur Elon Musk, who has donated $10m of his personal money to the institute. 8.6 – Conclusion We do not yet have a foolproof way to ensure that the first AGI is a Friendly AI. In fact we don’t yet know how best to approach the problem. But we have only just begun, and the resources allocated to the problem are small: Nick Bostrom estimated in 2014 that only six people in the world are working full-time on the Friendly AI problem, whereas many thousands of people work full-time on projects that could well contribute to the creation of the first AGI. (50) He argued that this equation needed urgent re-balancing. A very experienced AI researcher told me in spring 2015 that Bostrom’s estimate was significantly too low, and that many more AI researchers spend much of their time thinking about Friendly AI as part of their everyday jobs. Even if that is correct, I suspect Bostrom is still right about the imbalance, and the truth will emerge if we have the kind of debate I argue for in the next, concluding chapter.

The upshot is that there are seven billion of us and we are shaping much of the planet according to our will (regardless of whether that is a good idea or not), whereas there are fewer than 300,000 chimpanzees, and whether they become extinct or not depends entirely on the actions of humans. A superintelligence could become not just twice as smart as humans, but smarter by many orders of magnitude. It is hard to escape the conclusion that our future will depend on its decisions and its actions. Would that be a good thing or a bad thing? In other words, would a superintelligence be a “Friendly AI”? (Friendly AI, or FAI, denotes an AGI that is beneficial for humans rather than one that seeks social approbation and company. It also refers to the project to make sure that AGI is beneficial.) 7.2 – Optimistic scenarios: to immortality and beyond The ultimate problem solver Imagine having a big sister endowed with superhuman wisdom, insight and ingenuity. Her cleverness enables her to solve all our personal, inter-personal, social, political and economic problems.

What is clear is that a negative outcome cannot be ruled out. So if we take seriously the idea that a superintelligence may appear on the Earth in the foreseeable future, we should certainly be thinking about how to ensure that the event is a positive one for ourselves and our descendants. We should be taking steps to ensure that the first AGI is a friendly AI. PART FOUR: FAI Friendly Artificial Intelligence CHAPTER 8 CAN WE ENSURE THAT SUPERINTELLIGENCE IS SAFE? As we saw in the last chapter, Friendly AI (FAI) is the project of ensuring that the world’s superintelligences are safe and useful for humans. The central argument of this book is that we need to address this challenge successfully. It may well turn out to be the most important challenge facing this generation and the next. Indeed it may turn out to be the most important challenge humanity ever faces. 8.1 – Stop before you start Faced with the unpalatable possibilities explored in the last chapter, perhaps we should try to avoid the problem by preventing the arrival of AGI in the first place.

pages: 377 words: 97,144




Even if Eliezer and the Singularity Institute have no realistic chance of creating a friendly AI, they still easily justify their institute’s existence. As Michael Anissimov, media director for the Institute, once told me in a personal conversation, at the very least the Institute has reduced the chance of humanity’s destruction by repeatedly telling artificial-intelligence programmers about the threat of unfriendly AI. The Singularity Institute works toward the only goal I consider worthy of charitable dollars: increasing the survival prospects of mankind. Anna Salamon of the Singularity Institute did a credible back-of-the-envelope calculation showing that, based on some reasonable estimates of the effectiveness of friendly AI research and the harm of an unfriendly Singularity, donating one dollar to research on friendly AI will on average save over one life because slightly decreasing the odds that the seven billion current inhabitants of Earth will die yields you a huge average expected benefit.105 This expected benefit goes way up if you factor in people who are not yet born.

Within a year, we will probably have the technical ability to activate a seed AI, but once the Chinese threat has been annihilated, our team will have no reason to hurry and could take a decade to fine-tune their seed AI. If we delay, any intelligence explosion we eventually create will have an extremely high probability of yielding a friendly AI. Some people on our team think that, given another decade, they will be able to mathematically prove that the seed AI will turn into a friendly ultra-AI. A friendly AI would allow trillions and trillions of people to eventually live their lives, and mankind and our descendants could survive to the end of the universe in utopia. In contrast, an unfriendly AI would destroy us. I have decided to make the survival of humanity my priority. Consequently, since a thermonuclear war would nontrivially increase the chance of human survival, I believe that it’s my moral duty to initiate war, even though my war will kill billions of people.

If the firm really did gain the option to achieve utopia, everyone, including the firm’s investors, would want the firm to exercise this option. The firm, therefore, wouldn’t be able to raise start-up capital from small, self-interested investors. So now pretend that at the time the firm tries to raise capital there is a well-developed theory of friendly AI, which provides programmers with a framework for creating AI that is extremely likely to be well disposed toward humanity and create a utopia if it undergoes an intelligence explosion. To raise funds from self-interested investors, an AI-building firm would need to pick a research and development path that would make it difficult for the firm ever to use the friendly AI framework. Unfortunately, this means that any intelligence explosion the firm unintentionally brings about would be less likely to be utopian than if the firm had used the friendly framework. MULTIPLE AI-BUILDERS Multiple AI-building firms would increase the odds of a bad Singularity.

pages: 1,737 words: 491,616

Rationality: From AI to Zombies by Eliezer Yudkowsky


It’s such a compelling argument, you see. But compulsion is not a property of arguments; it is a property of minds that process arguments. So the reason I’m arguing against the ghost isn’t just to make the point that (1) Friendly AI has to be explicitly programmed and (2) the laws of physics do not forbid Friendly AI. (Though of course I take a certain interest in establishing this.) I also wish to establish the notion of a mind as a causal, lawful, physical system in which there is no irreducible central ghost that looks over the neurons/code and decides whether they are good suggestions. (There is a concept in Friendly AI of deliberately programming an FAI to review its own source code and possibly hand it back to the programmers. But the mind that reviews is not irreducible, it is just the mind that you created.

He’s just making Eliezer1997’s strategy even better by including a contingency plan for “the unlikely event that life turns out to be meaningless” . . . . . . which means that Eliezer2001 now has a line of retreat away from his mistake. I don’t just mean that Eliezer2001 can say “Friendly AI is a contingency plan,” rather than screaming “OOPS!” I mean that Eliezer2001 now actually has a contingency plan. If Eliezer2001 starts to doubt his 1997 metaethics, the intelligence explosion has a fallback strategy, namely Friendly AI. Eliezer2001 can question his metaethics without it signaling the end of the world. And his gradient has been smoothed; he can admit a 10% chance of having previously been wrong, then a 20% chance. He doesn’t have to cough out his whole mistake in one huge lump. If you think this sounds like Eliezer2001 is too slow, I quite agree. Eliezer1996–2000’s strategies had been formed in the total absence of “Friendly AI” as a consideration. The whole idea was to get a superintelligence, any superintelligence, as fast as possible—codelet soup, ad-hoc heuristics, evolutionary programming, open-source, anything that looked like it might work—preferably all approaches simultaneously in a Manhattan Project.

Good’s older term, “intelligence explosion,” to help distinguish his views from other futurist predictions, such as Ray Kurzweil’s exponential technological progress thesis.2 Technologies like smarter-than-human AI seem likely to result in large societal upheavals, for the better or for the worse. Yudkowsky coined the term “Friendly AI theory” to refer to research into techniques for aligning an AGI’s preferences with the preferences of humans. At this point, very little is known about when generally intelligent software might be invented, or what safety approaches would work well in such cases. Present-day autonomous AI can already be quite challenging to verify and validate with much confidence, and many current techniques are not likely to generalize to more intelligent and adaptive systems. “Friendly AI” is therefore closer to a menagerie of basic mathematical and philosophical questions than to a well-specified set of programming objectives. As of 2015, Yudkowsky’s views on the future of AI continue to be debated by technology forecasters and AI researchers in industry and academia, who have yet to converge on a consensus position.

pages: 48 words: 12,437

Smarter Than Us: The Rise of Machine Intelligence by Stuart Armstrong



So all is doomed and we’re heading to hell in a digitally engineered handbasket? Well, not entirely. Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford. FHI is dedicated to analyzing and reducing all existential risks—risks that could drive humanity to extinction or dramatically curtail its potential, of which AI risk is just one example.

But why would an alien mind such as the AI react in comparable ways? Are we not simply training the AI to give the correct answer in training situations? The whole approach is a constraint problem: in the space of possible AI minds, we are going to give priority to those minds that pass successfully through this training process and reassure us that they’re safe. Is there some quantifiable way of measuring how likely this is to produce a human-friendly AI at the end of it? If there isn’t, why are we putting any trust in it? These problems remain barely addressed, so though it is possible to imagine a safe AI being developed using the current approaches (or their descendants), it feels extremely unlikely. Hence we shouldn’t put our trust in the current crop of experts to solve the problem. More work is urgently, perhaps desperately, needed. * * * 1.

pages: 523 words: 148,929




To prevent a robot from enslaving us in order to save us, some have advocated that we must add the zeroth law of robotics: Robots cannot harm or enslave the human race.) But many scientists are leaning toward something called “friendly AI,” where we design our robots to be benign from the very beginning. Since we are the creators of these robots, we will design them, from the very start, to perform only useful and benevolent tasks. The term “friendly AI” was coined by Eliezer Yudkowsky, a founder of the Singularity Institute for Artificial Intelligence. Friendly AI is a bit different from Asimov’s laws, which are forced upon robots, perhaps against their will. (Asimov’s laws, imposed from the outside, could actually invite the robots to devise clever ways to circumvent them.) In friendly AI, by contrast, robots are free to murder and commit mayhem. There are no rules that enforce an artificial morality.

In the future, however, more and more funding for robots will come from the civilian commercial sector, especially from Japan, where robots are designed to help rather than destroy. If this trend continues, then perhaps friendly AI could become a reality. In this scenario, it is the consumer sector and market forces that will eventually dominate robotics, so that there will be a vast commercial interest in investing in friendly AI. MERGING WITH ROBOTS In addition to friendly AI, there is also another option: merging with our creations. Instead of simply waiting for robots to surpass us in intelligence and power, we should try to enhance ourselves, becoming superhuman in the process. Most likely, I believe, the future will proceed with a combination of these two goals, i.e., building friendly AI and also enhancing ourselves. This is an option being explored by Rodney Brooks, former director of the famed MIT Artificial Intelligence Laboratory.

Douglas Hofstadter has said, “It’s as if you took a lot of good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.” No one knows how this will play out. But I think the most likely scenario is the following. MOST LIKELY SCENARIO: FRIENDLY AI First, scientists will probably take simple measures to ensure that robots are not dangerous. At the very least, scientists can put a chip in robot brains to automatically shut them off if they have murderous thoughts. In this approach, all intelligent robots will be equipped with a fail-safe mechanism that can be switched on by a human at any time, especially when a robot exhibits errant behavior.

The Singularity Is Near: When Humans Transcend Biology by Ray Kurzweil



Also see note 30 above. Yudkowsky formed the Singularity Institute for Artificial Intelligence (SIAI) to develop "Friendly AI," intended to "create cognitive content, design features, and cognitive architectures that result in benevolence" before near-human or better-than-human Als become possible. SIAI has developed The SIAI Guidelines on Friendly AI: "Friendly AI," Ben Goertzel and his Artificial General Intelligence Research Institute have also examined issues related to developing friendly AI; his current focus is on developing the Novamente AI Engine, a set of learning algorithms and architectures. Peter Voss, founder of Adaptive A.I., Inc., has also collaborated on friendly-AI issues: 46. Integrated Fuel Cell Technologies, Disclosure: The author is an early investor in and adviser to IFCT. 47.

Vaupel, "Broken Limits to Life Expectancy," Science 296.5570 (May 10,2002): 1029–31. 29. Steve Bowman and Helit Barel, Weapons of Mass Destruction: The Terrorist Threat, Congressional Research Service Report for Congress, December 8, 1999, 30. Eliezer S. Yudkowsky, "Creating Friendly AI 1.0, The Analysis and Design of Benevolent Goal Architectures" (2001), The Singularity Institute,; Eliezer S. Yudkowsky, "What Is Friendly AI?" May 3, 2001, 31. Ted Kaczynski, "The Unabomber's Manifesto," May 14, 2001, http://www.KurzweilAI.netlmeme/frame.html?main=/articles/art0182.html. 32. Bill McKibben, Enough: Staying Human in an Engineered Age (New York: Times Books, 2003). 33.

But the dangers it presents are also profound precisely because of its amplification of intelligence. Intelligence is inherently impossible to control, so the various strategies that have been devised to control nanotechnology (for example, the "broadcast architecture" described below) won't work for strong AI. There have been discussions and proposals to guide AI development toward what Eliezer Yudkowsky calls "friendly AI"30 (see the section "Protection from 'Unfriendly' Strong AI," p. 420). These are useful for discussion, but it is infeasible today to devise strategies that will absolutely ensure that future AI embodies human ethics and values. Returning to the Past? In his essay and presentations Bill Joy eloquently describes the plagues of centuries past and how new self-replicating technologies, such as mutant bioengineered pathogens and nanobots run amok, may bring back long-forgotten pestilence.

pages: 428 words: 121,717

Warnings by Richard A. Clarke

active measures, Albert Einstein, algorithmic trading, anti-communist, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, Bernie Madoff, cognitive bias, collateralized debt obligation, complexity theory, corporate governance, cuban missile crisis, data acquisition, discovery of penicillin, double helix, Elon Musk, failed state, financial thriller, fixed income, Flash crash, forensic accounting, friendly AI, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Watt: steam engine, Jeff Bezos, John Maynard Keynes: Economic Possibilities for our Grandchildren, knowledge worker, Maui Hawaii, megacity, Mikhail Gorbachev, money market fund, mouse model, Nate Silver, new economy, Nicholas Carr, nuclear winter, pattern recognition, personalized medicine, phenotype, Ponzi scheme, Ray Kurzweil, Richard Feynman, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Ronald Reagan, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, smart grid, statistical model, Stephen Hawking, Stuxnet, technological singularity, The Future of Employment, the scientific method, The Signal and the Noise by Nate Silver, Tunguska event, uranium enrichment, Vernor Vinge, Watson beat the top human players on Jeopardy!, women in the workforce, Y2K

His work focuses on foundational mathematical research to ensure (he hopes) that artificial intelligence ultimately has only a positive impact on humanity. The ultimate problem: how to keep humanity from losing control of a machine of its own creation, to prevent artificial intelligence from becoming, in the words of James Barrat in the title of his 2013 book, Our Final Invention.6 A divisive figure, Yudkowsky is well known in academic circles and the Silicon Valley scene as the coiner of the term “friendly AI.” His thesis is simple, though his solution is not: if we are to have any hope against superintelligence, we need to code it properly from the beginning. The answer, Eliezer believes, is one of morality. AI must be programmed with a set of ethical codes that align with humanity’s. Though it is his life’s only work, Yudkowsky is pretty sure he will fail. Humanity, he tells us, is likely doomed.

Yudkowsky believes superintelligence must be designed from the start with something approximating ethics. He envisions this as a system of checks and balances so that advanced AI growth is auditable and controllable, so that even as it continues to learn, advance, and reprogram itself, it will not evolve out of its own benign coding. Such preprogrammed measures will ensure that superintelligence will “behave as we intend even in the absence of immediate human supervision.”12 Eliezer calls this “friendly AI.” According to Yudkowsky, once AI gains the ability to broadly reprogram itself, it will be far too late to implement safeguards, so society needs to prepare now for the intelligence explosion. Yet this preparation is complicated by the sporadic and unpredictable nature of scientific advancement and the numerous secret efforts to create superintelligence around the world. No supranational organization can track all of the efforts, much less predict when or which one of them will succeed.

Eliezer told us that humanity’s best hope is to perhaps create one highly funded, highly secure, multilateral effort to develop a friendly superintelligence with himself (or perhaps another futurist he approves of) at the helm. The work of this massive global Manhattan Project would be explicitly “for the benefit of humanity internationally.” It simultaneously would ban, starve, or simply outpace other, less-well-thought-out efforts to develop superintelligence. Once created, this friendly AI would be unleashed to attack and destroy any competing efforts, ensuring that the only superintelligence in existence would help, not destroy, humankind. Yudkowsky rejects the idea that a superintelligence should, or could, be tailored to parochial national security interests, believing instead that any solution must be considered at the human species level. “This stuff does not stop being lethal because it’s in American hands, or Australian hands, or even in Finland’s hands,” he told us, mildly annoyed.

pages: 574 words: 164,509

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom


agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, anti-communist, artificial general intelligence, autonomous vehicles, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, demographic transition, Donald Knuth, Douglas Hofstadter, Drosophila, Elon Musk,, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, Gödel, Escher, Bach, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, Menlo Park, meta analysis, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Norbert Wiener, NP-complete, nuclear winter, optical character recognition, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, transaction costs, Turing machine, Vernor Vinge, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

The programmers may try to guard against this possibility by secretly monitoring the AI’s source code and the internal workings of its mind; but a smart-enough AI would realize that it might be under surveillance and adjust its thinking accordingly.2 The AI might find subtle ways of concealing its true capabilities and its incriminating intent.3 (Devising clever escape plans might, incidentally, also be a convergent strategy for many types of friendly AI, especially as they mature and gain confidence in their own judgments and capabilities. A system motivated to promote our interests might be making a mistake if it allowed us to shut it down or to construct another, potentially unfriendly AI.) We can thus perceive a general failure mode, wherein the good behavioral track record of a system in its juvenile stages fails utterly to predict its behavior at a more mature stage.

Nonzero: The Logic of Human Destiny. New York: Vintage. Yaeger, Larry. 1994. “Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context.” In Proceedings of the Artificial Life III Conference, edited by C. G. Langton, 263–98. Santa Fe Institute Studies in the Sciences of Complexity. Reading, MA: Addison-Wesley. Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Machine Intelligence Research Institute, San Francisco, CA, June 15. Yudkowsky, Eliezer. 2002. “The AI-Box Experiment.” Retrieved January 15, 2012. Available at Yudkowsky, Eliezer. 2004. Coherent Extrapolated Volition. Machine Intelligence Research Institute, San Francisco, CA, May. Yudkowsky, Eliezer. 2007.

pages: 303 words: 67,891




Also, it is possible to directly edit the memory of a system to modify or implant certain knowledge. In theory, all of these shortcuts should be equivalent to certain possible experience of the system, so they do not conflict with the principle that all the knowledge of NARS comes, directly or indirectly, from experience. One important issue to be handled through education is ethics. Unlike argued by some other researchers, NARS is not an attempt to design a “friendly AI”. As far as its initial state is concerned, the system is ethically neutral, since it can has any beliefs and goals. To make a NARS implementation “human friendly” means to give it certain beliefs and goals, which is an education mission, not a design mission. Even if something like Asimov’s “Three Laws of Robotics” is implanted into the system’s memory (which is possible), it still cannot fully control the system’s behaviors, due to the insufficiency of knowledge and resources in the system.

[Audience]: Ben, you seem more optimistic. Could you talk about your perspective? [Ben Goertzel]: Well I have a quite different opinion than that of Steve Grand in that I don’t think an amazing conceptual breakthrough on the level of the discovery of the quantum or curved 4D space-time, or something like that, is needed to create general intelligence. It might be needed to create provably stable friendly AI, like Eliezer Yudkowsky would like. I tend to think of the brain as a complex system composed of a bunch of evolved kluges for solving particular problems, which have been hacked together and adapted by evolution. I think if you assemble subcomponents solving the appropriate set of specialized problems, as well as a fairly weak general problem solver, and they are hooked together in a knowledge representation that works for all the components, with learning mechanisms that let each component learn from each other -- then you are going to have an intelligent mind that can be taught.

pages: 481 words: 125,946




The system can be used between human parties or interspecies parties, exactly because it’s not necessary to know, trust, or understand the other entity, just the code (the language of machines). Over time, trust can grow through reputation. Blockchain technology could be used to enforce friendly AI and mutually beneficial interspecies interaction. Someday, important transactions (like identity authentication and resource transfer) will be conducted on smart networks that require confirmation by independent consensus mechanisms, such that only bona fide transactions by reputable entities are executed. While perhaps not a full answer to the problem of enforcing friendly AI, decentralized smart networks like blockchains are a system of checks and balances helping to provide a more robust solution to situations of future uncertainty. Trust-building models for interspecies digital intelligence interaction could include both game-theoretic checks-and-balances systems like blockchains and also, at the higher level, frameworks that put entities on the same plane of shared objectives.

pages: 222 words: 53,317

Overcomplicated: Technology at the Limits of Comprehension by Samuel Arbesman



naches is also a framework: The roboticist Hans Moravec has referred to our more powerful descendants as “mind children,” and a similar approach characterizes a short story by the science fiction writer Ted Chiang, in which technologically enhanced humans have long surpassed “regular” humans in their ability to make scientific discoveries. In the end, little to nothing is understood by (nonenhanced) humanity. But that’s okay, because “We need not be intimidated by the accomplishments of metahuman science. We should always remember that the technologies that made metahumans possible were originally developed by humans, and they were no smarter than we.” See Luke Muehlhauser and Nick Bostrom, “Why We Need Friendly AI,” Think 36, no. 13 (Spring 2014), 41–47; and Ted Chiang, Stories of Your Life and Others (New York: Tor Books, 2003), 203. understand the most complex parts of the world: In many cases, we might even want to have a technology too complex to understand, because it means that it is sophisticated and powerful. a grab bag of intriguing ideas: The World of Wonders: A Record of Things Wonderful in Nature, Science, and Art (London: Cassell, Petter, and Galpin, exact year of publication unknown),

Pandora's Brain by Calum Chace


3D printing, AI winter, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, brain emulation, Extropian, friendly AI, hive mind, mega-rich, Ray Kurzweil, self-driving car, Silicon Valley, Singularitarianism, Skype, speech recognition, stealth mode startup, Stephen Hawking, strong AI, technological singularity, theory of mind, Turing test, Wall-E

The European Commission was proposing an autonomous international agency with exceptional powers of inspection and restraint in order to stop rogue governments and individuals from doing any research that could lead to the creation of an AGI. This was a step too far for him, as it was for most Americans. ‘That game is far from over. And I wouldn’t be surprised if a coalition of the institutes which are researching human-friendly AI algorithms announced a breakthrough this spring. There’s been a lot of unusually cordial traffic between several of the bigger US-based ones during the winter. Anyway, fortunately, none of that affects what we’re doing here at the Foundation. We’ve managed to put clear blue water between brain preservation research and AI research in the public’s mind.’ ‘True. It’s funny, though,’ David mused.

pages: 798 words: 240,182

The Transhumanist Reader by Max More, Natasha Vita-More



One key point, though, is that the risks and rewards of AGI must be considered in the broader context of all the other technologies currently under development, and all the social, psychological, and technological change likely to come in the next decades and centuries. It’s not as though our choices are “life goes on exactly as is” versus “life as it is plus super-powerful AI.” Various technologies are advancing rapidly and society is changing accordingly, and the rate of advancement of AGI is just one aspect in the mix. I don’t think there are any magic bullets to resolve the dilemmas of AGI ethics. There will almost surely be no provably Friendly AI, in spite of the wishes of Eliezer Yudkowsky (2008) and some others. Nor, in my best guess, will there be an Artilect War in which pro-AGI and anti-AGI forces battle to the death with doomsday machines, as Hugo de Garis (2005) foresees. But I don’t pretend to be able to see exactly what the outcome will be. The important thing, as I see it, is that the human race as a whole engages as closely and intelligently as possible with AGI as it evolves – so that, as AGI comes about, it’s not a matter of “us versus them”, but rather a matter of AGIs and humans, that have become inseparable on various levels, moving forward together into new realms of science, technology, interaction, and experience.