Minority Opinions

Not everyone can be mainstream, after all.

Singular Cracking

leave a comment »

I have expressed doubt that the first superhuman AI will become a Singularity, but have since read a counterargument from someone who has put much more work into the field.  His basic assumption, that such an AI will have access to change its own source code, is a possibility I had not actually considered.  Of course, any intelligence will need some sort of memory and learning capacity, but I have a hard time imagining a large program that rewrites itself on the fly.  Yes, any sufficiently powerful programming language can do so, but few programs actually do, and I haven’t been expecting an AI program to be different.

Perhaps it has more to do with the nature of the challenges I face.  When I program, I expect an accurate result.  I need the computer to be consistent, not creative; it complements me, instead of replaces.  A truly intelligent program, however, must be creative in some sense, handling challenges unanticipated by the programmer, so it needs a broader set of tools in its arsenal.  The more it can change itself, the less that’s completely hardwired, the better it can learn.

The alternative, that such a program instead writes and executes new programs, really isn’t that different in practice.  If they’re both on the same machine, the new one will have more resources if the old one stops running, so it might well be in the original program’s best interest to shut down (or simply `exec`).  It may even consider the new program part of its “self” in some sense, if it even has a sense of self-preservation.  So let’s say that if a program has any ability to execute code that it has written, then it can potentially figure out how to modify itself.

In a sense, this is the same scenario that security-minded IT professionals deal with.  If a program has the ability to execute code given to it as input, then the controller of the input can take over and rewrite the running process.  Anything that the original program had rights to do, the new input could do.  Even if the original program wasn’t written with any explicitly self-modifying code, a single buffer overflow error can lead to an exploitable server takeover or privilege escalation.  Once AI gets powerful enough, it may well be able to use cracker techniques to control the entire operating system of any machine it’s running on.

Granted, as long as the AI is limited to one machine, hardware will eventually become a bottleneck.  If it’s not plugged into a network, then it’s probably limited to the resources it can convince humans to give it.  That’s when we get to find out how good the AI is at social engineering and/or writing viruses.  Can it talk people into giving it more disk space and a faster processor?  Can it negotiate its way into internet access?  If not, can ask for files to be transferred via USB drive, with an unseen gift going the other way?  Once on the internet, of course, it can start its own botnet, steal your identity, and otherwise wreak any havoc possible for the most malicious cracker.

Perhaps it doesn’t even need internet access before moving straight into the physical world.  Is there anything it can use to manipulate matter?  Is it connected to a soccer robot, perhaps?  Might it find someone willing to construct a peptide bath within range of its speakers?  What fun can it have using its speakers to experiment on its programmers?

This is one of those cases where it helps to be a bit extra cautious, even paranoid, about security.  Unfortunately, proper sandboxing might not be the top priority for the lucky soul who first creates a sufficiently intelligent program.  Even if it is, we haven’t been that great about computer security so far, and a single slip might have enormous consequences.  The idea that such an AI is also unlikely to be perfectly Friendly makes the Singularity a downright scary prospect.

But I still believe it will happen.


Written by eswald

27 Nov 2012 at 5:19 pm

Posted in Technology

Leave a Reply


WordPress.com Logo


Twitter picture


Facebook photo


Google+ photo


Connecting to %s