Except as noted for The Appendix An
Article by Bill Joy , this material is
Copyright © 2004 by Jay B. Gaskill
For permission to copy, publish,
distribute or print, contact:
Jay B. Gaskill, attorney at law,
via e mail: response@jaygaskill.com
The Bumper Sticker Mind
AVERTING THE NEXT DARK AGE
by
Jay B. Gaskill
Part One
Reflections From The Rear
Recently, while stuck in traffic, I paid close attention to all the bumper stickers on the car ahead of me. Slogans were extravagantly plastered all over the back of the car. Have you noticed that there are either just one or two bumper stickers on a car, or the shouting political wallpaper effect? I think political bumper stickers multiply like fruit flies.
On the same car I noted a sticker
promoting one of those strongly slanted media sources. In this instance, it was
a
When I noticed this perfect correlation between information source and slogans, between input and output, I realized I’d stumbled into a New Truth About The Human Condition.
That so called “
Our emerging circumstance is eerily reminiscent of medieval times, when church and state were the sole “media sources” and the common value arbiters for everyone. Of course in the Dark Ages, alien ideas and contrary opinion didn’t merely fail to “gain traction” (to use the modern description), they were virtually nonexistent. And we moderns are more diverse in our small mindedness. But those differences are trivial. Our culture is backsliding, heading into a “Dark Age Lite”, a high tech variation on the medieval model where people sort themselves into self-delusional cohorts.
How could this happen in the information age? The modern (and post-modern) media assaulted mind is numbed by a chaotic torrent of data, unevaluated raw information, and wildly competing claims. The minds most affected by this info-blizzard have begun to contract like snails in a bed of salt. Whole groups of people (of whom the bumper sticker crowd is just the most visible) have begun to self encapsulate, retreating into an information cocoon state, in effect withdrawing into virtual reality constructs, faux worlds that form intellectual and emotional refuges. Their self isolation is made possible by information filters that support a portable, closed system: Self-censored information input, local, group-enforced values, and common issue-position sets make a complete info-value loop. The unifying ideology need not be logical or coherent (and rarely is), just reducible to slogans.
For the first time in history, modern information technology has allowed large numbers of people to coexist and function in a socially interactive setting, exchanging goods and services and traveling great distances, while inhabiting sharply different info-realities. Increasingly we are acting like members of separate alien tour groups (each group sharing common info-cocoons) traveling about in the same territory.
Within the bumper sticker mind, the range of acceptable belief and behavior is actually very narrow. This is true whether the ethos is nominally tolerant or intolerant, since the ideology of tolerance tends to condemn all contrary value stances as intolerant. The people within the red and blue info-cocoon zones (a simplified model - there are other colors) can travel side by side, eat at the same restaurants, shop at the same stores and stay at the same hotels and live in the same neighborhoods. They can even witness or read about the same large scale events of history. But they are not seeing and understanding the same reality. The red and blue states are not geographical accidents; they are the aggregate outcomes of the tendency of info-cocoon cohorts to want to live together.
In extreme cases, the bumper sticker mind can cling to such radically unrealistic models of reality that to use the word delusional is not inappropriate. Unlike the fully deranged, most of the info-cocoon afflicted continue to function very well in the day-to-day survival sense.
Why are they so rarely shaken out of their cocoon states? Why are they locked in such intellectual isolation? How do they continue to function in day to day activities? How is it that they can cooperate on many levels with those who live outside the cocoon yet not able to be in authentic conversation with the same “outsiders” about whole areas of life? Is this a new development in the human condition or a reversion to a more primitive state?
Part of the answer to those questions is that human nature includes the capacity to erect internal emotional and psychological barriers, ranging from simple denial to profound dissociation. Part of the answer is that we all have the capacity under some conditions to revert to more primitive states. But the most general answer to these questions is that we are witnessing a form of technologically enhanced atavism. And we witnessing an innovation: The possible replacement of the nation state with encapsulated mental states.
The tendency of like minded info-cocoon people to gather and dominate an area reduces the chance that corrective mechanisms will emerge in those areas. In the extreme cases where strongly encapsulated self validating input is the norm, a fantasy based, highly distorted output is the consequence. Population shifts causing the in-migration of competing mindsets has replaced dialogue as a correction mechanism with increased social tension and slogan against counter slogan shouting.
In general we are missing the correction mechanisms that come into play when nearly everyone is linked by a common meta-normative structure that frames all the remaining differences among us, and allows for non-destructive, reality correcting discourse. The problem is, at root, the predictable consequence of the weakened normative infrastructure of civilization itself. This is the ugly legacy of cultural relativism, moral relativism, and the post-modern deconstruction of unifying values and the weakening authority of essential moral precepts. The normative incoherence among the intelligentsia and within the larger culture has led directly to the present growing social incoherence.
This is a large scale social pathology in the making. To reverse the trend, we must recover our moorings: the healthy intersection of religious insight and wisdom, the intersection of the deep classical tradition and the uniquely American version of the Enlightenment that animated our founders. Then we will also rediscover the virtues of civil discourse.
We need not descend into a new Dark Age. But until we are able to reanimate the authentic dialogue that takes place between people whose minds are open to new ideas and information because they are secure in the shared deeper verities, the trend is an ominous one.
Part
Two
Repairing The Media
The situation is not helped by the current state of the information media in this country. By now it should be patently evident to all discerning observers that the dominant, so called “mainstream” media, is shot though with an ideological bias that strongly filters the information (and the evaluation of information) fed to most Americans on broadcast television, newspapers and mass market magazines. Much has been said about the corrective effects of the new media, principally talk radio, the blogo-sphere, and the increased intellectual respectability of some special market print media.
But all this misses the underlying problem: We live in an info-swamp so extreme that filters of some kind are absolutely essential because our time is finite and the sheer volume of data seeking our attention is effectively infinite. Censorship in the old fashioned sense is no longer needed; mere data placement is enough to hide the truth from most people.
Consider the classic pyramid information array taught in classic journalism classes where the progression is from headline, to lead paragraph, and trailing paragraphs such that, as the story unfolds, the last lines and paragraphs can be safely cut without altering the essence of the story. In effect, the order is: (1) soundbite, (2) quick summary, (3) follow up information, and (3) trailing details of descending importance. In the hands of an ideologue, this technique is easily co-opted to bury non-conforming information and insights at the end of the story. In many cases the whole story is buried deep within the newspaper where few will read it at all. In extreme cases the story isn’t covered at all. Because most people still have time only for traditional media, the leakage of counter information is largely confined to radio broadcasts heard while driving or at work, or via “buzz” (i.e., gossip). People remain hungry for validated information sources and are for the most part ill equipped to surf the info-swamp without a guide.
The post modern ethos teaches that no objective journalism is possible. This is one of those malevolent quarter truths that validates the current trend toward self-encapsulation and the deterioration of the capacity for authentic dialogue about facts and their significance to the common human condition.
After all, journalism in the largest sense is about facts and relevance. The quest for more objective and balanced journalism is not only worthy, but necessary. It is grounded in two simple assessments:
(1) that there are events and facts in the world around us that are reasonably ascertainable and that can be described and communicated in a reasonably objective way;
(2) that there are deeper values common to the human condition and essential to the preservation of civilization.
Thus the essential quest for journalistic objectivity and balance invites us to return to fundamentals:
(A) The intelligent, conscientious separation of event reporting (while including necessary value and historical context) from agenda-driven opinion;
(B) The reintegration, as context, of the common meta-value structure of civilization (I see no contradiction here) into the event story;
(C) A standard of relevance that integrates common sense evidentiary standards, common values, and the large, emergent human issues that guide historians in connecting the dots between apparently disparate events. [More in Part Three.]
There are healthy, robust journalistic models. Here are my three favorites:
The sports page model.
Compare any reasonably competent sports page with PC infected reportage. It’s understood and accepted that the reporter will occasionally take sides, even blatantly, while recognizing and relating all the essential objective facts of the contest.
As a rule, sports reportage is grounded in three realities:
- There is a score.
- There is a standard of play.
- The audience/readership wants the truth in relevant and entertaining detail.
Sports stories don’t bury the other side’s good plays, and don’t fail to criticize an umpire’s bad call or a player’s screwup, all PC considerations notwithstanding.
The classic 1940’s and 50’s crime reporting model.
This is the origin of the detective genre and all the classic film noir crime stories. There were no concessions to PC, just the hard hitting, hard boiled facts thank you, without any concessions to moral ambiguity, even when the “bad guys” were glamorized. That old style crime reportage didn’t avoid the obvious moral judgments because the meta-value system on which it was based was robust: crime was a bad thing and warranted punishment. The stories weren’t sprinkled with the fairy dust words of pseudo-objectivity, revealed in the flagrant overuse of “alleged” and “suspected” and the avoidance of insensitive terms like “thug” and “predator.”
In a local newspaper Op Ed, I recently complained about PC fog. My piece was submitted in this form:
PC Fog
A
NEWS ACCOUNT:
“Three teenage boys stabbed and tried to
kidnap a man during a carjacking late Friday….Despite the stab wounds, (the
victim) was able to get out of his car and back to his three children… The
three suspects are described as two males, both approximately 14 years old, and
an older boy, possibly 16 years old…”
Contemporary
journalism has been infected with a special form of censorship driven by the
fear of identification with a politically incorrect position.
You’d
think that most street crime is committed by faceless males, whose identifying
marks and racial characteristics have been wiped away by some digital pixel
smear effect, like those television shows where the headshot of the crime
victim or informant is altered to obscure identity.
This is PC
Fog.
Illustration:
An ATM shooting of an elderly lady is witnessed by two bystanders. The robber
flees on foot. Police and EMT’s arrive minutes later.
The officers get detailed descriptions of the predator. A composite description
is broadcast over the police radio.
Police are
trained to compile descriptions as complete as can be gleaned, including gender,
approximate age, height, weight, build, head and facial hair, clothing, race,
distinguishing marks, direction and mode of flight-- everything that might help
officers in the field to stop and detain a suspect.
Over the
next crucial minutes, various officers are tasked to briefly detain any
suspects who seem to match the broadcast description, and to hold them for
further identification procedures.
Assume in
our example, the robber gets away.
What will
we probably learn from a PC media version of this incident? We’ll likely be treated to an incomplete
description, something like --”husky male, dressed in dark clothes.” The suspect’s race? Known but omitted. We’ve all seen the
pattern.
This is PC
journalism at work.
Assume
that, based on the description, one suspect is detained in the field, is
subdued after a struggle. But before the
suspect is hand cuffed, he manages to escape, leaving a pistol on the pavement.
When PC
reigns, the police will be criticized for “racial profiling” because the
suspect was stopped because he was a “minority.” Later, when ballistics match
bullet and discarded gun, the same officers will be accused of negligence for
letting the shooter get away.
PC Fog is
released to obscure uncomfortable truths.
We are expected to live in the fantasy world where things like physical
appearance are irrelevant, where minorities are to be protected by racial
anonymity if necessary, even if that hampers law enforcement and our need as citizens
to know what’s going on in our community.
What is
sometimes reflexively condemned as “profiling” represents a perfectly
reasonable conservation of scarce law enforcement resources.
Suppose
the victim were a frail 88 year old back lady at an ATM, while her mugger was a
head-shaved white male of 25? Or that the mugger was initially detained because
his presence in the parking lot of an all black church after hours seemed
suspicious? Would it matter if the victim were an elderly Chinese woman? Or that the mugger, a
bearded Hispanic male wearing sweats, was detained because his presence
in an all Chinese wedding party seemed suspicious?
Would we
have the police detain an artificially larger sample, including some young
black males and white females, just to avoid “profiling”? Does it make any sense in these examples to
alert the public to a mugger while omitting skin color and apparent ethnicity
from the description?
PC fog is
as dangerous as highway fog without headlights.
The piece ran without the news account, which was taken directly from the same paper, and my use of the word “predator” was editorially redacted. To the credit of this local paper, I noticed about two weeks later that actual, unredacted descriptions of unapprehended robbers had begun to creep into the crime stories.
PC fog has begun to lift in some places.
The
tabloid model.
One of contemporary journalism’s dirty little secrets is that some of the most effective reporting done in the last decade has run in the National Enquirer. With all their faults, their relentless pursuit of the lurid and prurient and their occasional three headed alien parodies, the tabloids embody two journalistic virtues that have been lost to most of the mainstream press:
(1) The tough minded, relentless investigation, as opposed to the spoon-fed pap that masquerades as authentic reporting in most print news sources. Most of what currently passes for investigative reporting is little more than a sexed up police report with a thinly disguised reportorial overlay, sometimes supplemented with a phone call for a “quote.”
(2) The absolutely fearless, take-no-prisoners exposé.
From all the available evidence, I see no realistic hope for reform in this country’s journalism schools especially in the near term. The remedy is serious competition, hence:
A
modest proposal for the print media.
- Fire most of your staff.
- Hire politically diverse replacements.
- Employ new criteria for the fresh blood:
· Ignore journalism degrees.
· Require instead a demonstrated proficiency in written English, supplemented by a 90 day crash course in fact gathering techniques. [This can be self-administered or farmed out to private investigators.]
· Recruit candidates with an underlying substantive degree in philosophy, science, history, economics or political science.
·
Require an internship doing sports writing,
crime reporting, and sordid scandal investigations for a tabloid.
Part Three
Recovery: Restoring The Value Consensus
The value consensus that animated journalism in its better days simply mirrored the value consensus that sustained the robust American brand of Western civilization.
On some level, that American consensus survives among a majority of the so-called “common” people, but it has been corrupted in the same sense that software can be corrupted by a virus.
The Original American Consensus
The original, uniquely American value consensus is still alive, though not well. Here are its five major elements:
- The notion that mainline criminal behavior, the kind that fills the nation’s jails and prisons (behavior like thievery, thuggery, exploitation of children, lying, cheating, raping and pillaging), warrants social condemnation and punishment.
- The recognition that honesty and veracity are the standard coinage of public information exchange, both in journalism and in (dare I say it?) politics.
- Scandals are wickedly entertaining because they are bad. Authentic scandals are to be relentlessly exposed regardless of whose ox is gored and whose sensibilities are offended.
- All possible points of view and opinions are not equal. In fact, some are truly delusional, malevolent misinformation, or simply pathological nonsense, as for example, the outrageous notion that 9-11 was a plot by the government, the “Jews” or both. Responsible journalism may report such things, when relevant to events, but has an obligation to identify and condemn lunacy and malevolent misinformation when it does so.
- American Exceptionalism. The affirmation of the core values on which this country was founded, including a respect for life, liberty and the pursuit of happiness and the supporting traditions: a rejection of social class barriers and royalism in all their forms, and the respect for the self-help, hard work achievement ethos. This value set is not only recognized as valid for Americans, but as simply valid for the human condition. These are the core, animating universals that still make the American experiment the fulcrum of an ongoing world revolution.
The Virus Attacks
Cultural relativism and its virulent cousin, moral relativism, have worked like a computer virus to disrupt and dismantle the American value consensus by promoting “value neutrality” in all things. This is why so many contemporary journalists fall into the trap of conflating “balance” with objectivity. To the mind infected by relativism, a commitment to “balance” means that all points of view are equal (leading to the uncritical inclusion of all possible voices, including the lunatics).
The post modern notion that objectivity is neither possible nor desirable has eaten away at the traditional hard-nosed realism that used to animate the journalist’s trade: the worldview where a lie is a lie, a crime is a crime, and a scandal is a scandal.
A related idea, critical theory, has been used by the enemies of Western civilization in general and the American consensus in particular. Critical theory (including critical race theory) is based on the silly but destructive notion that whole areas of human accomplishment, including law, the institutions that support civilization, the philosophical, scientific and cultural legacy of Greek civilization, among other things, can simply be dismissed on tribal, racial, or ethnic grounds, as (in one malign phrase) the mind-fossils of “dead white men”.
In the world of critical theory, there never ever be objective or balanced journalism.
Relevance Criteria
Not everything can “make the news.” Journalism is all about sorting information about events according to their significance.
Relevance and intrinsic entertainment value govern the final sort. Relevance depends on significance, which is a product of the connections between reported events, the human condition, and the specific concerns of the audience.
All this must be fit within the
common moral framework because that framework is necessary to assess
significance. Facts alone are partial truths, and in themselves, often uninteresting.
An underlying moral/normative framework supplies the essential context for
facts to acquire their meaning and relevance. For example, the threshold
relevance of all crime news stems from the anti-moral nature of crime. After
that, relevance is influenced by proximity in time, space and circumstance.
The fundamental nature of core morality does not change, but the times themselves do. Therefore journalism needs to re-educate itself constantly in order to assign relevance to the avalanche of facts that make up potential story material. We are now in a position to identify several developing large-scale shifts in the journalistic landscape of the 21st century.
To assess any event’s news relevance, journalism needs to be able to understand these emerging changes, place events in the larger context against the backdrop of the American value consensus, and to connect the dots to related events.
In this sense, journalism is multiple draft history writing, with one obvious handicap: No one knows how the story comes out in the end.
Some Developing Large-Scale Shifts
in the Journalistic Landscape of the 21st Century
1. Communism
is dead as a viable economic doctrine and is rapidly dying as a real world
military threat. But communism survives in other forms. It flourishes in the academy, where it
undermines American Exceptionalism. And it has
emerged in
2. A limited form of capitalism (the market guided control of of the means of production and distribution of goods and services) has become the dominant economic paradigm. But there are many capitalist / free market variations, involving varying degrees of political control over market forces, which will remain in vigorous mutual competition for some time.
3. The
value fragmentation and demographic suicide of Western Europeans and (to a
lesser degree of their descendants in the
4. A
wave pattern of repeated authoritarian challenges to Western civilization seems
to be emerging. The Nazi and communist challenges have been supplanted by militant,
authoritarian Islamist forces. The
unrecognized story is the deep connection between rapidly growing Muslim
immigration in
5.
The emerging techno-terror challenge. This trend was foretold by the
cyber-technologist, Bill Joy, inventor of JAVA software, in his now famous,
seminal article in the April 2000 issue of Wired,
“Why the Future Doesn’t Need Us.” The
thesis: nano-technology, computer technology, and
genetic engineering technology are rapidly converging to produce dramatic new
terrorist weapons whose availability will be aided by low development cost and
malevolent information sharing on the internet.
APPENDIX
ONE
Note this article is reproduced from
![[Animation Express]](bumberstickermind_files/image003.gif){kind=small}
WIRED – Issue
8.04 2000
And is separately copyrighted as
follows:
Copyright © 1993-2004 The Condé Nast Publications Inc. All
rights reserved.
Copyright © 1994-2003 Wired Digital, Inc. All rights
reserved.
Why The
Future Doesn’t Need Us.
Our most powerful
21st-century technologies - robotics, genetic engineering, and nanotech - are
threatening to make humans an endangered species.
By
Bill Joy
From the moment I became
involved in the creation of new technologies, their ethical dimensions have
concerned me, but it was only in the autumn of 1998 that I became anxiously
aware of how great are the dangers facing us in the 21st century. I can date
the onset of my unease to the day I met Ray Kurzweil,
the deservedly famous inventor of the first reading machine for the blind and
many other amazing things.
Ray and I were both speakers
at George Gilder’s Telecosm conference, and I
encountered him by chance in the bar of the hotel after both our sessions were
over. I was sitting with John Searle, a
I had missed Ray’s talk and
the subsequent panel that Ray and John had been on, and they now picked right
up where they’d left off, with Ray saying that the rate of improvement of
technology was going to accelerate and that we were going to become robots or
fuse with robots or something like that, and John countering that this couldn’t
happen, because the robots couldn’t be conscious.
While I had heard such talk
before, I had always felt sentient robots were in the realm of science fiction.
But now, from someone I respected, I was hearing a strong argument that they
were a near-term possibility. I was taken aback, especially given Ray’s proven
ability to imagine and create the future. I already knew that new technologies
like genetic engineering and nanotechnology were giving us the power to remake
the world, but a realistic and imminent scenario for intelligent robots
surprised me.
It’s easy to get jaded about
such breakthroughs. We hear in the news almost every day of some kind of
technological or scientific advance. Yet this was no ordinary prediction. In
the hotel bar, Ray gave me a partial preprint of his then-forthcoming book The
Age of Spiritual Machines, which outlined a utopia he foresaw - one in which
humans gained near immortality by becoming one with robotic technology. On
reading it, my sense of unease only intensified; I felt sure he had to be
understating the dangers, understating the probability of a bad outcome along
this path.
I found myself most troubled
by a passage detailing a dystopian scenario:
THE NEW LUDDITE CHALLENGE
First let us postulate that
the computer scientists succeed in developing intelligent machines that can do
all things better than human beings can do them. In that case presumably all
work will be done by vast, highly organized systems of machines and no human effort
will be necessary. Either of two cases might occur. The machines might be
permitted to make all of their own decisions without human oversight, or else
human control over the machines might be retained.
If the machines are permitted
to make all their own decisions, we can’t make any conjectures as to the
results, because it is impossible to guess how such machines might behave. We
only point out that the fate of the human race would be at the mercy of the
machines. It might be argued that the human race would never be foolish enough
to hand over all the power to the machines. But we are suggesting neither that
the human race would voluntarily turn power over to the machines nor that the
machines would willfully seize power. What we do suggest is that the human race
might easily permit itself to drift into a position of such dependence on the
machines that it would have no practical choice but to accept all of the
machines’ decisions. As society and the problems that face it become more and
more complex and machines become more and more intelligent, people will let
machines make more of their decisions for them, simply because machine-made
decisions will bring better results than man-made ones. Eventually a stage may
be reached at which the decisions necessary to keep the system running will be
so complex that human beings will be incapable of making them intelligently. At
that stage the machines will be in effective control. People won’t be able to
just turn the machines off, because they will be so dependent on them that
turning them off would amount to suicide.
On the other hand it is
possible that human control over the machines may be retained. In that case the
average man may have control over certain private machines of his own, such as
his car or his personal computer, but control over large systems of machines
will be in the hands of a tiny elite - just as it is
today, but with two differences. Due to improved techniques the elite will have
greater control over the masses; and because human work will no longer be
necessary the masses will be superfluous, a useless burden on the system. If
the elite is ruthless they may simply decide to
exterminate the mass of humanity. If they are humane they may use propaganda or
other psychological or biological techniques to reduce the
birth rate until the mass of humanity becomes extinct, leaving the world
to the elite. Or, if the elite consists of
soft-hearted liberals, they may decide to play the role of good shepherds to
the rest of the human race. They will see to it that everyone’s physical needs
are satisfied, that all children are raised under psychologically hygienic
conditions, that everyone has a wholesome hobby to keep him busy, and that
anyone who may become dissatisfied undergoes “treatment” to cure his “problem.”
Of course, life will be so purposeless that people will have to be biologically
or psychologically engineered either to remove their need for the power process
or make them “sublimate” their drive for power into some harmless hobby. These engineered
human beings may be happy in such a society, but they will most certainly not
be free.
They will have been reduced to
the status of domestic animals.1
In the book, you don’t
discover until you turn the page that the author of this passage is Theodore
Kaczynski - the Unabomber. I am no apologist for Kaczynski. His bombs killed
three people during a 17-year terror campaign and wounded many others. One of
his bombs gravely injured my friend David Gelernter, one of the most brilliant
and visionary computer scientists of our time. Like many of my colleagues, I
felt that I could easily have been the Unabomber’s next target.
Kaczynski’s actions were
murderous and, in my view, criminally insane. He is clearly a Luddite, but simply saying this does not dismiss his
argument; as difficult as it is for me to acknowledge, I saw some merit in the
reasoning in this single passage. I felt compelled to confront it.
Kaczynski’s dystopian vision describes unintended consequences, a
well-known problem with the design and use of technology, and one that is
clearly related to Murphy’s law - “Anything that can
go wrong, will.” (Actually, this is Finagle’s law,
which in itself shows that Finagle was right.) Our overuse of antibiotics has
led to what may be the biggest such problem so far: the emergence of
antibiotic-resistant and much more dangerous bacteria. Similar things happened
when attempts to eliminate malarial mosquitoes using DDT caused them to acquire
DDT resistance; malarial parasites likewise acquired multi-drug-resistant
genes.2
The cause of many such
surprises seems clear: The systems involved are complex, involving interaction
among and feedback between many parts. Any changes to such a system will
cascade in ways that are difficult to predict; this is especially true when
human actions are involved.
I started showing friends the
Kaczynski quote from The Age of Spiritual Machines; I would hand them Kurzweil’s book, let them read the quote, and then watch
their reaction as they discovered who had written it. At around the same time,
I found Hans Moravec’s book Robot: Mere Machine to
Transcendent Mind. Moravec is one of the leaders in
robotics research, and was a founder of the world’s largest robotics research
program, at
The Short Run (Early 2000s)
Biological species almost
never survive encounters with superior competitors. Ten million years ago,
South and
In a completely free
marketplace, superior robots would surely affect humans as North American placentals affected South American marsupials (and as
humans have affected countless species). Robotic industries would compete
vigorously among themselves for matter, energy, and space, incidentally driving
their price beyond human reach. Unable to afford the necessities of life,
biological humans would be squeezed out of existence.
There is probably some
breathing room, because we do not live in a completely free marketplace.
Government coerces non market behavior, especially by collecting taxes.
Judiciously applied, governmental coercion could support human populations in
high style on the fruits of robot labor, perhaps for a long while.
A textbook dystopia
- and Moravec iss just getting wound up. He goes on to
discuss how our main job in the 21st century will be “ensuring continued
cooperation from the robot industries” by passing laws decreeing that they be
“nice,”3 and to describe how seriously dangerous a human can be “once
transformed into an unbounded super intelligent robot.” Moravec’s
view is that the robots will eventually succeed us - that humans clearly face
extinction.
I decided it was time to talk
to my friend Danny Hillis. Danny became famous as the
cofounder of Thinking Machines Corporation, which built a very powerful
parallel supercomputer. Despite my current job title of Chief Scientist at Sun
Microsystems, I am more a computer architect than a scientist, and I respect
Danny’s knowledge of the information and physical sciences more than that of
any other single person I know. Danny is also a highly regarded futurist who
thinks long-term - four years ago he started the Long Now Foundation, which is
building a clock designed to last 10,000 years, in an attempt to draw attention
to the pitifully short attention span of our society. (See “Test of Time, ”Wired 8.03, page 78.)
So I flew to
But I guess I wasn’t totally
surprised. I had seen a quote from Danny in Kurzweil’s
book in which he said, “I’m as fond of my body as anyone, but if I can be 200
with a body of silicon, I’ll take it.” It seemed that he was at peace with this
process and its attendant risks, while I was not.
While talking and thinking
about Kurzweil, Kaczynski, and Moravec,
I suddenly remembered a novel I had read almost 20 years ago -The White Plague,
by Frank Herbert - in which a molecular biologist is driven insane by the senseless
murder of his family. To seek revenge he constructs and disseminates a new and
highly contagious plague that kills widely but selectively. (We’re lucky
Kaczynski was a mathematician, not a molecular biologist.) I was also reminded
of the Borg ofStar Trek, a hive of partly biological,
partly robotic creatures with a strong destructive streak. Borg-like disasters
are a staple of science fiction, so why hadn’t I been more concerned about such
robotic dystopias earlier? Why weren’t other people
more concerned about these nightmarish scenarios?
Part of the answer certainly
lies in our attitude toward the new - in our bias toward instant familiarity
and unquestioning acceptance. Accustomed to living with almost routine
scientific breakthroughs, we have yet to come to terms with the fact that the
most compelling 21st-century technologies - robotics, genetic engineering, and
nanotechnology - pose a different threat than the technologies that have come
before. Specifically, robots, engineered organisms, and nanobots
share a dangerous amplifying factor: They can self-replicate. A bomb is blown
up only once - but one bot can become many, and
quickly get out of control.
Much of my work over the past
25 years has been on computer networking, where the sending and receiving of
messages creates the opportunity for out-of-control replication. But while
replication in a computer or a computer network can be a nuisance, at worst it
disables a machine or takes down a network or network service. Uncontrolled
self-replication in these newer technologies runs a much greater risk: a risk
of substantial damage in the physical world.
Each of these technologies
also offers untold promise: The vision of near immortality that Kurzweil sees in his robot dreams drives us forward; genetic
engineering may soon provide treatments, if not outright cures, for most
diseases; and nanotechnology and nanomedicine can
address yet more ills. Together they could significantly extend our average
life span and improve the quality of our lives. Yet, with each of these
technologies, a sequence of small, individually sensible advances leads to an
accumulation of great power and, concomitantly, great danger.
What was different in the 20th
century? Certainly, the technologies underlying the weapons of mass destruction
(WMD) - nuclear, biological, and chemical (NBC) - were powerful, and the
weapons an enormous threat. But building nuclear weapons required, at least for
a time, access to both rare - indeed, effectively unavailable - raw materials
and highly protected information; biological and chemical weapons programs also
tended to require large-scale activities.
The 21st-century technologies
- genetics, nanotechnology, and robotics ((GNR) - are so powerful that they can
spawn whole new classes of accidents and abuses. Most dangerously, for the
first time, these accidents and abuses are widely within the reach of
individuals or small groups. They will not require large facilities or rare raw
materials. Knowledge alone will enable the use of them.
Thus we have the possibility
not just of weapons of mass destruction but of knowledge-enabled mass
destruction (KMD), this destructiveness hugely amplified by the power of
self-replication.
I think it is no exaggeration
to say we are on the cusp of the further perfection of extreme evil, an evil
whose possibility spreads well beyond that which weapons of mass destruction
bequeathed to the nation-states, on to a surprising and terrible empowerment of
extreme individuals.
Nothing about the way I got
involved with computers suggested to me that I was going to be facing these
kinds of issues.
My life has been driven by a
deep need to ask questions and find answers. When I was 3, I was already
reading, so my father took me to the elementary school, where I sat on the
principal’s lap and read him a story. I started school early, later skipped a
grade, and escaped into books - I was incredibly motivated to learn. I asked
lots of questions, often driving adults to distraction.
As a teenager I was very
interested in science and technology. I wanted to be a ham radio operator but
didn’t have the money to buy the equipment. Ham radio was the Internet of its
time: very addictive, and quite solitary. Money issues
aside, my mother put her foot down - I was not to be a ham; I was antisocial
enough already.
I may not have had many close
friends, but I was awash in ideas. By high school, I had discovered the great
science fiction writers. I remember especially Heinlein’s Have Spacesuit Will
Travel and Asimov’s I, Robot, with its Three Laws of Robotics. I was enchanted
by the descriptions of space travel, and wanted to have a telescope to look at
the stars; since I had no money to buy or make one, I checked books on
telescope-making out of the library and read about making them instead. I
soared in my imagination.
Thursday nights my parents
went bowling, and we kids stayed home alone. It was the night of Gene
Roddenberry’s original Star Trek, and the program made a big impression on me.
I came to accept its notion that humans had a future in space, Western-style,
with big heroes and adventures. Roddenberry’s vision of the centuries to come
was one with strong moral values, embodied in codes like the Prime Directive:
to not interfere in the development of less technologically advanced
civilizations. This had an incredible appeal to me; ethical humans, not robots,
dominated this future, and I took Roddenberry’s dream as part of my own.
I excelled in mathematics in
high school, and when I went to the
I was lucky enough to get a
job programming early supercomputers and discovered the amazing power of large
machines to numerically simulate advanced designs. When I went to graduate
school at UC Berkeley in the mid-1970s, I started staying up late, often all
night, inventing new worlds inside the machines. Solving
problems. Writing the code that argued so strongly to
be written.
In The Agony and the Ecstasy,
Irving Stone’s biographical novel of Michelangelo, Stone described vividly how
Michelangelo released the statues from the stone, “breaking the marble spell,”
carving from the images in his mind.4 In my most ecstatic moments, the software
in the computer emerged in the same way. Once I had imagined it in my mind I
felt that it was already there in the machine, waiting to be released. Staying
up all night seemed a small price to pay to free it - to give the ideas
concrete form.
After a few years at Berkeley
I started to send out some of the software I had written - an instructional
Pascal system, Unix utilities, and a text editor called vi (which is still, to
my surprise, widely used more than 20 years later) - to others who had similar
small PDP-11 and VAX minicomputers. These adventures in software eventually
turned into the
Still, by the early 1980s, I
was drowning. The Unix releases were very successful,
and my little project of one soon had money and some staff, but the problem at
From all this, I trust it is
clear that I am not a Luddite. I have always, rather,
had a strong belief in the value of the scientific search for truth and in the
ability of great engineering to bring material progress. The Industrial
Revolution has immeasurably improved everyone’s life over the last couple
hundred years, and I always expected my career to involve the building of
worthwhile solutions to real problems, one problem at a time.
I have not been disappointed.
My work has had more impact than I had ever hoped for and has been more widely
used than I could have reasonably expected. I have spent the last 20 years
still trying to figure out how to make computers as reliable as I want them to
be (they are not nearly there yet) and how to make them simple to use (a goal
that has met with even less relative success). Despite some progress, the
problems that remain seem even more daunting.
But while I was aware of the
moral dilemmas surrounding technology’s consequences in fields like weapons
research, I did not expect that I would confront such issues in my own field,
or at least not so soon.
Perhaps it is always hard to
see the bigger impact while you are in the vortex of a change. Failing to
understand the consequences of our inventions while we are in the rapture of
discovery and innovation seems to be a common fault of scientists and
technologists; we have long been driven by the overarching desire to know that
is the nature of science’s quest, not stopping to notice that the progress to
newer and more powerful technologies can take on a life of its own.
I have long realized that the
big advances in information technology come not from the work of computer
scientists, computer architects, or electrical engineers, but from that of
physical scientists. The physicists Stephen Wolfram and Brosl
Hasslacher introduced me, in the early 1980s, to
chaos theory and nonlinear systems. In the 1990s, I learned about complex
systems from conversations with Danny Hillis, the
biologist Stuart Kauffman, the Nobel-laureate physicist Murray Gell-Mann, and
others. Most recently, Hasslacher and the electrical
engineer and device physicist Mark Reed have been giving me insight into the
incredible possibilities of molecular electronics.
In my own work, as codesigner of three microprocessor architectures - SPARC, picoJava, and MAJC - and as the
designer of several implementations thereof, I’ve been afforded a deep and
firsthand acquaintance with
Until last year I believed
that the rate of advances predicted by
But because of the recent
rapid and radical progress in molecular electronics - where individual atoms
and molecules replace lithographically drawn transistors - and related nanoscale technologies, we should be able to meet or exceed
the
As this enormous computing
power is combined with the manipulative advances of the physical sciences and
the new, deep understandings in genetics, enormous transformative power is
being unleashed. These combinations open up the opportunity to completely
redesign the world, for better or worse: The replicating and evolving processes
that have been confined to the natural world are about to become realms of
human endeavor.
In designing software and
microprocessors, I have never had the feeling that I was designing an
intelligent machine. The software and hardware is so fragile and the capabilities of the machine to “think” so clearly absent that, even
as a possibility, this has always seemed very far in the future.
But now, with the prospect of
human-level computing power in about 30 years, a new idea suggests itself: that
I may be working to create tools which will enable the construction of the
technology that may replace our species. How do I feel about this? Very uncomfortable. Having struggled
my entire career to build reliable software systems, it seems to me more than
likely that this future will not work out as well as some people may imagine.
My personal experience suggests we tend to overestimate our design abilities.
Given the incredible power of
these new technologies, shouldn’t we be asking how we can best coexist with
them? And if our own extinction is a likely, or even possible, outcome of our
technological development, shouldn’t we proceed with great caution?
The dream of robotics is,
first, that intelligent machines can do our work for us, allowing us lives of
leisure, restoring us to
How soon could such an
intelligent robot be built? The coming advances in computing power seem to make
it possible by 2030. And once an intelligent robot exists, it is only a small
step to a robot species - to an intelligent robot that can make evolved copies
of itself.
A second dream of robotics is
that we will gradually replace ourselves with our robotic technology, achieving
near immortality by downloading our consciousnesses; it is this process that
Danny Hillis thinks we will gradually get used to and
that Ray Kurzweil elegantly details inThe Age of Spiritual Machines. (We are beginning to see
intimations of this in the implantation of computer devices into the human
body, as illustrated on thecover ofWired
8.02.)
But if we are downloaded into
our technology, what are the chances that we will thereafter be ourselves or
even human? It seems to me far more likely that a robotic existence would not
be like a human one in any sense that we understand, that the robots would in
no sense be our children, that on this path our
humanity may well be lost.
Genetic engineering promises
to revolutionize agriculture by increasing crop yields while reducing the use
of pesticides; to create tens of thousands of novel species of bacteria,
plants, viruses, and animals; to replace reproduction, or supplement it, with
cloning; to create cures for many diseases, increasing our life span and our
quality of life; and much, much more. We now know with certainty that these
profound changes in the biological sciences are imminent and will challenge all
our notions of what life is.
Technologies such as human
cloning have in particular raised our awareness of the profound ethical and
moral issues we face. If, for example, we were to reengineer ourselves into
several separate and unequal species using the power of genetic engineering,
then we would threaten the notion of equality that is the very cornerstone of
our democracy.
Given the incredible power of
genetic engineering, it’s no surprise that there are significant safety issues
in its use. My friend Amory Lovins recently cowrote, along with Hunter Lovins,
an editorial that provides an ecological view of some of these dangers. Among
their concerns: that “the new botany aligns the development of plants with
their economic, not evolutionary, success.” (See “A Tale of Two Botanies,” page
247.) Amory’s long career has been focused on energy and resource efficiency by
taking a whole-system view of human-made systems; such a whole-system view
often finds simple, smart solutions to otherwise seemingly difficult problems,
and is usefully applied here as well.
After reading the Lovins’ editorial, I saw an op-ed by Gregg Easterbrook inThe New York Times (November 19, 1999) about genetically
engineered crops, under the headline: “Food for the Future: Someday, rice will
have built-in vitamin A. Unless the Luddites win.”
Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all would agree that golden
rice, with its built-in vitamin A, is probably a good thing, if developed with
proper care and respect for the likely dangers in moving genes across species
boundaries.
Awareness of the dangers
inherent in genetic engineering is beginning to grow, as reflected in the Lovins’ editorial. The general public is aware of, and
uneasy about, genetically modified foods, and seems to be rejecting the notion
that such foods should be permitted to be unlabeled.
But genetic engineering
technology is already very far along. As the Lovins
note, the USDA has already approved about 50 genetically engineered crops for
unlimited release; more than half of the world’s soybeans and a third of its
corn now contain genes spliced
in
from other forms of life.
While there are many important
issues here, my own major concern with genetic engineering is narrower: that it
gives the power - whether militarily, accidentally, or in a deliberate
terrorist act - to create a White Plague.
The many wonders of
nanotechnology were first imagined by the Nobel-laureate physicist Richard
Feynman in a speech he gave in 1959, subsequently published under the title
“There’s Plenty of Room at the Bottom.” The book that made a big impression on
me, in the mid-’80s, was Eric Drexler’s Engines of
Creation, in which he described beautifully how manipulation of matter at the
atomic level could create a utopian future of abundance, where just about
everything could be made cheaply, and almost any imaginable disease or physical
problem could be solved using nanotechnology and artificial intelligences.
A subsequent book, Unbounding the Future: The
Nanotechnology Revolution, which Drexler co-wrote,
imagines some of the changes that might take place in a world where we had
molecular-level “assemblers.” Assemblers could make possible incredibly
low-cost solar power, cures for cancer and the common cold by augmentation of
the human immune system, essentially complete cleanup of the environment,
incredibly inexpensive pocket supercomputers - in fact, any product would be manufacturable by assemblers at a cost no greater than that
of wood - spaceflight more accessible than transoceanic travel today, and
restoration of extinct species.
I remember feeling good about
nanotechnology after reading Engines of Creation. As a technologist, it gave me
a sense of calm - that is, nanotechnology showed us that incredible progress
was possible, and indeed perhaps inevitable. If nanotechnology was our future,
then I didn’t feel pressed to solve so many problems in the present. I would
get to Drexler’s utopian future in due time; I might
as well enjoy life more in the here and now. It didn’t make sense, given his
vision, to stay up all night, all the time.
Drexler’s
vision also led to a lot of good fun. I would occasionally get to describe the
wonders of nanotechnology to others who had not heard of it. After teasing them
with all the things Drexler described I would give a
homework assignment of my own: “Use nanotechnology to create a vampire; for
extra credit create an antidote.”
With these wonders came clear
dangers, of which I was acutely aware. As I said at a nanotechnology conference
in 1989, “We can’t simply do our science and not worry about these ethical
issues.”5 But my subsequent conversations with physicists convinced me that
nanotechnology might not even work - or, at least, it wouldn’t work anytime
soon. Shortly thereafter I moved to
Then, last summer, Brosl Hasslacher told me that nanoscale molecular electronics was now practical. This was
new news, at least to me, and I think to many people -
and it radically changed my opinion about nanotechnology. It sent me back to
Engines of Creation. Rereading Drexler’s work after
more than 10 years, I was dismayed to realize how little I had remembered of
its lengthy section called “Dangers and Hopes,” including a discussion of how
nanotechnologies can become “engines of destruction.” Indeed, in my rereading
of this cautionary material today, I am struck by how naive some of Drexler’s safeguard proposals seem, and how much greater I
judge the dangers to be now than even he seemed to then. (Having anticipated
and described many technical and political problems with nanotechnology, Drexler started the Foresight Institute in the late 1980s
“to help prepare society for anticipated advanced technologies” - most
important, nanotechnology.)
The enabling breakthrough to
assemblers seems quite likely within the next 20 years. Molecular electronics -
the new subfield of nanotechnology where individual molecules are circuit elements
- should mature quickly and become enormouusly lucrative within this decade,
causing a large incremental investment in all nanotechnologies.
Unfortunately, as with nuclear
technology, it is far easier to create destructive uses for nanotechnology than
constructive ones. Nanotechnology has clear military and terrorist uses, and
you need not be suicidal to release a massively destructive nanotechnological
device - such devices can be built to be selectively destructive, affecting,
for example, only a certain geographical area or a group of people who are
genetically distinct.
An immediate
consequence of the Faustian bargain in obtaining the great power of
nanotechnology is that we run a grave risk - the risk that we might destroy the
biosphere on which all life depends.
As Drexler
explained:
“Plants” with “leaves” no more
efficient than today’s solar cells could out-compete real plants, crowding the
biosphere with an inedible foliage. Tough omnivorous
“bacteria” could out-compete real bacteria: They could spread like blowing
pollen, replicate swiftly, and reduce the biosphere to dust in a matter of
days. Dangerous replicators could easily be too
tough, small, and rapidly spreading to stop - at least if we make no
preparation.
We have trouble enough controlling
viruses and fruit flies.
Among the cognoscenti of
nanotechnology, this threat has become known as the “gray goo
problem.” Though masses of uncontrolled replicators
need not be gray or gooey, the term “gray goo”
emphasizes that replicators
able to obliterate life might be less inspiring than a single species of
crabgrass. They might be superior in an evolutionary sense, but this need not
make them valuable.
The gray goo
threat makes one thing perfectly clear: We cannot afford certain kinds of
accidents with replicating assemblers.
Gray goo
would surely be a depressing ending to our human adventure on Earth, far worse
than mere fire or ice, and one that could stem from a simple laboratory
accident.6 Oops.
It is most of all the power of
destructive self-replication in genetics, nanotechnology, and robotics (GNR)
that should give us pause. Self-replication is the modus operandi of genetic
engineering, which uses the machinery of the cell to replicate its designs, and
the prime danger underlying gray goo in
nanotechnology. Stories of run-amok robots like the Borg, replicating or
mutating to escape from the ethical constraints imposed on them by their
creators, are well established in our science fiction books and movies. It is
even possible that self-replication may be more fundamental than we thought,
and hence harder - or even impossible - to control. A recent article by Stuart
Kauffman in Nature titled “Self-Replication: Even Peptides Do It” discusses the
discovery that a 32-amino-acid peptide can “autocatalyse
its own synthesis.” We don’t know how widespread this ability is, but Kauffman
notes that it may hint at “a route to self-reproducing molecular systems on a
basis far wider than Watson-Crick base-pairing.”7
In truth, we have had in hand
for years clear warnings of the dangers inherent in
widespread knowledge of GNR technologies - of the possibility of knowledge
alone enabling mass destruction. But these warnings haven’t been widely
publicized; the public discussions have been clearly inadequate. There is no
profit in publicizing the dangers.
The nuclear, biological, and
chemical (NBC) technologies used in 20th-century weapons of mass destruction
were and are largely military, developed in government laboratories.
In sharp contrast, the
21st-century GNR technologies have clear commercial uses and are being
developed almost exclusively by corporate enterprises. In this age of
triumphant commercialism, technology - with science as its handmaiden - is
delivering a series of almost magical inventions that are the most phenomenally
lucrative ever seen. We are aggressively pursuing the promises of these new
technologies within the now-unchallenged system of global capitalism and its
manifold financial incentives and competitive pressures.
This is the first moment in
the history of our planet when any species, by its own voluntary actions, has
become a danger to itself - as well as to vast numbers of others.
It might be a familiar
progression, transpiring on many worlds - a planet, newly formed, placidly
revolves around its star; life slowly forms; a kaleidoscopic procession of
creatures evolves; intelligence emerges which, at least up to a point, confers
enormous survival value; and then technology is invented. It dawns on them that
there are such things as laws of Nature, that these
laws can be revealed by experiment, and that knowledge of these laws can be
made both to save and to take lives, both on unprecedented scales. Science,
they recognize, grants immense powers. In a flash, they create world-altering
contrivances. Some planetary civilizations see their way through, place limits
on what may and what must not be done, and safely pass through the time of
perils. Others, not so lucky or so prudent, perish.
That is Carl Sagan, writing in 1994, in Pale Blue Dot, a book describing
his vision of the human future in space. I am only now realizing how deep his
insight was, and how sorely I miss, and will miss, his voice. For all its
eloquence, Sagan’s contribution was not least that of
simple common sense - an attribute that, along with humility, many of the
leading advocates of the 21st-century technologies seem to lack.
I remember from my childhood
that my grandmother was strongly against the overuse of antibiotics. She had
worked since before the first World War as a nurse and
had a commonsense attitude that taking antibiotics, unless they were absolutely
necessary, was bad for you.
It is not that she was an
enemy of progress. She saw much progress in an almost 70-year nursing career;
my grandfather, a diabetic, benefited greatly from the improved treatments that
became available in his lifetime. But she, like many levelheaded people, would
probably think it greatly arrogant for us, now, to be designing a robotic
“replacement species,” when we obviously have so much trouble making relatively
simple things work, and so much trouble managing - or even understanding -
ourselves.
I realize now that she had an awareness of the nature of the order of life, and of the
necessity of living with and respecting that order. With this respect comes a
necessary humility that we, with our early-21st-century chutzpah, lack at our
peril. The commonsense view, grounded in this respect, is often right, in
advance of the scientific evidence. The clear fragility and inefficiencies of
the human-made systems we have built should give us all pause; the fragility of
the systems I have worked on certainly humbles me.
We should have learned a
lesson from the making of the first atomic bomb and the resulting arms race. We
didn’t do well then, and the parallels to our current situation are troubling.
The effort to build the first
atomic bomb was led by the brilliant physicist J. Robert Oppenheimer.
Oppenheimer was not naturally interested in politics but became painfully aware
of what he perceived as the grave threat to Western civilization from the Third
Reich, a threat surely grave because of the possibility that Hitler might
obtain nuclear weapons. Energized by this concern, he brought his strong
intellect, passion for physics, and charismatic leadership skills to
What is striking is how this
effort continued so naturally after the initial impetus was removed. In a
meeting shortly after V-E Day with some physicists who felt that perhaps the
effort should stop, Oppenheimer argued to continue. His stated reason seems a
bit strange: not because of the fear of large casualties from an invasion of
We know that in preparing this
first atomic test the physicists proceeded despite a large number of possible
dangers. They were initially worried, based on a calculation by Edward Teller,
that an atomic explosion might set fire to the atmosphere. A revised
calculation reduced the danger of destroying the world to a three-in-a-million
chance. (Teller says he was later able to dismiss the prospect of atmospheric
ignition entirely.) Oppenheimer, though, was sufficiently concerned about the
result of Trinity that he arranged for a possible evacuation of the southwest
part of the state of
Within a month of that first,
successful test, two atomic bombs destroyed
It’s important to realize how
shocked the physicists were in the aftermath of the bombing of
In November 1945, three months
after the atomic bombings, Oppenheimer stood firmly behind the scientific
attitude, saying, “It is not possible to be a scientist unless you believe that
the knowledge of the world, and the power which this gives, is a thing which is
of intrinsic value to humanity, and that you are using it to help in the spread
of knowledge and are willing to take the consequences.”
Oppenheimer went on to work,
with others, on the Acheson-Lilienthal report, which,
as Richard Rhodes says in his recent book, Visions of Technology, “found a way
to prevent a clandestine nuclear arms race without resorting to armed world government”;
their suggestion was a form of relinquishment of nuclear weapons work by
nation-states to an international agency.
This proposal led to the
Baruch Plan, which was submitted to the United Nations in June 1946 but never
adopted (perhaps because, as Rhodes suggests, Bernard Baruch had “insisted on
burdening the plan with conventional sanctions,” thereby inevitably dooming it,
even though it would “almost certainly have been rejected by Stalinist Russia
anyway”). Other efforts to promote sensible steps toward internationalizing
nuclear power to prevent an arms race ran afoul either of US politics and
internal distrust, or distrust by the Soviets. The opportunity to avoid the
arms race was lost, and very quickly.
Two years later, in 1948,
Oppenheimer seemed to have reached another stage in his thinking, saying, “In
some sort of crude sense which no vulgarity, no humor, no overstatement can
quite extinguish, the physicists have known sin; and this is a knowledge they
cannot lose.”
In 1949, the Soviets exploded
an atom bomb. By 1955, both the
Nearly 20 years ago, in the
documentary, The Day After Trinity, Freeman Dyson
summarized the scientific attitudes that brought us to the nuclear precipice:
“I have felt it myself. The glitter of nuclear weapons. It is irresistible if you
come to them as a scientist. To feel it’s there in your hands, to release this
energy that fuels the stars, to let it do your bidding. To
perform these miracles, to lift a million tons of rock into the sky. It
is something that gives people an illusion of illimitable power, and it is, in
some ways, responsible for all our troubles - this, what you might call technical
arrogance, that overcomes people when they see what they can do with their
minds.”8
Now, as then, we are creators
of new technologies and stars of the imagined future, driven - this time by
great financial rewards and global competition - despite the clear dangers,
hardly evaluating what it may be like to try to live in a world that is the
realistic outcome of what we are creating and imagining.
In 1947,The
Bulletin of the Atomic Scientists began putting a Doomsday Clock on its cover.
For more than 50 years, it has shown an estimate of the relative nuclear danger
we have faced, reflecting the changing international conditions. The hands on
the clock have moved 15 times and today, standing at nine minutes to
In our time, how much danger
do we face, not just from nuclear weapons, but from all of these technologies?
How high are the extinction risks?
The philosopher John Leslie
has studied this question and concluded that the risk of human extinction is at
least 30 percent,9 while Ray Kurzweil believes we
have “a better than even chance of making it through,” with the caveat that he
has “always been accused of being an optimist.” Not only are these estimates
not encouraging, but they do not include the probability of many horrid outcomes
that lie short of extinction.
Faced with such assessments,
some serious people are already suggesting that we simply move beyond Earth as
quickly as possible. We would colonize the galaxy using von Neumann probes,
which hop from star system to star system, replicating as they go. This step
will almost certainly be necessary 5 billion years from now (or sooner if our
solar system is disastrously impacted by the impending collision of our galaxy
with the Andromeda galaxy within the next 3 billion years), but if we take Kurzweil and Moravec at their
word it might be necessary by the middle of this century.
What are the moral
implications here? If we must move beyond Earth this quickly in order for the
species to survive, who accepts the responsibility for the fate of those (most
of us, after all) who are left behind? And even if we scatter to the stars,
isn’t it likely that we may take our problems with us or find, later, that they
have followed us? The fate of our species on Earth and our fate in the galaxy
seem inextricably linked.
Another idea is to erect a
series of shields to defend against each of the dangerous technologies. The
Strategic Defense Initiative, proposed by the Reagan administration, was an
attempt to design such a shield against the threat of a nuclear attack from the
Clarke continued: “Looking
into my often cloudy crystal ball, I suspect that a total defense might indeed
be possible in a century or so. But the technology involved would produce, as a
by-product, weapons so terrible that no one would bother with anything as
primitive as ballistic missiles.” 10
In Engines of Creation, Eric Drexler proposed that we build an active nanotechnological shield - a form of immune system for the
biosphere - to defend against dangerous replicators
of all kinds that might escape from laboratories or otherwise be maliciously
created. But the shield he proposed would itself be extremely dangerous -
nothing could prevent it from developing autoimmune problems and attacking the
biosphere itself. 11
Similar difficulties apply to
the construction of shields against robotics and genetic engineering. These
technologies are too powerful to be shielded against in the time frame of
interest; even if it were possible to implement defensive shields, the side
effects of their development would be at least as dangerous as the technologies
we are trying to protect against.
These possibilities are all
thus either undesirable or unachievable or both. The only realistic alternative
I see is relinquishment: to limit development of the technologies that are too
dangerous, by limiting our pursuit of certain kinds of knowledge.
Yes, I know, knowledge is
good, as is the search for new truths. We have been seeking knowledge since
ancient times. Aristotle opened his Metaphysics with the simple statement: “All
men by nature desire to know.” We have, as a bedrock value in our society, long
agreed on the value of open access to information, and recognize the problems
that arise with attempts to restrict access to and development of knowledge. In
recent times, we have come to revere scientific knowledge.
But despite the strong
historical precedents, if open access to and unlimited development of knowledge
henceforth puts us all in clear danger of extinction, then common sense demands
that we reexamine even these basic, long-held beliefs.
It was Nietzsche who warned
us, at the end of the 19th century, not only that God is dead but that “faith
in science, which after all exists undeniably, cannot owe its origin to a
calculus of utility; it must have originated in spite of the fact that the
disutility and dangerousness of the ‘will to truth,’ of ‘truth at any price’ is
proved to it constantly.” It is this further danger that we now fully face -
the consequences of our truth-seeking. The truth that science seeks can certainly
be considered a dangerous substitute for God if it is likely to lead to our
extinction.
If we could agree, as a
species, what we wanted, where we were headed, and why, then we would make our
future much less dangerous - then we might understand what we can and should
relinquish. Otherwise, we can easily imagine an arms race developing over GNR
technologies, as it did with the NBC technologies in the 20th century. This is
perhaps the greatest risk, for once such a race begins, it’s
very hard to end it. This time - unlike during the Manhattan Project - we
aren’t in a war, facing an implacable enemy that is threatening our
civilization; we are driven, instead, by our habits, our desires, our economic
system, and our competitive need to know.
I believe that we all wish our
course could be determined by our collective values, ethics, and morals. If we
had gained more collective wisdom over the past few thousand years, then a
dialogue to this end would be more practical, and the incredible powers we are
about to unleash would not be nearly so troubling.
One would think we might be
driven to such a dialogue by our instinct for self-preservation. Individuals
clearly have this desire, yet as a species our behavior seems to be not in our
favor. In dealing with the nuclear threat, we often spoke dishonestly to
ourselves and to each other, thereby greatly increasing the risks. Whether this
was politically motivated, or because we chose not to think ahead, or because
when faced with such grave threats we acted irrationally out of fear, I do not
know, but it does not bode well.
The new Pandora’s boxes of
genetics, nanotechnology, and robotics are almost open, yet we seem hardly to
have noticed. Ideas can’t be put back in a box; unlike uranium or plutonium,
they don’t need to be mined and refined, and they can be freely copied. Once
they are out, they are out. Churchill remarked, in a famous left-handed
compliment, that the American people and their leaders “invariably do the right
thing, after they have examined every other alternative.” In this case,
however, we must act more presciently, as to do the right thing only at last
may be to lose the chance to do it at all.
As Thoreau said, “We do not
ride on the railroad; it rides upon us”; and this is what we must fight, in our
time. The question is, indeed, Which is to be master?
Will we survive our technologies?
We are being propelled into
this new century with no plan, no control, no brakes.
Have we already gone too far down the path to alter course? I don’t believe so,
but we aren’t trying yet, and the last chance to assert control - the fail-safe
point - is rapidly approaching. We have our first pet robots, as well as
commercially available genetic engineering techniques, and our nanoscale techniques are advancing rapidly. While the
development of these technologies proceeds through a number of steps, it isn’t
necessarily the case - as happened in the Manhattan Project and the Trinity
test - that the last step in proving a technology is large and hard. The breakthrough
to wild self-replication in robotics, genetic engineering, or nanotechnology
could come suddenly, reprising the surprise we felt
when we learned of the cloning of a mammal.
And yet I believe we do have a
strong and solid basis for hope. Our attempts to deal with weapons of mass
destruction in the last century provide a shining example of relinquishment for
us to consider: the unilateral
The clear conclusion was that
we would create additional threats to ourselves by pursuing these weapons, and
that we would be more secure if we did not pursue
them. We have embodied our relinquishment of biological and chemical weapons in
the 1972 Biological Weapons Convention (BWC) and the 1993 Chemical Weapons
Convention (CWC).12
As for the continuing sizable
threat from nuclear weapons, which we have lived with now for more than 50
years, the US Senate’s recent rejection of the Comprehensive Test Ban Treaty
makes it clear relinquishing nuclear weapons will not be politically easy. But
we have a unique opportunity, with the end of the Cold War, to avert a multipolar arms race. Building on the BWC and CWC
relinquishments, successful abolition of nuclear weapons could help us build
toward a habit of relinquishing dangerous technologies. (Actually, by getting
rid of all but 100 nuclear weapons worldwide - roughly the total destructive
power of World War II and a considerably easier task - we could eliminate this
extinction threat. 13)
Verifying relinquishment will
be a difficult problem, but not an unsolvable one. We are fortunate to have
already done a lot of relevant work in the context of the BWC and other
treaties. Our major task will be to apply this to technologies that are
naturally much more commercial than military. The substantial need here is for
transparency, as difficulty of verification is directly proportional to the
difficulty of distinguishing relinquished from legitimate activities.
I frankly believe that the
situation in 1945 was simpler than the one we now face: The nuclear
technologies were reasonably separable into commercial and military uses, and
monitoring was aided by the nature of atomic tests and the ease with which
radioactivity could be measured. Research on military applications could be
performed at national laboratories such as
The GNR technologies do not
divide clearly into commercial and military uses; given their potential in the
market, it’s hard to imagine pursuing them only in national laboratories. With
their widespread commercial pursuit, enforcing relinquishment will require a
verification regime similar to that for biological weapons, but on an
unprecedented scale. This, inevitably, will raise tensions between our
individual privacy and desire for proprietary information, and the need for
verification to protect us all. We will undoubtedly encounter strong resistance
to this loss of privacy and freedom of action.
Verifying the relinquishment
of certain GNR technologies will have to occur in cyberspace as well as at
physical facilities. The critical issue will be to make the necessary
transparency acceptable in a world of proprietary information, presumably by
providing new forms of protection for intellectual property.
Verifying compliance will also
require that scientists and engineers adopt a strong code of ethical conduct,
resembling the Hippocratic oath, and that they have the courage to whistle blow
as necessary, even at high personal cost. This would answer the call - 50 years
after
Thoreau also said that we will
be “rich in proportion to the number of things which we can afford to let
alone.” We each seek to be happy, but it would seem worthwhile to question
whether we need to take such a high risk of total destruction to gain yet more
knowledge and yet more things; common sense says that there is a limit to our
material needs - and that certain knowledge is too dangerous and is best
forgone.
Neither should we pursue near
immortality without considering the costs, without considering the commensurate
increase in the risk of extinction. Immortality, while perhaps the original, is
certainly not the only possible utopian dream.
I recently had the good
fortune to meet the distinguished author and scholar Jacques Attali, whose book, Lignes d’horizons (Millennium, in the English translation) helped
inspire the Java and Jini approach to the coming age
of pervasive computing, as previously described in this magazine. In his new
book, Fraternités, Attali
describes how our dreams of utopia have changed over time:
“At the dawn of societies, men
saw their passage on Earth as nothing more than a labyrinth of pain, at the end
of which stood a door leading, via their death, to the company of gods and to Eternity.
With the Hebrews and then the Greeks, some men dared free themselves from
theological demands and dream of an ideal City where
Jacques helped me understand
how these three different utopian goals exist in tension in our society today.
He goes on to describe a fourth utopia, Fraternity, whose foundation is
altruism. Fraternity alone associates individual happiness with the happiness
of others, affording the promise of self-sustainment.
This crystallized for me my
problem with Kurzweil’s dream. A technological
approach to Eternity - near immortality through robotics - may not be the most
desirable utopia, and its pursuit brings clear dangers. Maybe we should rethink
our utopian choices.
Where can we look for a new
ethical basis to set our course? I have found the ideas in the book Ethics for
the New Millennium, by the Dalai Lama, to be very helpful. As is perhaps well
known but little heeded, the Dalai Lama argues that the most important thing is
for us to conduct our lives with love and compassion for others, and that our
societies need to develop a stronger notion of universal responsibility and of
our interdependency; he proposes a standard of positive ethical conduct for
individuals and societies that seems consonant with Attali’s
Fraternity utopia.
The Dalai Lama further argues
that we must understand what it is that makes people happy, and acknowledge the
strong evidence that neither material progress nor the pursuit of the power of
knowledge is the key - that there are limits to what science and the scientific
pursuit alone can do.
Our Western notion of
happiness seems to come from the Greeks, who defined it as “the exercise of
vital powers along lines of excellence in a life affording them scope.” 15
Clearly, we need to find
meaningful challenges and sufficient scope in our lives if we are to be happy
in whatever is to come. But I believe we must find alternative outlets for our
creative forces, beyond the culture of perpetual economic growth; this growth
has largely been a blessing for several hundred years, but it has not brought
us unalloyed happiness, and we must now choose between the pursuit of
unrestricted and undirected growth through science and technology and the clear
accompanying dangers.
It is now more than a year
since my first encounter with Ray Kurzweil and John
Searle. I see around me cause for hope in the voices for caution and
relinquishment and in those people I have discovered who are as concerned as I
am about our current predicament. I feel, too, a deepened sense of personal
responsibility - not for the work I have already done, but for the work that I
might yet do, at the confluence of the sciences.
But many other people who know
about the dangers still seem strangely silent. When pressed, they trot out the
“this is nothing new” riposte - as if awareness of what could happen is
response enough. They tell me, There are universities
filled with bioethicists who study this stuff all day
long. They say, All this has been written about
before, and by experts. They complain, Your worries
and your arguments are already old hat.
I don’t know where these
people hide their fear. As an architect of complex systems I enter this arena
as a generalist. But should this diminish my concerns? I am aware of how much
has been written about, talked about, and lectured about so authoritatively.
But does this mean it has reached people? Does this mean we can discount the
dangers before us?
Knowing is not a rationale for
not acting. Can we doubt that knowledge has become a weapon we wield against
ourselves?
The experiences of the atomic
scientists clearly show the need to take personal responsibility, the danger
that things will move too fast, and the way in which a process can take on a
life of its own. We can, as they did, create insurmountable problems in almost
no time flat. We must do more thinking up front if we are not to be similarly surprised
and shocked by the consequences of our inventions.
My continuing professional
work is on improving the reliability of software. Software is a tool, and as a
tool builder I must struggle with the uses to which the tools I make are put. I
have always believed that making software more reliable, given its many uses,
will make the world a safer and better place; if I were to come to believe the
opposite, then I would be morally obligated to stop this work. I can now
imagine such a day may come.
This all leaves me not angry
but at least a bit melancholic. Henceforth, for me, progress will be somewhat
bittersweet.
Do you remember the beautiful
penultimate scene in
He leads himself to the
question, “Why is life worth living?” and to consider what makes it worthwhile
for him: Groucho Marx, Willie Mays, the second
movement of the Jupiter Symphony, Louis Armstrong’s recording of “Potato Head
Blues,” Swedish movies, Flaubert’s Sentimental Education, Marlon Brando, Frank Sinatra, the apples and pears by Cézanne, the
crabs at Sam Wo’s, and, finally, the showstopper: his
love Tracy’s face.
Each of us has our precious
things, and as we care for them we locate the essence of our humanity. In the
end, it is because of our great capacity for caring that I remain optimistic we
will confront the dangerous issues now before us.
My immediate hope is to
participate in a much larger discussion of the issues raised here, with people
from many different backgrounds, in settings not predisposed to fear or favor
technology for its own sake.
As a start, I have twice
raised many of these issues at events sponsored by the Aspen Institute and have
separately proposed that the American Academy of Arts and Sciences take them up
as an extension of its work with the Pugwash
Conferences. (These have been held since 1957 to discuss arms
control, especially of nuclear weapons, and to formulate workable policies.)
It’s unfortunate that the Pugwash meetings started only well after the nuclear genie
was out of the bottle - roughly 15 years too late. We are also getting a
belated start on seriously addressing the issues around 21st-century
technologies - the prevention of knowledge-enabled mass destruction - and
further delay seems unacceptable.
So I’m still searching; there
are many more things to learn. Whether we are to succeed or fail, to survive or fall victim to these technologies, is not
yet decided. I’m up late again - it’s almost
1 The passage Kurzweil quotes is from Kaczynski’s Unabomber Manifesto,
which was published jointly, under duress, by The New York Times and The
Washington Post to attempt to bring his campaign of terror to an end. I agree
with David Gelernter, who said about their decision:
“It was a tough call for the
newspapers. To say yes would be giving in to terrorism, and for all they knew
he was lying anyway. On the other hand, to say yes
might stop the killing. There was also a chance that someone would read the
tract and get a hunch about the author; and that is exactly what happened. The
suspect’s brother read it, and it rang a bell.
“I would have told them not to
publish. I’m glad they didn’t ask me. I guess.”
(Drawing Life: Surviving the
Unabomber. Free Press, 1997: 120.)
2 Garrett,
Laurie. The Coming Plague: Newly Emerging Diseases in a World Out of
Balance. Penguin, 1994: 47-52, 414, 419, 452.
3 Isaac Asimov described what
became the most famous view of ethical rules for robot behavior in his book, I,
Robot in 1950, in his Three Laws of Robotics: 1. A
robot may not injure a human being, or, through inaction, allow a human being
to come to harm. 2. A robot must obey the orders given it by human beings,
except where such orders would conflict with the First Law. 3. A robot must
protect its own existence, as long as such protection does not conflict with
the First or Second Law.
4 Michelangelo wrote a sonnet
that begins:
Non ha l’ ottimo artista
alcun concetto
Ch’ un marmo solo in
sè non circonscriva
La man che ubbidisce
all’ intelleto.
Stone translates this as:
The best of artists hath no
thought to show
which
the rough stone in its superfluous shell
doth
not include; to break the marble spell
is
all the hand that serves the brain can do.
Stone describes the process:
“He was not working from his drawings or clay models; they had all been put
away. He was carving from the images in his mind. His eyes and hands knew where
every line, curve, mass must emerge, and at what depth in the heart of the
stone to create the low relief.”
(The Agony
and the Ecstasy. Doubleday, 1961: 6, 144.)
5 First Foresight Conference
on Nanotechnology in October 1989, a talk titled “The Future of Computation.” Published in Crandall, B. C. and James Lewis, editors. Nanotechnology:
Research and Perspectives. MIT Press, 1992: 269. See
alsowww.foresight.org/Conferences/MNT01/Nano1.html.
6 In his 1963 novel, Cat’s
Cradle, Kurt Vonnegut imagined a gray-goo-like accident
where a form of ice called ice-nine, which becomes solid at a much higher
temperature, freezes the oceans.
7 Kauffman,
Stuart. “Self-replication: Even Peptides Do It.” Nature, 382,
8 Else, Jon.The
Day After Trinity: J. Robert Oppenheimer and The
Atomic Bomb (available at www.pyramiddirect.com).
9 This estimate is in Leslie’s
book, The End of the World: The Science and Ethics of Human Extinction, where
he notes that the probability of extinction is substantially higher if we
accept Brandon Carter’s Doomsday Argument, which is, briefly, that “we ought to
have some reluctance to believe that we are very exceptionally early, for
instance in the earliest 0.001 percent, among all humans who will ever have
lived. This would be some reason for thinking that humankind will not survive
for many more centuries, let alone colonize the galaxy. Carter’s doomsday
argument doesn’t generate any risk estimates just by itself. It is an argument for
revising the estimates which we generate when we consider various possible
dangers.” (Routledge, 1996: 1, 3,
145.)
10 Clarke, Arthur C.
“Presidents, Experts, and Asteroids. ”Science,
11 And, as David Forrest
suggests in his paper “Regulating Nanotechnology Development,” available
atwww.foresight.org/NanoRev/Forrest1989.html, “If we used strict liability as
an alternative to regulation it would be impossible for any developer to
internalize the cost of the risk (destruction of the biosphere), so
theoretically the activity of developing nanotechnology should never be
undertaken.” Forrest’s analysis leaves us with only government regulation to
protect us - not a comforting thought.
12 Meselson, Matthew. “The Problem of
Biological Weapons.” Presentation to the 1,818th
Stated Meeting of the
13 Doty,
Paul. “The Forgotten Menace: Nuclear Weapons Stockpiles Still Represent
the Biggest Threat to Civilization. ”Nature, 402,
14 See also Hans Bethe’s 1997 letter to President Clinton, at www.fas.org/bethecr.htm.
15 Hamilton,
Bill Joy, cofounder and Chief Scientist of Sun Microsystems, was cochair of the presidential commission on the future of IT
research, and is coauthor of The Java Language Specification. His work on theJini pervasive computing technology was featured in Wired
6.08.
Copyright © 1993-2004 The Condé Nast
Publications Inc. All rights reserved.
Copyright © 1994-2003 Wired
Digital, Inc. All rights reserved.
This material was posted on The
Policy Think Site < www.jaygaskill.com
> where there are other topical and perennially relevant articles. All
material herein, except The APPENDIX is Copyright ©
2004 by Jay B. Gaskill