“We write at root to make sense of life. And what do we seek to understand more than anything? Our deepest fears.”  Josh Weil

George Rebane

Union publisher Don Rogers’ Friday essay – ‘Write where you fear the most’ – was an unexpected helping of food for thought this morning.  Mr Rogers takes us through a bit of personal history about his education and membership in the guild of wordsmiths, as he promotes our attendance at the Sierra Writers Conference in January at which “top fiction author” Josh Weil of Nevada City will be the keynote speaker.  Weil’s dictum to the rest of us is “Write what scares you.  If what you are writing doesn’t scare you, you probably shouldn’t write it.”

Now Don Rogers reads a lot of books that don’t make it onto my reading list, simply because time is limited and we come at life from different angles.  But his dissertation, with references to multiple authors (mostly unknown to me) and famous personages, on the meta-motivations for good writing struck a chord which I’ll let vibrate a bit in what follows.  I recommend you read Rogers to see what, if anything, starts vibrating in you.

Weil’s observations and advice struck my motivation, for these years of scribbling on RR, right between the eyes – ‘Observations and Interpretations of Events from the Last Great Century of Man’.  My technical training and work in machine intelligence, combined with the experiences in man’s greatest war and its aftermath, has focused my attention on the Singularity and humanity’s race, during these pre-Singularity years of accelerating technology, between proponents of a global tyranny and a stable Westphalian order in which most people will enjoy liberty and their own pursuits of happiness.  Once the Singularity is ensconced, all bets are off – that’s why it’s called a singularity.  And therein lie the things that scare me.

Right now it looks like the forces promoting global tyranny are in the lead, and may win the race to a single global government, or a perpetually fluid order of a few continually warring massive Orwellian states.   The champions of such an end are the central planner and controller elites of the Left who sell their unachievable goals to the masses as a utopic great age of Man in which we are all healthy, equal, and organize our pursuits in an environment of ‘from each according to his abilities, to each according to his needs.’  The outlined road through multi-culturalism, diversity, inclusivity, social justice, … will lead to a world homogeneously happy in thought and deed.  Ten thousand identical flowers will bloom; weeds will be culled.  State enforced altruism will rule.

However, every serious student of history knows that human tragedy has inevitably followed kingdoms and nations grown beyond the ability of governments to serve its citizens instead of its cadres of elites.  Tyranny has been the end state leading to dissolution of all such orders of power since pre-biblical days.  Tyranny has been the most stable form of governance – when one tyranny falls, one or more new ones immediately replace it.  When transport, communication, and weaponry relied on easily replicated low-technology solutions, revolution against an autocracy was conceivable and successfully practiced a few times.  Today we are rapidly departing that idyllic and egalitarian world in which an aggrieved people could organize and mount a successful resistance against an established technologically-savvy government.  Tomorrow it will be impossible as the state perfects today’s technologies for monitoring and controlling our every behavior so that it remains within the ‘norms’ it has established – China is showing the way and willing to sell the needed technology to all the world’s would-be tyrants.

In the meantime left-leaning developed countries are busy installing the same technologies for the claimed salutary purposes of reducing crime, drug addiction, providing affordable healthcare, preventing terrorism, facilitating travel and deliveries, and generally doing all the things to take risk out of life.  Not many people yet know that such embedded smart systems can be repurposed overnight to control and constrain a population.

So what kind of futures are we looking at?  The simple matrix shown below summarizes for me the main ones to consider.

The three columns indicate the kind of AGI (Artificial General Intelligence) that the Singularity will produce.  The benign kind will be controllable and will be able to help its human masters achieve the things that it is commanded to achieve, whether those be new medicines or designs of starships or devastating weapons.  The neutral AGI will be an autonomous being that seeks to maximize its own utility within unknown behavioral constraints using whatever resources it can command.  It will look at humanity simply as another entity in its environment and possibly in its spectrum of available resources.  The last kind is an autonomous and unconstrained AGI whose utility maximization requires the removal of human life from earth.

The two rows denote a globe with a mostly free (Westphalian) pre-Singularity world order, and a not free global world order as described above.  The matrix elements are colored to reflect possible ‘utopian’ (green) and ‘dystopian’ (red) post-Singularity futures.  A global tyrannical or even autocratic social order will not benefit mankind no matter which AGI is realized.  The autocratic government(s) will use the benign AGI for implementing and enforcing policies that will primarily serve the elites.  And the neutral and vicious promise more of the same to the extent that they can be influenced by any residual human control or argument.

The only salutary post-Singularity futures which can be achieved require either a benign AGI whose utility and constraints can be determined by Man, or a neutral AGI with whom humans can reason and/or negotiate a mutually satisfactory co-existence with the knowledge that the AGI will always be the dominant power in the solar system.  The red (dystopian) half of the Free-Neutral contingency will occur if the AGI decides to deploy/harness mass and energy in a manner to benefit itself without regard to the collateral damage it will do to mankind.  The destruction of the human race will then be the result of amoral actions by the AGI.

From this rough summary, one sees that there is a lot to be scared about as the Singularity approaches.  This vision is reinforced when we consider the fact that R&D toward AGI is both uncontrolled and, I argue, uncontrollable.  There is a hopeful school of computer scientists and philophers (e.g. Max Tegmark, Nick Bostrom) today who believe that humans will be able to not only impose a utility function on an AGI, but also to ‘program in’ sets of values, mores, and behavioral constraints that the new being will evince as it starts to serve Man (benign) or autonomously better itself (neutral).   Will the AGI be conscious, we’ll leave that discussion for another time – in the interval consider reading The Consciousness Instinct (2018) by Michael Gazzaniga.

In sum, this is what I see as the important stuff happening in the remainder of this century, because in the next century we will either have done ourselves in, or have been done in by AGI, or, if God is kind, we will be busy making ourselves into trans-humans.  And all of it is pretty scary.