Tuesday, November 24, 2015

Why Leo is Noteworthy

This post discusses what I think matters most about Leo. This post assumes you are an experienced Leo user. It does not try to duplicate Leo's Tutorial.

Leo is a superb tool for understanding, studying and organizing any kind of complex data, including computer programs. The first great Aha in Leo's history was that webs (literate programs) are outlines in disguise. Leo's importers (@auto) make it easy to studying other people's programs. Leo's always-present outline structure clarifies overall program structure and makes many kinds of comments unnecessary.

Leo is a superb browser for code and data. Unlike many other folding editors, Leo remembers which nodes were expanded when you last saved an outline. This is surprisingly important. And Leo's clones let you organize any data as you want, even if all folds are collapsed.

Leo is a uniquely powerful scripting environment. This power comes from three sources: Leo's API, Leo's ability to compose scripts from outlines and Leo's underlying data structure, a Directed Acyclic Graph, the basis for Leo's clones.

Leo's API consists primarily of generators, such as c.all_positions(), p.self_and_subtree(), etc. and properties, such as p.b, p.h, p.gnx and p.v.u. Leo's API makes it trivial to write scripts to access or change any node. AFAIK, these capabilities are unique. Simulating them in vim or Emacs is possible, but so is simulating Python's capabilities in C...

Afaik, no other scripting environment allows you to compose scripts from outlines. @file, @clean, @auto, @others and section references and definitions make this possible. Section references and definitions are modeled on the noweb language, but all of Leo's script composition features are fully integrated into Leo's outline structure.

Leo's outline nodes have headlines (p.h) and body text (p.b) and extensible information (p.v.u). Headlines are descriptions (meta-data) of the data in p.b and p.v.u. Scripts can rapidly discover and categorize data using metadata. Leo's @ convention for headlines (@clean, @file, @auto, @html, etc.) show how extensible this node typing is.

So much for the theory. The following also are important in practice:

The invention/discovery of @clean earlier this year completes Leo in some sense.

Acknowledgements: Working with Leo's community of Leo's developers and users has been a great pleasure for over 20 years. My only regret is that Bernhard Mulder and Bob Fitzwater are no longer with us. Both made essential contributions. Bob Fitzwater was my mentor. He gently pushed me to consider design, not just "bit twiddling". Bernhard Mulder contributed two of the most important elements of Leo: Leo's traversers (generators) and the original @shadow algorithm. Neither @clean nor the revised Mulder/Ream algorithm could possibly have happened without him. I miss both these great thinkers. Both would have been proud of what they helped create.

A successful software tool is one that was used to do something undreamed of by its author.' -- Stephen Johnson
Leo is a wild success on this score. I foresaw none of these developments 20 years ago: Leo's minibuffer, @button, @test, @auto, @clean, Leo's plugin architecture, the rst3 command, the Leo bridge and the IPython bridge. Surely many other features and uses could be added. None of these would have happened without Leo's community of brilliant people. These features create the Leonine world. Who knows what will be the result...


P. S. As I write this, I see that @button is nowhere mentioned in Leo's History Chapter. That's crazy: @button is arguably the most brilliant scripting idea ever created anywhere. Many thanks to 'e', whoever you are. I'd like to thank you by name. @button lead directly to @test.


My last lecture: Reading List

If I remember correctly, in Larry Niven's novel Ringworld, one of the characters says something like "our scientists have proved that God does not exist". At the time I wondered whether how such a statement could possibly be proved (or disproved).

Years later, after reading several books by Stephen Jay Gould, including The Panda's Thumb, it gradually dawned on me that Charles Darwin did, in fact, begin that proof, by showing that the Argument from Design is fallacious. Yes, it's natural to believe that plants and animals are designed, but in fact the "designer" is evolution, not a supernatural being.

In fact, the evidence for atheism is overwhelming. For those who still believe in belief, I recommend the following books:

These ideas may seem profoundly threatening. That need not be so. Atheists, like everyone else, feel love, beauty, spirituality and other selfless or transcendent emotions. Atheism does not diminish these aspects of life, it simply explains them as arising from evolution. People don't need non-existent beings to be happy, to live morally and with a sense of purpose. We create our purposes. Nothing else does.

Enjoying the discoveries of science is one my great pleasures. In particular, the theory of evolution is the most beautiful, elegant, fertile, powerful and successful scientific theory ever created:

My last lecture part 2: dealing with the world as it is

This is part 2 of My Last Lecture. It describes my dealing with the world as it is.

The practice of mindfulness promotes an equanimous, happy and compassionate life. It creates a visceral sense of connection and compassion with others, even those whose beliefs and actions we profoundly disagree with. Here are some resources:

A compassion meditation for time of war. I remember weeping during this meditation as my anger about 9/11 shifted.

Practicing Mindfulness: An Introduction to Meditation by Professor Mark W. Muesse. This superb introduction might be the only guide you will ever need. I have gotten great benefit from two contemplative practices discussed my Prof. Muesse:

The Impermanence of All Things. This leads me to appreciate what is here, right now

Just Like Me. For me, this practice reinforces the compassion developed in "A compassion meditation for time of war" mentioned above.

In addition, I recommend two more practices:

Mudita is my primary antidote for envy. Mudita is the delight in other people's well-being, accomplishments and good fortune. I cultivate mudita by going where people are enjoying themselves, say by riding a bicycle around Madison on a beautiful summer day. If you want to be happy, practice mudita and compassion. If you want to suffer, envy others and enjoy schadenfreude.

Laughter yoga. A great way to connect with people and to remove barriers to joy.

That's it. In the long run, we can't know whether our actions are solutions, mitigations, or entirely useless​, ​but in the short run​​ they certainly will have some effect.

Tuesday, November 3, 2015

My last lecture part 1: Looming catastrophes

This is my last lecture.  Or perhaps it is the first lecture of the rest of my life.
A lot of professors give talks titled "The Last Lecture." Professors are asked to consider their demise and to ruminate on what matters most to them. And while they speak, audiences can't help but mull the same question: What wisdom would we impart to the world if we knew it was our last chance? If we had to vanish tomorrow, what would we want as our legacy? From this page.
Here are my thoughts about what matters most in life.  This first part contemplates my demise, and the demise of everything I hold dear. This is no longer a theoretical exercise.

The second part (to be posted later) will explain how contemplating the impermanence of all things allows me to live with joy, compassion and laughter in the face of immanent disaster.

Ground rules

I welcome all constructive comments. They must be based on credible scientific evidence. This is a moderated forum.  Abuse,  repetitions of denialist tropes,  and religious and political dogma will not be tolerated.

Executive summary

Credible computer modeling suggests that human civilization will collapse in this century. This collapse will be caused by capital shortages arising from the increased difficulty in extracting resources. The collapse will not happen because of actual shortages of resources. The collapse will occur while significant resources remain in the ground.

We are experiencing the sixth mass extinction in earth's history. This extinction is driven by the impacts of rising human population and industrialization. Humans will likely go extinct relatively early in this mass extinction.

Human caused global warming will accelerate the collapse of both civilizations and ecosystems.

Neither free markets nor technical innovation can be expected to prevent these collapses. Economic and political forces have delayed required responses for the last 40 years. Significant decreases in population or CO2 concentrations are unlikely before collapse becomes inevitable.

The collapse of civilization

The world's societies face collapse within 100 years or sooner. This overshoot and collapse was described 40 years ago in the Limits to Growth (LTG). For the last 40 years, the world has unjustifiably ignored those warnings. The LTG looks more valid than ever:
  • There have been no serious critiques of the LTG. There have been egregious smears.
  • The World3 model is the product of acknowledged experts in systems modeling.
  • The results of the World3 model arise from reasonable and justifiable assumptions.
  • No other world model has been as successful as the World3 model.
  • The world has generally tracked the BAU (Business as Usual) scenario for the past 40 years.
Graham Turner, a professor at the University of Melbourne, provides a recent assessment of LTG. I recommend this paper to anyone who prefers to dismiss these warnings. Here is Turner's description of the BAU scenario:
As described below, data from the forty years or so since the LTG study was completed indicates that the world is closely tracking the BAU scenario. In the BAU, during the 20th century increasing population and demand for material wealth drives more industrial output, which grows at a faster rate than population. Pollution from increasing economic activity increases, but from a very low level, and does not seriously impact the population or environment.

However, the increased industrial activity requires ever increasing resource inputs (albeit offset by improvements in efficiency), and resource extraction requires capital (machinery) which is produced by the industrial sector (which also produces consumption goods). Until the non-renewable resource base is reduced to about 50 per cent of the original or ultimate level, the World3 model assumed only a small fraction (5 per cent) of capital is allocated to the resource sector, simulating access to easily obtained or high quality resources, as well as improvements in discovery and extraction technology. However, as resources drop below the 50 per cent level in the early part of the simulated 21st century and become harder to extract and process, the capital needed begins to increase. For instance, at 30 per cent of the original resource base, the fraction of total capital that is allocated in the model to the resource sector reaches 50 per cent, and continues to increase as the resource base is further depleted (shown in Meadows et al., 1974).

With significant capital subsequently going into resource extraction, there is insufficient capital available to fully replace degrading capital within the industrial sector itself. Consequently, despite heightened industrial activity attempting to satisfy multiple demands from all sectors and the population, actual industrial output (per capita) begins to fall precipitously from about 2015, while pollution from the industrial activity continues to grow. The reduction of inputs to agriculture from industry, combined with pollution impacts on agricultural land, leads to a fall in agricultural yields and food produced per capita. Similarly, services (e.g. health and education) are not maintained due to insufficient capital and inputs. Diminishing per capita supply of services and food causes a rise in the death rate from about 2020 (and a somewhat lower rise in the birth rate, due to reduced birth control options). The global population therefore falls, at about half a billion per decade, starting at about 2030. Following the collapse, the output of the World3 model for the BAU (Figure 1) shows that average living standards for the aggregate population (material wealth, food and services per capita basically reflecting OECD-type conditions) resemble those of the early 20th century.

The implications of the BAU scenario are stark: [a] global collapse of the economic system and population. Essentially this collapse is caused by resource constraints (Meadows et al.,1972), following the dynamics and interactions described above. The calibrated dynamics reflect observed responses within the economy to changing levels of abundance or scarcity (Meadows et al., 1974), obviating the need for modeling prices as the communication channel of the economic responses.
Note the last sentence. It is not necessary to model financial markets in order to reach these conclusions. The model merely assumes that as resources become depleted, more physical capital is required to extract those resources. This is an entirely reasonable assumption.

Systems thinking

The critics of the World3 model evidently understand neither the model nor its foundations. The field of systems dynamics is well established and has successfully modeled many different kinds of systems. Thinking in Systems, by Donella Meadows, is a superb introduction.

Systems thinking will change your view of the world. These quotes give a flavor of this altered view. Here is one of Meadows' most famous quotes:
Folks who do systems analysis have a great belief in "leverage points." These are places within a complex system—a corporation, an economy, a living body, a city, an ecosystem—where a small shift in one thing can produce big changes in everything...

The systems analysis community has a lot of lore about leverage points. Those of us who were trained by the great Jay Forrester at MIT have all absorbed one of his favorite stories. 'People know intuitively where leverage points are,” he says. "Time after time I've done an analysis of a company, and I've figured out a leverage point—in inventory policy, maybe, or in the relationship between sales force and productive force, or in personnel policy. Then I've gone to the company and discovered that there’s already a lot of attention to that point. Everyone is trying very hard to push it IN THE WRONG DIRECTION!"

The classic example of that backward intuition was my own introduction to systems analysis, the world model. Asked by the Club of Rome to show how major global problems—poverty and hunger, environmental destruction, resource depletion, urban deterioration, unemployment—are related and how they might be solved, Forrester made a computer model and came out with a clear leverage point1: Growth. Not only population growth, but economic growth. Growth has costs as well as benefits, and we typically don’t count the costs—among which are poverty and hunger, environmental destruction, etc.—the whole list of problems we are trying to solve with growth! What is needed is much slower growth, much different kinds of growth, and in some cases no growth or negative growth.

The world’s leaders are correctly fixated on economic growth as the answer to virtually all problems, but they’re pushing with all their might in the wrong direction...

So one day I was sitting in a meeting about how to make the world work better—actually it was a meeting about how the new global trade regime, NAFTA and GATT and the World Trade Organization, is likely to make the world work worse. The more I listened, the more I began to simmer inside. “This is a HUGE NEW SYSTEM people are inventing!” I said to myself. “They haven’t the SLIGHTEST IDEA how this complex structure will behave,” myself said back to me. “It’s almost certainly an example of cranking the system in the wrong direction—it’s aimed at growth, growth at any price!! And the control measures these nice, liberal folks are talking about to combat it—small parameter adjustments, weak negative feedback loops—are PUNY!!!

The collapse of ecosystems

We are living in the midst of the sixth mass extinction in earth's history. The human race may go extinct as the result.
If present trends continue one half of all species of life on earth will be extinct in less than 100 years, as a result of habitat destruction, pollution, invasive species, and climate change. -- here.

Primary sources

Ceballos et. al., Science Magazine: Accelerated modern human–induced species losses: Entering the sixth mass extinction
If it is allowed to continue, life would take many millions of years to recover and our species itself would likely disappear early on. -- Gerardo Ceballos, Science Magazine.
S. L. Pimm, et. al., Science Magazine: The biodiversity of species and their rates of extinction, distribution, and protection
Current rates of extinction are about 1000 times the background rate of extinction.
William Rees, the originator and co-developer (with his graduate students) of ecological footprint analysis has this to say:
The average world citizen has an eco-footprint of about 2.7 global average hectares while there are only 2.1 global hectare of bioproductive land and water per capita on earth. This means that humanity has already overshot global biocapacity by 30% and now lives unsustainably by depleting stocks of 'natural capital'.
Humans may pride themselves as being the best evidence for intelligent life on Earth, but an alien observer would record that the (un)sustainability conundrum has the global community floundering in a swamp of cognitive dissonance and collective denial...Indeed, our alien friend might go so far as to ask why our reasonably intelligent species seems unable to recognize the crisis for what it is and respond accordingly.

Secondary sources

The Sixth Extinction: An Unnatural History, by Elizabeth Kolbert.  This book won the Pulitzer Prize for general non-fiction.

When Life Nearly Died: The Greatest Mass Extinction of All Time, by Michael J. Benton.  This is the story of the greatest mass extinction of all time: the Permian-Triassic extinction.


Earth enters sixth extinction phase with many species–including our own–labeled 'the walking dead'

Will Humans Survive the Sixth Great Extinction?
The new study that's generated so much conversation estimates that as many as three-quarters of animal species could be extinct within several human lifetimes.
Hundreds of links re the sixth mass extinction.

Global warming is accelerating collapse

Human caused climate change is settled science. It is contributing to the sixth extinction and it will accelerate the collapse of world civilization. The World3 model does not model climate, so the model is likely too optimistic.

For those still confused about global warming, I recommend the following:

How Climate Change Deniers Sound to Normal People

99 One-Liners Rebutting Denier Talking Points--With Links To The Full Climate Science

Climate Denial Crock of the Week, by Peter Sinclair. This is a superb blog that is improving how scientists communicate with the general public.

James' Hanson's TED Talk

Another TED Talk What a 4C temperature increase will mean for the world.

The end of another denialist meme: T. R. Karl et al. in Science magazine: Possible artifacts of data biases in the recent global surface warming hiatus Far from being alarmist, the IPCC has been way too cautious.
Here, we present an updated global surface temperature analysis that reveals that global trends are higher than those reported by the Intergovernmental Panel on Climate Change, especially in recent decades, and that the central estimate for the rate of warming during the first 15 years of the 21st century is at least as great as the last half of the 20th century. These results do not support the notion of a 'slowdown' in the increase of global surface temperature.

Free markets and innovation will fail

Humanity is putting blind faith in the power of free markets and human ingenuity to avert catastrophe. Neither is likely to be effective.

Free markets will fail

  • The causes of overshoot and collapse (overpopulation, habitat destruction, pollution and extinction) are invisible to world markets. They are treated as externalities.
  • Economists usually want unlimited growth, which must lead to overshoot and collapse.
  • Corporations often use their power to the detriment of their customers and society. Some examples: The columns of Gretchen Morgenson, and recent front-page story in the New York Times about abusive contracts that prevent consumers from getting redress from courts.

Human ingenuity will fail

  • Human ingenuity can impede progress. Climate denialism may be on the wane but it has delayed humanity's response by at least four decades.
  • Human ingenuity itself has limits. Innovators require supportive environments. The stresses that undermine societies and ecosystems will undermine the infrastructure needed by science and technology.
  • Technological change does not happen instantly. We have already run out of time to implement easy solutions such as lower birth rate, lower rates of economic growth or reduce humanity's ecological footprint.
  • Many innovations actually increase the likelihood of collapse.

Population and CO2 will increase

Population: Only a drastic, immediate, planned decrease in earth's population would raise hopes for the future, but population estimates are being revised upward. Hopes for a demographic transition look like wishful thinking.

Worse, any decrease in population would cause near-term economic disruption. Most economists want steadily increasing populations. For ongoing discussions of the adverse economic effects of falling population, see the writings of John Mauldin.

CO2: Reducing the rate of increase of CO2 emissions is not enough. Climate will stabilize only CO2 is actually removed from the atmosphere. Increasing renewable energy does not even necessarily decrease CO2 emissions, and certainly does not decrease CO2 from the air.

There is presently no economically viable way to remove CO2 in bulk from the atmosphere. Many researchers are working on this problem, but such approaches will not be competitive unless a global carbon tax is imposed on all fossil fuels. The world needs two things immediately:
  • A way of removing billions of tons of CO2 from the atmosphere each year.
  • A carbon tax big enough to make removing CO2 cheaper than extracting fossil fuels.
Neither looks likely.


Here is my list of things that should be done, with hope or without it.  They have the potential to delay or mitigate impending collapses.

Improve the health of all mothers

The late James P. Grant traveled the word teaching that reducing infant mortality reduces overall birth rates. He pioneered the State of the World's Children reports.  See also: http://www.unicef.org/mdg/maternal.html

Make contraception and other reproductive services freely available to all women

It is scandalous that millions of unwanted babies are born every year.

Impose a carbon tax and repeal the huge subsidies for fossil fuels

This is essential.  It should have been done 30 years ago.

End bottom trawling of the oceans


End the war on drugs

All you really need to know about this hopeless multi-trillion-dollar war can be found in the book, Undoing Drugs, by Daniel Benjamin. Written in 1993, it is just as applicable today. Googling "failed war on drugs" updates the mess.

Kill the F-35 bomber 

The F-35 is a trillion dollar mistake that has no chance of doing better than the F-16 and the A-10, the planes designed by John Boyd and his colleagues.  For background, see Boyd, The Fighter Pilot Who Changed the Art of War by Robert Coram.

Make the Pentagon accountable for its spending

I am a contributor to the Straus Military Reform Project. The Pentagon can not even be audited at present, much less pass an audit. This is a national scandal.


Here is our predicament as I see it:
  • Increasing population will accelerate the ongoing sixth mass extinction in earth's history.
  • The World3 model suggests the imminent collapse of civilization.
  • Global warming will accelerate the collapses of societies and ecosystems.
  • Market forces are driving the world's unsustainable actions.
  • There are no obvious forces powerful enough to counteract market forces. There seems to be no real alternative to business as usual.
  • Human ingenuity will be overwhelmed by the impeding crises.
  • Human history might turn out to be more favorable than this analysis suggests, but it could turn out to be less favorable. Collapse could happen sooner rather than later.
  • Mitigation seems hopeless, but we must carry on.
To my knowledge, no credible evidence contradicts these assertions, despite dogmatic and denialist claims to the contrary.

The essence of the situation can be summarized as the inability of the human race to understand the exponential function. Constant growth rate creates exponential actual growth. Human society, and yes, the human race itself, are doomed unless we can reduce the growth rate of both population and related industrial capital to below zero. Powerful forces make this unlikely.


It has been an interesting experience working on this part of my last lecture. It gives me no pleasure to confront horrible and likely possibilities. However, I have enjoyed the process of summarizing evidence in a dispassionate way. There is no use blaming individuals for our predicament. That would not be thinking in systems.

Peter Sinclair's comment, via private email
Obviously, I think about this kind of thing a lot. It keeps me up late at night, and wakes me in the morning to get back to work. Because I have children, I have a commitment not to give in to despair, but I do have moments where things do seem overwhelming.

I'm terribly concerned about the developing extinction crisis in the natural world, and have only become more so, since completing my recent video on the topic.  I interviewed Gerardo Ceballos, whose paper you mention.

I find it hard to imagine how we avoid a massive extinction without curbing population growth, but I don't see that population solution on the horizon, short of massive catastrophe. In fact, I've even read that in some studies, not even nuclear war or massive epidemics curb population enough to make a difference.

Perhaps I am hoping  that we may yet in our lifetime see some kind of enlightenment or turning point in human consciousness that might make survival possible - but spectacles such as the current election campaign certainly cast doubt on that possibility.

Nevertheless, I continue my work, in the hope that, if we awaken a critical - even if small - percentage of the populace, we can turn away from the worst impacts of industrial civilization.  I also still have hope for some technological improvements, for instance in energy production, i.e. solar etc, that may greatly reduce human impacts on the biosphere.

What I tell my children is that we are moving into a very dangerous bottleneck in planetary history, and that the earth is going to take a hit. Whether that is a 20 percent hit, or an 80 percent hit - is what the work of our lifetimes will decide.

Saturday, April 18, 2015

Leo 5.1 final released

Leo 5.1 final is now available at SourceForge. Leo is Open Software, using the MIT License.

The highlights of Leo 5.1

This release features @clean, one of the most important developments in Leo's history. @clean trees create external files without sentinel comments, yet Leo can update @clean trees from changes made to the corresponding external files. Steve Zatz explains why @clean changes everything.

More highlights
  • @clean trees preserve clone links and user attributes (uA's).
  • Reading @clean trees is faster than reading @auto or @shadow trees.
  • A new web page displays .leo files in the browser.
  • Added command history to Leo's minibuffer.
  • A new IdleTime class greatly simplifies idle-time handling.
  • Leo now honors @language inside @doc parts
  • @data nodes can be composed of their descendant nodes.
  • @shadow is now deprecated. @clean is superior to @shadow in all respects.

Mulder/Ream algorithm updates @clean trees.
Leo's home page
Tutorial videos
Leo's forum
Leo on Github
What people are saying about Leo
A web page that displays .leo files

Saturday, April 11, 2015

Leo 5.1 b2 released

Leo 5.1 b2 is now available at SourceForge.

b2 restores the proper operation of @nosent.  It is recommended for anyone using Leo 5.1 b1.

The highlights of Leo 5.1

This release features @clean, one of the most important developments in Leo's history.

@clean trees create external files without sentinel comments, yet Leo can update @clean trees from changes made to the corresponding external files, something long thought impossible. @clean trees preserve clone links and user attributes (uA's). Reading @clean trees is faster than reading @auto or @shadow trees. Steve Zatz explains why @clean changes everything.

More highlights
  • A new web page displays .leo files in the browser.
  • Added command history to Leo's minibuffer.
  • A new IdleTime class greatly simplifies idle-time handling.
  • Leo now honors @language inside @doc parts
  • @data nodes can be composed of their descendant nodes.
  • @shadow is now deprecated. @clean is superior to @shadow in all respects.

Mulder/Ream algorithm updates @clean trees from changes made in the corresponding external files.
Leo's home page
Tutorial videos
Leo's forum
Leo on Github
What people are saying about Leo
A web page that displays .leo files

Friday, March 27, 2015

Leo 5.1-b1 released

Leo 5.1 b1 is now available at SourceForge.

This release features @clean trees, one of the most important developments in Leo's history.

The highlights of Leo 5.1

@clean trees create external files without sentinel comments, yet Leo can update @clean trees from changes made to the corresponding external files, something long thought impossible.

@clean trees preserve clone links and user attributes (uA's). Reading @clean trees is faster than reading @auto or @shadow trees. Steve Zatz explains why @clean changes everything.
The Mulder/Ream algorithm updates @clean trees from changes made in the corresponding external files. This is a completely rewritten and much simpler version of Bernhard Mulder's original @shadow update algorithm.

More highlights
  • A new web page displays .leo files in the browser.
  • Added command history to Leo's minibuffer.
  • A new IdleTime class greatly simplifies idle-time handling.
  • Leo now honors @language inside @doc parts
  • @data nodes can be composed of their descendant nodes.
  • @int qt-cursor-width = 5 is great for geriatric eyes.