Converting Leo to Python 3K: surprises

This post will conclude the discussion of how I converted Leo to Python 3k. In fact, the work was mostly complete several weeks ago. Vacation got in the way of writing up these notes.

The biggest surprise was that almost all the surprises were good surprises. Aside from the complications previously discussed, porting Leo was a matter of running Leo and fixing the obvious problems reported by Python itself.

To recap, it took just a few days to get Leo to the point at which Leo could run its own unit tests with Python 3k. After that, it took just a few more hours, spread over several days, to get all the unit tests to pass.

Yesterday I completed the one and only "tricky" part of the port. This was a library problem. I won't bore you with the details: the problem arose from a mis-reading of the Python 3k docs. Today, as I was writing this post, I discovered my mistake and was able to clean up an ugly workaround.

So that's it. Porting a major app like Leo from Python 2.x to Python 3.x was almost completely straightforward.

Edward

Coverting Leo to Python 3k: tactics

In this posting I'll discuss how to create code that will compile on both Python 2.x and Python 3.x. This is mostly straightforward, but there are some interesting tactical details.

1. The g.isPython3 constant

Leo uses the toPython3 constant to distinguish between old (Python 2.x) and new (Python 3.x) code:

   isPython3 = sys.version_info >= (3,0,0)

This constant is defined in only one place, in Leo's leoGlobals module, known as 'g' throughout Leo's code. All of Leo's modules import this module before any other imports as follows:

   import leo.core.leoGlobals as g

Leo distinguishes old and new code this way:

   if g.isPython3:
       << old code >>
   else:
       << new code >>

Naturally, we want to keep this kind of code to a minimum by refactoring common instances of this pattern into new functions. In Leo, the obvious place for these new functions is in g, the leoGlobals module.

2. Print statements and g.pr

Print statements with a trailing comma pose a special problem. Such statements are syntactically invalid in Python 3.x, so there is *no way* to use the pattern above. Late last year I converted almost all of Leo's print statements to the g.pr function. This conversion was the single biggest change to Leo's code base for Python 3.x.

Theoretically, I only needed to eliminate print statements with trailing commas, but even "normal" print statements pose problems. In Python 3.x, 'print(a,b)' produces the same output as 'print a,b' produces in Python 2.x. But in Python 2.x, 'print(a,b)' prints the *tuple* (a,b).

Yes, there is a workaround: to translate 'print a,b' to `print('%s %s' % (a,b))'. But I thought this was too ugly. Besides, defining g.pr gave me the opportunity to use a clever convention that supports automatic translation of some, but not all arguments. The convention is just this: g.pr translates only *odd* arguments. For example:

   g.pr(a,b,c,d,e)

prints the translation of a, c and e, but prints b and d as they are, untranslated. To suppress the translation of the first argument, use an empty string as the first argument:

   g.pr('',a)

Imo, this convention is much better than the typical convention of using _(a) to denote the translation of a. Besides being cleaner visually, it is often exactly what is desired anyway. For example:

   g.pr('can not open', fileName)

does exactly the right thing: we want to translate 'can not open' but we should not translate fileName!

In short, removing print statements from Leo makes a virtue out of necessity. A few print statements do remain in Leo, but they are just for debugging and are seldom used. It doesn't matter whether those statements produce the same results in Python 3.x as they do in Python 2.x.

One more detail. There is no way to use 'print' in the g.pr function so that it compiles on both Python 2.x and Python 3.x. Instead, the g.pr function uses sys.stdout.write. Alas, sys.stdout.write is not precisely equivalent to print in all situations, but this can not be helped.

3. Unicode constants

Unicode constants of the form u'whatever' are invalid in Python 3.x. The workaround is code such as the following:

   if isPython3:
       f = str
   else:
       f = unicode
   aConstant = f('whatever')

This kind of code is common enough to have been refactored:

   aConstant = g.toUnicode('whatever')

I'll be saying more about unicode issues in later posts.

Coverting Leo to Python 3k: strategy

In the first post of this series, I stated the goal of this project: to create a common code base that will run Leo on both Python 2.x and Python 3.x. I am confident this goal will be accomplished because the following 3-step strategy guarantees success:

Step 1: Create a common code base that compiles on both Python 2.6 and Python 3.1. I completed this step yesterday, although one or two minor issues remain. Leo continues to run with Python 2.6 as it always has, and all of Leo's existing unit tests pass with the new code base when run with Python 2.6.

Step 2: Run the new code using Python 3.1, fixing bugs as they are found, until Leo executes its startup code without crashing. I completed this step yesterday as well.

Step 3: Run all of Leo's unit tests using Python 3.1. When this step is complete, there will be little or nothing left to do: Leo's unit tests cover almost all the essential features of Leo, except for some gui-related issues that already appear to work properly.

This strategy is guaranteed to work: porting Leo to Python 3.x is a low-risk project.

Only tactical details remain. Most details need no further comments, but I'll discuss two tactical issues in further posts:

A. How Leo's code can be made to compile in Python 2.6/3.1.

B. How Leo handles unicode in both Python 2.6/3.1.

The second is more important theoretically, the first more difficult practically.

P.S. A word or two about timing. Leo depends on Qt, so I had to wait until Qt itself supported Python 3.x before I could attempt to finish steps 2 and 3 above. PyQt supported Python 3.x at version 4.6, and it is now at version 4.6.2, so I can be pretty confident that Qt will handle Python 3.x pretty well.

I restarted this project yesterday after reading a comment praising Python 3.x. My guess is that 2010 will be the year the Python world moves decisively to Python 3.x.

Coverting Leo to Python 3k: goals

This is the first in a series of posts that will describe how I am converting my Leo app to run on Python 3k. I will be writing these posts before the project is complete so that the details are fresh in my mind. But the intention is to say something that will be generally useful to anyone contemplating a similar project: I'll keep Leo-specific details to a minimum. My emphasis will be on strategy and tactics, not code-level details.

The one and only goal of this project is straightforward: the final product will be code that compiles and runs on both Python2k and Python3k without any modification whatsoever. A common code base is an absolute requirement, for two reasons:

1. Leo is under active development using bzr. It would be intolerable to attempt to support multiple code bases for any length of time.

2. Python's 2to3 code-conversion tool does not begin to have the sophistication needed to automatically convert Leo's code. But even if 2to3 did have the smarts, it would odious to add an intermediate code-conversion step every time anyone changed Leo's code base.

This goal is just a bit ambitious, but I have absolute confidence that it can be accomplished. I'll tell you why in the next post.

Announcing Leo 4.6.2 final

Leo 4.6.2 final is now available here. For more info, see Leo's web site.


The highlights of Leo 4.6
--------------------------
- Cached external files *greatly* reduces the time to load .leo files.
- Leo supports both the Tk and Qt gui interfaces.
- New --config, --file and --gui command-line options.
- Leo tests syntax of .py files when saving them.
- Leo can now open any kind of file into @edit nodes.
- @auto-rst nodes allow easy editing of reStructuredText files.
- Properties of commanders, positions and nodes simplify programming.
- Improved Leo's unit testing framework.
- Leo now requires Python 2.5 or later.

Better Python logging

This is a response to
http://sayspy.blogspot.com/2009/07/results-of-informal-poll-about-standard.html

Back when Python's logging module was being designed, I asserted that there is a much better way to enable and disable logs than a linear sequence of logging levels. That proposal went nowhere, perhaps because people heard it as dismissing the whole notion of having a standard logging module.

In fact, though, it would be easy to graft a small extension on to Python's logging module that would make logging much more flexible. Here's how

1. Add another logging level, say 'custom'. When custom logging is in effect, log messages are generated if they match a **tracepoint name**. Such names are simply strings.

2. Add the following method to the logging module::

logging.custom("abc",'This a custom message logged when "abc" is enabled.')

3. Enable or disable custom tracepoint names with::

logging.enable_custom(pattern1,pattern2,...)

logging.disable_custom(pattern,pattern2,...)

Each patten is a string, possibly containing wildcard characters '*' and '?', that matches zero or more tracepoint names. The enable/disable_custom methods simply add or delete entries in a **pattern database** of enable and disabled names.

When logging.custom(s,message) is executed, it searches the pattern database for a match with the string s. It prints the message to the log if a match is indeed found.

That's about it. It's easy to do, compatible with the present logging system, but much more flexible.

Many small tweaks to this scheme are possible, but I'll save those in case there is some real interest.

The Age of Entanglement

Here is a quote from the preface of "The Age of Entanglement", by Louisa Gilder,
http://www.amazon.com/Age-Entanglement-Quantum-Physics-Reborn/dp/1400044170
one of the few books I have read lately from start to finish:

QQQ
"Science rests of experiments," wrote Heisenberg, but "science is rooted in conversations."

Nothing could be further from the impression physics textbooks give to students. There, physics seems to be a perfect sculpture sitting in a vacuum-sealed case, as if brains, only tenuously connected to bodies, had given birth to insights fully formed...

Physics, in actuality, is a never-ending search made by human beings...The schoolbook simplifications obscure the crooked, strange and fascinating paths that stretch out from each idea, not only back into the past but onwards into the future...

Conversations are essential to science. But the off-the-cuff nature of conversation poses a difficulty. It is rare, even in these digital times, to have a complete transcript of every word spoken between two people on a given day, even if that conversation someday leads to a new understanding of the world. The result is that history books rarely have much of the to-and-fro of human interaction. Heisenberg's statement suggests that something is therefore lost.
QQQ

Being part of the Python world via its many conversations is one of my life's great joys.