Tuesday, June 14, 2016

And Now For Something Completely Different

For a couple of years, this blog has been totally focused on my adventures with Python, Qt, and PyQt as I developed a major-ish app, PPQT2.

Well, with the latest round of fixes (see two prior posts) I believe that PPQT2 is about as done as it can be. I will leave the distribution files up indefinitely. But I don't intend to add any function, and I don't believe it contains any bugs serious enough to prevent its productive use. A couple of people are using PPQT2, and I wish them well. I am not among them, because I no longer spend any of my own time editing books for Distributed Proofreaders. The first PPQT was done to meet my own requirements for a book editor. The second one was done, really, to do the job right from a software engineering standpoint, and to meet the evolving needs of the Distributed Proofreading community. It succeeded in the first goal, but, based on limited uptake, failed to meet the second. So it goes.

Back to Authorship

A decade ago I wrote a book. Well, I've written several books over the years, mostly software. But that latest book was not about software. It was in a way, an extended answer to a question my mother asked me a few months before she died. I wrote the book and self-published it.

You can read about the book at its home website. In fact, you can read the whole book online there if you wish. Or you can follow links from there to buy your very own hard-copy for a very reasonable fee.

Don't do that!

Well, you can read it if you really want to. But don't order a copy (not that you were likely to do so, but still).

Why? Because I've embarked on a major revision, is why.

Adam Osborne is famous for killing his company by pre-announcing a better model before it was ready to ship. Sales of the existing Osborne computer dried up, choking off the cash flow that he needed to finish development. I'm not afraid of killing sales for my existing book because mostly, it has none to kill.

But why am I doing a revision? What improvements do I mean to make? How am I going about it?

Such questions will be the focus of forthcoming posts in this blog. In the meantime, if you want to get an early look, go to the new publishing site, slide the "You Pay" slider all the way to the left, to a cost of $0.00, and "buy" a copy of the book as it stands. If you give leanpub your email in that process, you'll get a notification as new versions are published, which is about monthly as I add chapters.

Friday, June 10, 2016

Fixing the Edit Menu Fix

A year and a half ago, I wrote an incomplete description of my solution for separate edit menus. While using PPQT2 in Windows 7 to test the logging/dictionary bug, I noticed a problem with this system and fixed it.

To review, under Qt there is a single menu bar that is owned by the main window. (The main window is a parentless widget based on the QMainWindow class.) In the menu bar there can be just one Edit menu.

In the Qt system, any menu is populated by QAction objects. In the Edit menu, if there is, say, a "Cut" item, it is installed in the menu by adding a QAction('Cut'). Any QAction has four properties:

  • A name string, like 'Cut'
  • An optional icon
  • An optional accelerator keystroke, like control-x
  • A "slot", which is an executable

The slot is the important item. It is a function that receives the "triggered" signal when the user selects that menu item or keys that accelerator.

This is a fine system, provided that there is no ambiguity about which slot to call. In an app with a simple UI, the one main window perhaps, the slot for File:Save or Edit:Cut is a method of the QMainWindow object, or of some sub-widget that it owns, and that never changes.

In PPQT2, there are multiple different widgets that can implement Edit actions. In the simplest case, the user has a single book open. But there is an Edit panel with the book text, a Notes panel with a separate simple document to edit, and a Words panel which supports copying selected words from a table. If the focus is in the Edit panel, we want the Edit:Cut action to be implemented by the cut() method of the QPlainTextEdit object that is part of that panel. But if the focus is in the Notes panel, we want Edit:Cut to be implemented by the cut() method that is part of the QPlainTextEdit object that is part of that panel. And if the focus is in the Words panel, we don't want to implement Edit:Cut at all, but we do want to support Edit:Copy via a method of the QTableView that implements the vocabulary table.

Worse: if the user has two or three books open, each book has its own Edit, Notes and Words panels. When the user hits control-x, it is important that the cut be implemented by the Edit or Notes panel for the book that is in front -- not by some panel containing the text of a different book that isn't visible.

The only solution is to continually change out the actions in the one-and-only Edit menu whenever the focus changes. When the Notes panel gets focus, it loads the Edit menu with the set of actions it supports, including slots that are methods of that unique Notes widget (distinguished by the "self" parameter). The code to do this, before this week, was as follows:

def set_up_edit_menu( action_list ) :
    global _EDIT_MENU, _LAST_MENU

    _EDIT_MENU.setEnabled(True)
    if id( action_list ) != _LAST_MENU :
        _LAST_MENU = id( action_list )
        _EDIT_MENU.clear()
        for (title, slot, key) in action_list :
            if title is None :
                _EDIT_MENU.addSeparator()
            else :
                _EDIT_MENU.addAction(title, slot, key)
def hide_edit_menu():
    global _EDIT_MENU
    _EDIT_MENU.setEnabled(False)

Any widget that wanted to support the edit menu had to do three things. It had to set up a list of Edit menu actions it supported, including the title, the slot, and a key code for each. It had to catch the focus-in event and call set_up_edit_menu() with that list. It had to catch the focus-out event and call hide_edit_menu(). As a result the Edit menu would be disabled (grayed-out) whenever focus was in a widget that didn't support it. When focus entered a widget that did support the Edit menu, it would be enabled and populated with that widget's actions, including references to that widget's slots.

I observed that it was quite frequent for a widget to get a focus-out followed by a focus-in, with no intervening visit to another widget. That's why the work of changing the menu is only done if the id() of the list is different. The Python id() of an object is basically its memory address. In my first description of this code, I was requiring the caller to pass a unique key, but I soon figured out that was not needed; id() did the job.

This all worked fine under Mac OS. Unfortunately I'd never really checked that it worked under Windows or Linux. This week I noticed that it did not. (imagine a blushing embarrassed emoji here)

In Windows and under Ubuntu Unity at least, the Edit menu was disabled, grayed-out, all the time! Why?!? The contents of the menu were changing properly, for example when focus was in the Edit panel, the menu contained the to-Upper and to-Lower actions that only the Edit panel supports. They went away when the focus moved to the Notes panel. So set_up_edit_menu() was being called. Which meant _Edit_MENU.setEnabled(True) was being executed.

It took a good half-hour of swearing and inserting print() statements before I realized that hide_edit_menu() was being called when I clicked on the word Edit in the menu bar. (imagine an open-mouth amazed emoji)

It seems that in Windows (and in Ubuntu, I later found out), a click anywhere in the menu bar causes a change of focus! You click in the Edit panel; the focus-in event calls set_up_edit_menu(). You click on the word Edit in the menu bar and the Edit panel gets a focus-out event and immediately calls hide_edit_menu(). It's a beautiful catch-22: the menu is enabled all the time until you try to use the menu, then it is disabled.

It took some pondering to figure out how to fix this. My solution was to ask, in hide_edit_menu(), if the current mouse location was within the global menu bar. If so, do not disable the menu. Here's the new code:

def hide_edit_menu():
    global _EDIT_MENU, _MENU_BAR
    
    if not C.PLATFORM_IS_MAC :
        relative_cursor_pos = _MENU_BAR.mapFromGlobal( QCursor.pos() )
        if _MENU_BAR.geometry().contains( relative_cursor_pos ) :
            return
    # either this is Mac OS, or the cursor is not over the menu bar
    _EDIT_MENU.setEnabled(False)

That made the Edit menu behave the way I wanted in all three systems.

A Logging problem in Windows -- and a pivot!

This is the first of a few posts about PPQT2, the hobby software project that's been the focus of the blog for two years. Then there will be a post announcing a turn to something completely different.

This week I had to deal with a reported bug in PPQT -- note, the first one in six months or more, which is more a reflection of its very low usage than on its code quality -- and found a couple other things that needed work.

Dictionary bug

The bug was present on all platforms, but was only visible on Windows. Here's the problem. The user opens a file that contains some non-Latin-1 characters, maybe some words in Greek. The default dictionary is set to en_US and the Greek words are not tagged for an alternate dict. So when the user refreshes the Words panel, every Greek word gets presented to a Hunspell object initialized with the en_US dict.

The encoding of any dict is specified in its name.aff file. The en_US dict is encoded ISO8859-1, and Hunspell expects any word checked against it to be encoded the same. Obviously a Greek word will have non-Latin characters, and Hunspell will correctly suffer an encoding error.

Not a problem! I was ready for this:

            try :
                return dict_to_use.spell(word)
            except UnicodeError as UE :
                dictionaries_logger.error("error encoding spelling word {}".format(word))
                return False

However, in Windows the call to the logger .error() itself raised an encoding error! Because the logged string contained the offending Greek word, and the log file had been opened using the default encoding which in Windows, was some wonky code page that couldn't encode Greek.

So there was an exception raised in the Except block, and the return False to indicate misspelling was never executed. But that did not cause a problem! Because the caller used the return value in an if statement, if check(word):. After Python displayed the traceback in the console window, it resumed execution in the calling function with the conventional default return value of None. None is just as False as False is, so everything worked, except for a ton of error messages in the console window.

The fix was to go back to the top-level module where I set up the logging handler for the whole program, and add encoding='UTF-8' to that call. A one-line fix and the error messages disappeared.

Thursday, February 4, 2016

byteplay ok on linux

Oh my but this was a strenuous afternoon. I believe I shall now drone on about my adventures, just because the story ends well.

First, I installed Ubuntu 14.04LTS on my 64-bit dev virtual machine. I kind of hated to do this because the install ISO does not offer what to a Mac user is the sensible choice, of reinstalling the OS without changing the user files. Nope. Either it will install "beside" the existing system, meaning in a different disk partition, or it installs "over" the existing system which wipes all your user files and settings. So I did the latter, knowing full well I would have to reinstall all my dev tools and dependencies.

In a month or two I will need to install Py/Qt5.6 and some Python modules, so I can upgrade PPQT and Cobro. But for the time being, I only needed to get a working Wing IDE and Python 3.5. The Wing IDE is a simple install, then activate with my license key.

Python.org does not offer an installable package for Linux. There are Ubuntu/Debian packages for Python 2.7 and for Python 3.4, but I needed 3.5. The only way to get that is by downloading the source package and making it. Well, I've done that often enough. Download, unzip, make, make test, sudo make install. All seems to go well until I try to start Wing, and it can't start Python. "Missing module _struct".

DuckDuck that phrase and you will find that hundreds of people have encountered it. What does it mean? It can be properly translated as, "you suck at installing software". Something didn't go well in the make-install step. More missing things show up when I find out that neither pip nor easy-install were installed. When I try to download and run get-pip.py, it fails with an error because the SSL lib wasn't installed, and also the zlib was not installed.

Scrolling back in the make output I find a list of modules that it couldn't locate. Did it put the list at the end where a person would notice it? Of course not! It buried it between a couple of hundred lines of output before and after. Did it tell you what to do? Of course not! Well, it did; it helpfully advised that you "look for the name of the missing module in setup.py".

Which I did, and saw where it built a list of libraries to search for things. Then I got a terminal window and used the find command to find the things it was not finding. Oh, there they are, in a special 64-bit library. Add the path to that library to the code of setup.py and rerun the make install. OK then! Now I've got pip, and Wing is working, whee.

So now I can attempt what I set out to do a couple hours earlier, pip install byteplay3. Which of course fails with an obscure message, "Could not find a version that satisfies the requirement". What requirement? Back to DuckDuckGo. Oh yeah, plenty of people were having this problem a couple years ago, when pip began to require version numbers that complied with PEP400. I read PEP400. No, my version number fits the pattern. But in one of these postings I see a remark that, oh, also, pip does not support modules that are not hosted directly at PyPi. A module hosted at github, for example. Oh. Which I am doing.

So I find out about setup.py's build and sdist and upload commands. And find out the hard way, that you can't do

python setup.py sdist
python setup.py upload

oh no no no! You have to do

python setup.py sdist upload

all one line. Because...? Who knows.

Anyway, did that, and then in my Linux system and my Mac OS system both I could do pip install byteplay3 and it did it and I could start Python and run the example from the readme, and it worked.

Tomorrow, Windows. I am so looking forward to that.

Wednesday, February 3, 2016

On the treadmill again

So part of supporting byteplay3, which I have optimistically uploaded to PyPi, is to check that it installs on linux and on Windows. So I fired up Parallels and opened my 64-bit Ubuntu dev machine.

Now, I do this not very often, so it seems inevitable that every time I do it, I have in the interim updated Parallels itself, and every time Parallels updates, it wants to update the Parallels Tools component it inserts into every virtual machine.

So of course as soon as I open the Ubuntu dev VM, it starts installing a new Parallels Tools. Which for some reason takes 5 minutes to crawl its progress bar across, and then wants a reboot.

Meanwhile I thought, hey, let's make sure Ubuntu is up to snuff also. I bring up the Synaptic package manager. Current Ubuntu has a much simpler "Software Updater" program; and I know that the hard-core Linux user doesn't want any of that GUI shit, nothing but sudo apt-get on the command line will do. But me, I remember the SGI package manager for IRIX (but can't remember its name, though) and Synaptic is a dead ringer for it. So I use Synaptic.

I tell it, reload your info and it says it is reloading but then it says it couldn't find a bunch of repositories. It marks a bunch of upgrades but when I tell it to apply them, it grinds a while and says it couldn't download the packages. 404's everywhere.

imagine an hour or so of increasingly irritable investigation here...

Long story short, at some time in the near past, I unwisely and unwittingly approved an upgrade from Ubuntu 14.04 LTS, to Ubuntu 14.10. Probably seemed like a reasonable thing to do at the time, you know? Minor point-release upgrade? But a bad, bad move on my part!

What was making Synaptic (and the Software Updater) fail flat was that their apt sources-list file specified "utopic" release in the repositories. Because, you see, I had upgraded to 14.10 Utopic Unicorn. Unfortunately, 14.10 Utopic was end-of-lifed back last July and all the "utopic" repositories were pulled from Ubuntu and Canonical. So that's why the 404's—the repositories are all gone.

Wait, what? I thought that Ubuntu 14 was a long-term support version!

No no no, you foolish person. Ubuntu 14.04 Titillated Tarsier, that was the LTS release.

When I stupidly approved an upgrade to 14.10, I moved away from the LTS system and once more stepped onto the every-six-months upgrade treadmill. I cannot apply maintenance to 14.10, I must upgrade to 15.04 Violent Vixen, and very soon again to 15.10 Willful Wildebeast, as per this diagram.

Alternatively I can find a DVD-ROM image of 14.04 and install it over my system to force a downgrade, which I may just do.

Meanwhile, fortunately, the 64-bit test system (without Python or Qt) is still on 14.04, as are, I hope the two 32-bit systems.

Meanwhile, less fortunately, the Windows 7 dev system seems to have forgotten how to run Pip-Win. It looks as if it is starting Python 3 but with the Python 2.7 lib. So some stupid error in the PATH or other environment variable. So I have that to look forward to, tomorrow.

Meanwhile again, I applied the 15.14 update and guess what? The dev system no longer remembers that it is a 1440-px wide window. Lost all the window sizes except 800. Another config file to remember the name of and edit.

Tuesday, February 2, 2016

Function signatures; bytecode lacks "computed go-to"

Well, sure, there was a bug. A good thing I blogged about it, or who knows it would have gone untested and unfound?

The Code class represents a code object, but in a form that can be manipulated. It has two crucial methods. from_code() accepts a Python code object and captures all its contents, returning a new instance of Code class. I had properly updated that method to notice the Python 3 features of varkwargs and the kwonlyargcount.

I had not completely dealt with these in the other method, to_code(). Calling to_code() of a Code object returns a code object that is supposedly equivalent. But I had not included the kwonlyargcount in the calculation of the code.co_argcount. So it was off by 1, causing a TypeError exception, wrong number of arguments, when you called the code.

Testing for that also revealed a bug in my unit-test scaffold code. But it's all good now.

Tiny Basic and the computed go-to

In the first post in this series, I mentioned that one use of byteplay would be to implement domain-specific languages using Python bytecode as a the implementation language. And I suggested I might try to implement a Tiny Basic compiler in this manner. Turns out? Not so much.

Historic background: Tiny BASIC was the term for several, minimal BASIC interpreters for early microcomputers. The first was written by Tom Pittman of Itty Bitty Computers. The more influential version—because it was for the 8080 where Pittman's was for the 1802—was by Li-Chen Wang. I was aware of Tiny BASIC although I never used it. (I programmed my Z80-based CP/M system in assembler, thank you.)

Anyway, Tiny BASIC is such a minimal language that its interpreter, including an adequate editor, can be implemented in a few kilobytes. On a walk yesterday I thought about it and at first got rather excited about the possibility. I quickly arrived at a program structure (a dict keyed by the line number with the text of the line as value) and visualized how I could use the built-in compile() function to reduce expressions to bytecode, and glue those bytecode bits together with more bytecode and thus produce a whole Python function from a BASIC program.

Then, sitting in a coffee shop, I used my phone to look up the syntax of the language...

...and is it not a fabulous age we are living in, where one can be sitting over a capuccino and have the passing notion, "I wonder what was the syntax of a programming language last used over thirty years ago?" and be reading the answer in less time than it takes to describe it? Seriously, people, why are we not all happy as kings?

So, here's the manual. Less than five minutes of skimming on the little screen of the phone revealed a major, major problem.

The problem is that Python bytecode has no real "jump" instruction as found in an assembly language. It has several jump opcodes, for example POP_JUMP_IF_TRUE and JUMP_ABSOLUTE, but the argument to all of these is a fixed integer offset in the bytecode string. The amount to jump forward or backward, or the absolute offset to jump to, is hard-coded in the instruction. There is no way in bytecode language to say, jump to the offset encoded in the top-of-stack item.

Without such a "computed go-to" it becomes much more difficult to implement Tiny BASIC. Because Tiny BASIC has both a computed GOTO and a GOSUB. Of the GOTO, Pittman's manual explicitly says, "the next statement to be executed after a GOTO has the line number derived by the evaluation of the expression in the GOTO statement. Note that this permits you to compute the line number of the next statement on the basis of program parameters during program execution. (my emphasis)". The GOSUB presents the same problem in two ways. First, it takes an expression, so the actual target is determined at execution time. Second, it stores the return location on a stack for use by the RETURN statement. RETURN is effectively a GOTO where the destination is fetched from the call stack.

None of these things can be implemented using a bytecode jump. So that pretty well ends any thought of compiling a Tiny BASIC source file into a single Python function.

I did think of a complicated mechanism, basically each line of the BASIC program would compile into a separate function that could operate on globals (the BASIC variables) and must return the number of the next line to be executed, with None meaning, "next sequential". But it would be kind of ugly and clumsy and not a good demo of byteplay3.

Monday, February 1, 2016

Byteplay3 uploaded to pypi

The PyPi registration process turned out to be remarkably easy. I had to register as a user, something I'd never done despite hundreds of uses of pypi over the years. Then it was just python setup.py register and that was it.

It took a little while to show up in a search at PyPi, but here it is now. I do not understand why the descriptive paragraphs (which the PyPi "edit package" page clearly says, can use reStructured Text markup) are not formatted right. The rST markup for the example and the headings is just being ignored. Meh. Don't care.

What I did to Noam's code

Heh heh quite a lot, really. For one thing, I commented the bejeezus out of it. That was how I went about understanding it (and it took some understanding, lemme tell ya). I read it (and read it and read it) and then I commented what it was doing. Initially I peppered it with comments like #TODO what the bleep is this for? and the like, then gradually replaced all those with meaningful comments.

Prior to and during this work, I was watching Philip Guo's lectures on Python internals. Although these are based on Python 2.7, he leads you through the modules of CPython that are involved with compiling and executing code. So part of the experience of making byteplay work in Python 3 was discovering what about CPython internals had changed from 2.7 to 3.4. The changes are mostly cosmetic, for instance the names of attributes of the function object are different. A few bytecodes have been added and deleted. But the concepts are pretty consistent.

Function Signatures

Some significant changes involve the code object and its handling of the function signature. I'm not sure if the *args and **kwargs features are Python 3 innovations? But the Python 3 code object has properties to deal with them that are not in Python 2.

If the signature of a function has some arguments, then *args, then another argument:

def foo( a, b=False, *args, z=True ):

then the argument z is a keyword-only argument. It cannot be entered as a positional value; it must be specified with z=:

res = foo( 1, True, 2,3,4, z='z' )

The code object attribute kwonlyargcount is the count of such arguments. If the function has such arguments, it is nonzero. This attribute wasn't present in Python 2, and I had to add it to the Code object that is the centerpiece of byteplay.

Byteplay2 did know about the varargs attribute of a code object, a flag bit to say if it has a *args in its signature. (So, that's from Python 2.) However, Python 3 has added a varkwargs flag. That also I had to incorporate into the Code class.

All of these affect the interpretation of other parts of the code object. One crucial thing in a code object is a tuple, varnames, composed of the names (strings) of all local variables. The sequence of names in this tuple is:

  • Names of normal arguments, ones that precede a *args or **kwargs item, in their declared order. From the example above, a and b.
  • Names of keyword-only arguments, if any. In the example, z
  • Name of a *args argument, if any (it doesn't have to be args).
  • Name of a **kwargs argument, if any.
  • Names of other local variables.

The varnames tuple is rather important because bytecodes that reference a local, such as STORE_FAST, have as their argument an index to this tuple. But the composition of the tuple is not drop-dead simple. The Code object method from_code() takes one apart and stores the pieces; and the to_code() method assembles the pieces to recreate a code object. And that had to change a bit for Python 3.

And now I've reminded myself, that my unit test suite lacks a test of function signatures with *args and **kwargs. I think I shall go and do one now...