Tuesday, June 30, 2015

All Built, Waiting to Announce

I ran the PyInstaller builds on all four platforms, Mac OS, Windows 7 32-bit, and Ubuntu 32- and 64-bit. The process in each case is

  • Build the app and run it for a sanity check
  • create the PPQT2 folder containing the app, the README, the COPYING.TXT, and the extras folder
  • zip that folder with output to ~/Dropbox/Public/PPQT2
  • do something else while Dropbox loads the ~50MB file to the cloud
  • suspend that development VM
  • activate the test VM of the same platform (identical OS but no Python or Qt installed)
  • do something else while Dropbox updates that VM's copy of /Public
  • copy the zip file to the local desktop and unzip it
  • take a deep breath and hold it
  • run the app and exhale Yessssss! when it comes up and all features work.

Repeat for the next VM.

I was relieved to find that a 32-bit Win7 app runs just fine on a 64-bit Win7 installation. I was also relieved that the dev version of PyInstaller for Python3 worked perfectly in all three OS's. This is such a change from just a few months ago when I went through several weeks of agony trying different ways to bundle an app, including cxFreeze, pyqtdeploy, and nuitka.

What changed everything was that Hartmut Goebel and the other key maintainers of PyInstaller suddenly returned to activity with a long (and continuing) flurry of updates and fixes. The long-idle Python3 fork was merged to the mainline and got some key updates, and everything got good again. I feel a lot of gratitude toward those guys.

So now what for PPQT2? Right now I am waiting for the right moment to announce its availability on the DP forum. I think that will be Wednesday night; then I will be available to respond to forum comments promptly for four days straight. Meantime I am thinking hard about how to announce it, especially how to convey the importance of third-party contribution of Translators.

I have a couple of enhancements still to code. I promised I would write an ASCII Translator, just because I don't want my Knuth-Pratt paragraph justification code to be lost. And of course there will be some bug reports to deal with. But with any luck this project will be finito by the end of July.

Saturday, June 27, 2015

Major oops! revealed in windows test

So I had intended to release PPQT2 on Windows 7 64-bit only. But a query from an alpha user made me rethink that, and I decided better to build and release it on Windows 7 32-bit, which I trust and hope will work also on a 64-bit system. Unlike Linux, where I have to create versions for both widths.

Anyway, that meant setting up a 32-bit Win7 system. A couple weeks ago I went on eBay and bought a Win7 Professional 32-bit DVD. And installed it in a VM and provisioned it for development -- a process I meant to blog about today. But...

When I got to the point where I could run PPQT from source, I did so, and did a superficial checkout of various features. And immediately noticed that the Edit panel appeared to be using a non-monospaced font, probably Arial or whatever the Win7 default font is. Opened the preferences dialog, looked at the Edit Font choice popup menu. There were my expected Liberation Mono and Cousine font choices, along with several mono fonts from the local environment, like Consolas and Courier New. But selecting either Liberation Mono or Cousine from the menu and clicking Apply had no effect. The Edit panel remained showing Arial. Selecting a local font like Consolas did have immediate effect, so the whole mechanism of choosing and applying a font was working. But the two preferred fonts were not.

Hmmmm.

They were working perfectly well on Mac OS and Linux.

Weren't they?

Well, they seem to work.

To review, these two fonts are good looking mono fonts that both have a very wide repertoire of Unicode glyphs. I want them available to all PPQT users. To make that happen, I carefully built a resource file naming them, along with some minor graphic icon pngs, and compiled it with pyrcc5 into a Python file, resources.py. As a result, this code at the head of fonts.py:

_FONT_DB = QFontDatabase()
_FONT_DB.addApplicationFont(':/liberation_mono.ttf')
_FONT_DB.addApplicationFont(':/cousine.ttf')

...should prepare a local font database that includes all locally-available fonts plus those two, loaded from the Qt resources. They should then show up in the following code which prepares the list of families from which the "choose an edit font" popup is built.

# Return a list of available monospaced families for use in the preferences
# dialog. Because for some stupid reason the font database refuses to
# acknowledge that liberation mono and cousine are in fact, monospaced,
# insert those names too.

def list_of_good_families():
    selection = _FONT_DB.families() # all known family names
    short_list = [family for family in selection if _FONT_DB.isFixedPitch(family) ]
    short_list.insert(0,'Liberation Mono')
    short_list.insert(0,'Cousine')
    return short_list

Read the comments... I think I may know what that "stupid reason" was, now. Because, although those two family names were in the menu, choosing them had no effect. The Qt font database is carefully designed so that it always returns a font when you ask for one, falling back to some default if it can't give you what you ask for. Clearly in Windows, it couldn't provide the "Cousine" family and was falling back to the system default.

But not in Mac OS or Linux...

But you know, in both those dev systems, I believe I had installed both fonts locally...

Could it be that the font database had never been supplying those fonts from the resources module?

OK, so where did I import that module? I apply Wing IDE's "search in files" widget to look for "resources" in all project files.

I never did import it. I built the ****ing resources file and never imported it to the running app!

Added one line, import resources, to PPQT2.py and reran the app. Bingo! I can now set Liberation Mono or Cousine as Edit fonts!

It's a bit of a mystery how I managed to get along so far without encountering any problem from the missing resources. But I don't care. Just glad to have found it before shipping.

Thursday, June 25, 2015

A tough bug

So before I ship PPQT for real, I thought I better take one more look at an elusive problem. I had noticed that sometimes, keying ^f in the Edit panel did not bring the Find panel to the front as it is supposed to do. But the behavior seemed to be intermittent.

Recall that PPQT has on the left, a tabset of Edit panels, one per open document. And on the right, a tabset of various functional Panels like the Notes panel, Images panel, and Find panel. Each open document has its own set of function panels. So when you switch to a different document's Edit panel, the right half of the window is repopulated with the function panels for that document.

So what would happen is this. Say there are two documents open and document A is active (its Edit panel is visible). And on the right, the Images panel is displaying a scanned page from A. The user is typing and wants to find something, so keys ^f. What should, and usually does, happen is that the Find panel replaces the Images panel and the cursor focus moves to the Find text field.

What sometimes happened was that instead, the Images panel remained, but a narrow blue keyboard-focus rectangle appeared over it, outlining the position of the Find text field on the (invisible) Find panel.

It took a half-hour of repeated experiments to work out the failing conditions. There had to be at least two documents open. You had to switch between them in a certain order.

Internally, what I supposed was happening was that the Edit panel was trapping the ^f keystroke in its key event handler, and immediately issuing a signal named editKeyEvent, which was connected to a slot in the Find panel code. Using a breakpoint I verified that control got to the Find panel, and that it would call into a main window function to get itself displayed. The main window code is simple,

    def make_tab_visible(self, tabwidg):
        ix = self.panel_tabset.indexOf(tabwidg)
        if ix >= 0 : # widget exists in this tabset
            self.panel_tabset.setCurrentIndex(ix)
            return
        mainwindow_logger.error('Request to show nonexistent widget')

It asks the active tabset if it knows this widget; if it does, please make it active. What was happening was that the Find panel making this call was not the Find panel in the active tabset. It was the Find panel widget of the other document.

Wut?

This sent me off on a long tail-chase on the signal/slot setup. Somehow, I thought, the Edit panel for one book must have connected its ^f keystroke event signal to the wrong Find panel. But that code was all solid.

In the course of investigating the hook-up of the Edit signal to the Find panel, I noticed that there was another way for that signal to be emitted. It might come from the keyPressEvent handler of the edit panel. But it might also come from the Find... action of the Edit menu. Oho!

Months ago I implemented variable Edit menus. Each Edit panel has its own Edit menu, in which the menu Action for Find... (with a ^F keyboard accelerator option) was hooked to call a little function that emitted that Edit panel's signal to that Edit panel's Find panel. The Word panel and Notes panel also have their own Edit menus. This was all to get around Qt problems that plagued version 1, where different panels feuded over the ownership of the single global Edit menu. Now the Words panel has its own Edit menu that does things that relate to Words panel facilities, etc.

When any of these panels get a focusIn Event, they immediately call the main window asking it to put up their own custom Edit menu. When I implemented this, I put in a little gimmick to save some time.

_LAST_KEY = None
def set_up_edit_menu(key, action_list) :
    global _EDIT_MENU, _LAST_KEY
    if key != _LAST_KEY :
        _LAST_KEY = key
        _EDIT_MENU.clear()
        # code to populate the Edit menu from action_list
    _EDIT_MENU.setEnabled(True)

Each caller would pass a simple letter key, and the main window could avoid clearing and populating the menu when, as often happens, focus goes in and out of the same panel over and over.

The problem was, every Edit panel called with a key of 'E'. Ooops! That meant that when the focus moved from one document's Edit panel to another's—without a stop between in some other panel that had an Edit menu—the Edit menu would not be repopulated. It would still have the Actions defined by the first document. And that included a signal to the first document's Find panel when ^f was keyed or when Edit>Find was selected. So the signal would go to the wrong Find panel widget. It would see it wasn't visible, so it would call the main window; the main window would not find that widget in the current tabset; no Find panel would be displayed; but the Find panel would then call for keyboard focus to its Find text field, resulting in a focus rectangle over the top of the Images panel.

The fix was to make the key be something unique to the caller, and Python supplies a unique id via the id() built-in function. For a bonus, the callers no longer have to provide that key as an argument.

def set_up_edit_menu(action_list) :
    global _EDIT_MENU, _LAST_MENU
    if id(action_list) != _LAST_MENU :
        _LAST_MENU = id(action_list)
        _EDIT_MENU.clear()
        # populate the Edit menu from action_list
    _EDIT_MENU.setEnabled(True)

Wednesday, June 24, 2015

Learning about callable types

Another technique I used in the HTML translator is the Python equivalent of a "computed go-to". The Translator API passes a series of "events" and any translator has to deal with them something like this:

    for (code, text, stuff, lnum) in event_generator :
        # deal with this event

There are 34 possible event codes. A naive way of "dealing with this event" would be to write an if..elif..elif stack 34 items high. If you put the most frequent codes at the top this would not be too bad in performance, but it makes for rather unwieldy code to edit. A better way is to to have a dict in which the keys are the 34 code values, and the values are the actions to be performed for a given key. I've provided a skeleton of a Translator module with code like this:

    actions = {
        XU.Events.LINE          : None,
        XU.Events.OPEN_PARA     : None,
        XU.Events.CLOSE_PARA    : None,
...
        XU.Events.PAGE_BREAK    : note_page_break ,
...
        XU.Events.CLOSE_TABLE   : "</table>" ,
        }

    for (code, text, stuff, lnum) in event_generator :
        action = actions[ code ]
        if action : # is not None or null string,
            if isinstance( action, str ) :
                BODY << action # write string literal
            else :
                action() # call the callable
        # else do nothing

In this version, items in the dict can be one of three things:

  • None or a null string, meaning either "do nothing" or "not implemented yet"
  • A string literal to be copied to the output file immediately
  • A reference to a callable which will do something more involved, perhaps formatting the text value in some way before writing it.

(The callable functions named in this action dict were the "flock of little helper functions" that I referred to in yesterday's post—the ones that needed access to variables initialized by their parent function, and which had to become globals due to Python's eccentric scoping rules.)

For one part of the HTML Translator, I had a similar action dict but in it I wanted to have four possible types of actions:

  • None, meaning "do nothing"
  • A string literal to be copied to the output file immediately
  • A reference to a callable that would do something more involved such as setting or clearing a status flag.
  • A lambda that yielded a string value based on the text value but didn't actually contain a file-output call.

No problem, thought I. The above loop logic can easily be stretched to deal with this along these lines:

   for (code, text, stuff, lnum) in event_generator :
        action = actions[ code ]
        if action : # is not None or null string,
            if isinstance( action, str ) :
                BODY << action # write string literal
            elif type(action) == types.LambdaType :
                BODY << action() # invoke lambda, write its value
            else : # type(action) == FunctionType
                action() # call the callable
        # else do nothing

Surprise! It didn't work. Why? Turns out, although the types module has distinct names FunctionType and LambdaType, they are equal. You cannot distinguish between a reference to a lambda and a reference to a function based on type().

That kinda makes sense, in that we are told repeatedly that a lambda is just shorthand for an anonymous function. But it would have been handy to tell the difference.

In the end, for this part of the code (it was not the main translate loop but one with fewer possible codes) I made each value of the action dict a tuple ('f', funct_name) or ('s','literal string'). That allowed me to distinguish between a lambda that generated a string, and a function that didn't. But the whole thing felt like a kludge and I believe I will go back and recode that particular loop as an if/elif stack instead.

Tuesday, June 23, 2015

Learning about Python scoping

Variable scoping is a big topic in some programming languages, or was. "Scoping" refers to the rules about how the system resolves a variable name to its value. Here's an example of the problem I ran into while writing the HTML translator.

def parent():
    par_var = True
    def child():
        if par_var :
            print('yes')
        else :
            print('no')
    child()

When you execute parent(), what happens? Specifically, how is the child function's reference to par_var resolved to a value? My naive expectation was that Python would look first in the scope of child(), and fail; then look in the scope of parent(), and succeed, finding a value of True and printing "yes". Which it does! Expectations confirmed! But make this small change:

def parent():
    par_var = True
    def child():
        if par_var :
            print('yes')
            par_var = False
        else :
            print('no')
            par_var = True
    child()
    child()
    child()

What do you think? If you execute parent(), will it perhaps print yes, no, and yes? Nope. It will not run at all! It will immediately terminate with an error, "builtins.UnboundLocalError: local variable 'par_var' referenced before assignment".

What!?!

In the first example, Python had no problem resolving the child's reference to the parent's variable. But with this small change—assignment of values to par_var—the scoping rule changed. Now Python searches in the scope of child(), fails, and stops looking. It doesn't look out in the enclosing scope.

A little research turns up agreement that yes, this is the rule: if a variable is assigned a value in the scope of a function, Python assumes (in fact, insists) that the variable is local to that function. The only exception is if you specifically include a global statement. So let's do that:

def parent():
    par_var = True
    def child():
        global par_var
        if par_var :
            print('yes')
            par_var = False
        else :
            print('no')
            par_var = True
    child()
    child()
    child()

Does it work now? Nunh-unh. "builtins.NameError: name 'par_var' is not defined". The global statement modifies the scoping rule, all right. But it does not simply say, "global to me"; it says "global in the sense of being at the top level of this namespace/module". Which it is not, in that example. The only way to make the above code work is to move the first assignment to par_var outside the body of the parent function.

So in Python, it is possible to have a variable that is "relatively global"—not local but declared in some a containing scope—but only if that variable is read-only. As soon as the inner function attempts assignment, the variable must be either purely local, or purely global.

This is kind of wacky. It made me have to revise a bunch of code I'd written, where a parent function declared a whole batch of little helper child functions, and shared the use of the parent's variables. All the variables had to move out to the module level and get ALLCAP names. Also, I have this little evil thought: what if the child function does not assign to par_var but instead passes it to another function, and that function assigns to it? par_var might have to be a list or other mutable collection to make that work, but... hmmm.

Whatever, that's done. HTML conversion works nicely and I am happy to say, is really quick. It takes less time to translate a document and create a new document, than it does to load the source document in the first place. Later this week I will be packaging PPQT for release.

Friday, June 19, 2015

HTML Translator going together fast

I spent several hours coding up an HTML translator and have it 75% complete. Another coding session will finish it; then I'll have to test it a wee bit. Finished by Tuesday, I reckon.

I only found one awkward spot in the design of the Translator API. An Illustration markup like this:

[Illustration:fig_55.png|fig_55.jpg Fig. 55 The whang-dangle is..

Produces a sequence of "events" as follows:

OPEN_ILLO with the two filenames
OPEN_PARA
LINE "The whang-doodle..."

So at the time I am generating the code for an image div, I have only the filenames. (Passing the filenames after the colon is a PPQT extension. If the user doesn't do that, the Translator has no way to get them.) The HTML Translator can use this to build a nice opening string of:

<div class=image>
<a href='images/fig_55.jpg'>
<img src='images/fig_55.png' /></a>

However, it would be a useful service if it could also generate alt= and title= attributes from the starting text of the caption, for example alt="Fig. 55 The whang-dangle.... But that can't happen because the opening text of the caption will not arrive for two more events.

I thought about postponing actual output of the image markup until the first paragraph arrived, but that would make the code just horribly ugly. On every OPEN_PARA event you'd have to ask, is this the start of a caption?

I put in a similar but lesser kludge in translating a Footnote. The OPEN_FNOTE event has the footnote key value, so it can generate the target anchor and the back-link to the footnote reference. Unfortunately all that goes inside the first paragraph of the note:

<div class='footnote'>
<p><a id='Footnote_X'></a><a href='FNref_X'>[X]</a>

So here is the <p> being put out ahead of the coming OPEN_PARA event. So I had to create a switch footnote_starting and test it in the OPEN_PARA code, and when it is on, not generate anything and turn it off.

But by and large, the HTML Translator just kind of fell together. You just consider each of the 34 event codes in sequence, write a little bit of code, and repeat.

Tuesday, June 16, 2015

Just about ready for official release

I committed all the translator material to github. Except (OMG!) it appears I forgot to add the two new modules, translators.py and xlate_utils.py. OK, done. So that is all wrapped up and documented very well if I do say so (and as a retired tech writer, I think I know when an API is properly documented). I'm very pleased at how I used a formal syntax to define and verify the document structure. The code of translators.py is clean and well-organized. The mechanism for defining and displaying an "Options Dialog" is simple and (I hope) understandable.

There are parts of the xlate_utils.py module that I'm not quite so proud of. The tokenize() function is going to be fairly heavily used and I confess there are parts of it that are rather ad-hoc not to say downright kludgy. And not superbly well tested, yet. However, in coming days I'll be writing an HTML translator that will test it.

That done, on Monday and Tuesday I turned to my list of issues and resolved most of them. One that came out better than I originally expected was this: PPQT2 expects input files to be encoded UTF-8. The user can get a Latin-1 file correctly opened by renaming it to a suffix of .ltn, but if that is not done, the file will be input through the UTF-8 codec. Some special characters will not decode right and will be replaced with Unicode \ufffd, the "replacement character". And if this isn't noticed right away, and the file is saved, there's permanent loss of data.

So I very much wanted to catch this error early and warn the user. But how? I researched the methods of QTextStream, QFile, and QTextCodec. I know that while QTextStream is executing a readAll() call, it must use a QTextCodec.toUnicode() function. That function is capable of returning a count of invalid characters, but there doesn't seem to be any way to find it out.

It looked as if the only ways I could use to find out if the file decoded properly would be either, one, to read it with Python, in which case the readall method would throw an exception; or two, to use QTextStream.readAll() into a string and search the string for replacement characters. Either method would require me to change the API between the main window and the Book, or else to read the possibly-large document file twice.

Then it dawned on me that the QPlainTextEditor has a perfectly good find() method. All I had to do was, in the Book just after it has loaded the editor from the file, to call the editor's find() to look for a replacement character. One hopes the search fails. But if it does not, I can notify the user with a warning message, including the character position of the first replacement character. I made the warning message detailed and also included a pointer to the Help topic.

Another long-standing issue, more of a major loose end to clean up, was logging. There are lots and lots of log messages being issued all over the program. But the logging output was going nowhere. I had initially thought that I would add argument parsing, and use it to support --log-path= and --log-level= parameters. But that's dumb; I'm packaging the Windows and Mac OS versions as clickable apps, with no command-line input. And the Linux version doesn't have to be launched from a command line. So I did some study and reading and chose writable locations for log files based on the platform: /var/tmp for Linux, ~/Library/Logs in Mac OS, and \Windows\Temp in Windows. Oops, I just realized I committed the code for that, but I should also update the Help file to document it. Or maybe put it in the README for each version?

Anyway: tomorrow is Museum day. Thursday I need to spend most of my free time studying the docs for a new volunteer gig. I have an online training session for that at 4pm that day and I want to be prepared. But Friday I will start coding an HTML Translator. Should have that done early next week. By the end of next week I should have PPQT2 packaged up ready to announce. Can't wait.