Don't think of it as a bug...

...think of it as material for a blog post!

OK so there is an annoying bug in the edit display. I am putting a pale highlight on the current line (the line with the cursor) -- see the image on this post. This fixes an annoyance I had with version 1, that on a large screen it was easy to lose track of the cursor, especially after paging up or down. I'd often have to rattle the left and right arrow keys so I could spot the cursor moving.

Here's the basic code, cut down:

Note: The following is not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

    def _cursor_moved(self, force=False):
        tc = self.Editor.textCursor()
        self.ColNumber.setText( str( tc.positionInBlock() ) )
        tb = tc.block()
        if tb == self.last_text_block and not force :
            return # still on same line, nothing more to do
        # ...here fill in line-number, scan filename, and folio widgets...
        # Clear any highlight on the previous current line
        bc = QTextCursor(self.last_text_block)
        bc.setBlockFormat(self.normal_line_fmt)
        # remember this new current line
        self.last_text_block = tb
        # and set its highlight
        tc.setBlockFormat(self.current_line_fmt)

That's pretty simple: if the cursor is on the same line as last time, set the column number widget and bug out quick. (The force argument is so this same method can be used in certain rare cases by other routines to repaint the widgets even if the cursor hasn't moved.)

If the cursor is on a different line than the last time it moved, which we will know because the QTextBlock returned by the current cursor is not the same as before, then clear out the highlight on the previous line/block, and set a highlight on the new line/block. It never sets a highlight without also clearing one.

And it mostly works. Move the cursor with the arrow keys or by mouse-clicking, and the pale highlight moves around with it. Great!

Except in one case: if I select multiple lines—by dragging or by shift-clicking or by shift-down-arrowing—then all selected lines get the current-line highlight and it does not clear if I click somewhere else. It only clears if I run the cursor back over those lines. And here's the kicker: as I drag or shift-down-arrow to extend the selection, the line-number widget is updating properly. So the code above is being executed, recognizing that it is on a different line (or it wouldn't update the line-number widget), which means it must be clearing the highlight. But it doesn't. WUT?

Time to put in the print statements.

        dbg1 = tb.blockNumber()
        dbg2 = self.last_text_block.blockNumber()
        print('clearing block {0} setting block {1}'.format(dbg2,dbg1))

And it reports exactly what it should: Put the cursor on line 0, key shift-down-arrow to make a selection, it reports "cleaning block 0 setting block 1, clearing block 1 setting block 2" and so on, but the highlights on line 0 and 1 are not clearing. They are partly hidden by the brighter selection highlight, but there. Here's a picture.

If I click on line 4, the highlight on line 2, the last current line, is cleared, as is the selection highlight, but the highlights on lines 0 and 1 remain.

So there's no logic error, but some issue with setting a block format when the block is under another format, the selection highlight. I note that setting the current_line_fmt works, even when selection is in progress. You can see it sticking out from under the selection on line 2 above. But setting the normal_line_fmt doesn't take. What's the difference? It is initialized so:

        self.normal_line_fmt = QTextBlockFormat()

It's an empty, default format object. Whereas the current line format is set up:

        self.current_line_fmt.setBackground(colors.get_current_line_brush())

It is a format object with an explicit background brush. (The colors module returns basically QBrush(QColor('#FAFAE0')); eventually the user will be able to choose this color as a Preference.) Anyway the current line format has an explicit background brush and the normal format does not. Should I give it one, and if so, what color? Back to the Qt Assistant...

Answer: No. I changed the initialization of the normal format:

        self.normal_line_fmt = QTextBlockFormat()
        self.normal_line_fmt.setBackground(QBrush(Qt.white))

I also tried Qt.color0, the transparent color. The bug continues. So it is not the lack of an explicit background brush that prevents the clearing. Back to the Assistant...

Ah. Re-reading the doc for QTextCursor.setBlockFormat() I note it reads "Sets the block format of the current block (or all blocks that are contained in the selection) to format." Oops.

The problem, I betcha, is in the line

tc.setBlockFormat(self.current_line_fmt)

Because tc is the actual cursor of the current document, the cursor that represents the current selection. And when it is used to set the block format of the "current" block, it sets a block format for "all blocks in the selection" which includes the prior block whose format was just cleared!

And that's it. If I set the current line format using a new cursor with no selection, it all works. So the final lines of code now read:

        # clear any highlight on the previous current line
        temp_cursor = QTextCursor(self.last_text_block)
        temp_cursor.setBlockFormat(self.normal_line_fmt)
        # remember this new current line
        self.last_text_block = tb
        # and set its highlight
        temp_cursor = QTextCursor(tb)
        temp_cursor.setBlockFormat(self.current_line_fmt)

Make a new text cursor based on the current block, one which has (by default) no selection. Use that to set the format for the current line. It affects only the actual current line, not any other lines that might be in the current user selection. Bug fixed! Do a commit and break for lunch!

Being Resourceful

According to blogger, after each new post here, there's a bit of a spike in views. So at least a dozen people out there have this blog in their RSS readers. Howdy! I'll try to keep it going.

Another Unknown is Known

One of the unknowns that's been niggling at the back of my mind for weeks, is how to use the Qt resource system. I have a font that I want to carry along in the app so I can be sure it will be available. I think there will be some custom icons as well, like this one: which I think may be the icon for a button in the Find dialog that establishes a limited range of text for find/replace.

Anyway all such things shouldn't be carried along as separate files, but should be incorporated right into the program as resources. I knew Qt had a scheme for this and PyQt supported it; and I knew I would need to use it; but I didn't know how it worked and hadn't stirred myself to find out. So today I did.

As usual with these things, once you lay it out clearly it's no big deal. In a nutshell,

• You list your resource files using XML syntax, in a file with type .rcc
• You run a utility pyrcc5 which writes a python module.
• You import that python module.

Then at run time, any file you named in the .rcc is available using the path prefix :/, as in :/hand-gripping.png.

The .rcc file consists of the following lines:

<!DOCTYPE RCC><RCC version="1.0">
<qresource>
   a list of files to embed, one line per file
</qresource>
</RCC>

Each file to embed is described with a line like this:

<file alias='run-time-name-of-resource'>relative-path-to-file</file>

For example, <file alias='hand-gripping.png'>hand-gripping.png</file>

And that's about it. I set up a folder Resources at the same level as my source modules. In it I put the .rcc file and the various files it named. Then from the command line, at the module level, I gave the command

pyrcc5 -o resources.py Resources/resources.rcc

And magically a 1.5MB file resources.py appeared. That file starts out with:

qt_resource_data = b"\
\x00\x04\xc8\x40\
\x00\
\x01\x00\x00\x00\x12\x01\x00\x00\x04\x00\x20\x46\x46\x54\x4d\x61\
\x13\x1d\x0b\x00\x04\x91\x38\x00\x00\x00\x1c\x47\x44\x45\x46\x79\

And ends, 20,000 lines later, with

def qInitResources():
    QtCore.qRegisterResourceData(0x01, qt_resource_struct, qt_resource_name, qt_resource_data)

qInitResources()

And that, executed upon the first import of the module, presumably tells the QApplication about these resources. It's pretty straightforward. Now I can get on with finalizing initialization of the font resources.

Context Menus are Hard

Really, context menus are the orphan children of Qt. First, you cannot define one using Qt Designer. So even if all the rest of your UI (including the "real" menus) is designed with the designer, your context menu has to be hand-coded. And second, it appears to be impossible to test one using QTest.

As reported yesterday, it doesn't seem possible to use QTest mouse events to invoke a context menu—at least for a QTextEdit under Mac OS. So today I'm constructing a QContextMenuEvent object my ownself, and passing it to the editor to process. This works, kind of, but not in a useful way for testing. Here's some code.

ev = the_book.editv.Editor
ev_w = ev.width()
ev_h = ev.height()
ev_m = QPoint(ev_w/2,ev_h/2)

cxe = QContextMenuEvent(
    QContextMenuEvent.Mouse,
    ev_m, ev.mapToGlobal(ev_m),
    Qt.NoModifier)

app = QCoreApplication.instance()
app.notify(ev,cxe)
cxm = ev.childAt(ev_w+5,ev_h+5)

QCoreApplication.notify(target,event) "Sends event to receiver... Returns the value that is returned from the receiver's event handler." My supposition was that the context menu would pop up, and I'd get control back and use childAt() to get a reference to that menu, and then I could feed it keystrokes, for example a couple of down-arrows and a return to select the third item.

Well as so often happens, my supposition was wrong. Because I don't get control back! Notice what the documentation says, "the value that is returned from the receiver's event handler"? But that event handler consists entirely of, quote:

    def contextMenuEvent(self, event):
        self.context_menu.exec_(event.globalPos())

It won't be returning until the menu's exec_() method returns! The nice editview window sits there with its context menu open in the middle, and waits for user input. It won't finish until the menu is satisfied with a Return or dismissed with Escape. At which time, there is no longer a menu at any location to be captured or tested.

Another problem I realized yesterday evening was this. Suppose I could get a reference to the menu object and feed it keystrokes. Two of its actions are toggle-able or check-able choices; in order to verify they had worked, I'd have to somehow sample their toggle status. How does one ask, does this menu choice have a check-mark on it?

The other two choices are worse. One invokes a native OS file-selection dialog, which is certainly beyond the reach of QTest or anything based on the Qt event mechanism. So how could my unit-test code browse to a test scanno file and click OK? And then how could it verify that (at least one of) the expected words was now highlighted?

And the other pops up a dialog with a drop-down menu of available dictionary tags. I'd like to have a test that verified that this dialog appeared, that it had the expected number of entries, that the last-chosen dictionary tag was the selected one, that another could be chosen, and so forth. But my test code doesn't get control back while the context menu is up, and it won't end until that dictionary dialog ends, so... bleagh.

What's Happening?

The last couple of days have been a long digression into the world of QTest and QEvents. It was motivated by my desire to make an automated unit test particularly of the context menu I've added to the editview.

The motivation for a context menu, like so many changed features of PPQT2, is that there can be multiple books open at once. In V1, there was only one book and accordingly only one:

• Primary spelling dictionary
• File of "scannos" (common OCR errors like "arid")
• Choice of whether or not to highlight spelling errors
• Choice of whether or not to highlight scannos

But now these choices have to be individualized per book. In V1, there could be a File menu action, "Choose scanno file..." but in V2, if that action was in the File menu, there'd have to be a convention about which of the (possibly several) open books those scannos should apply to. The one with the keyboard focus? Suppose the keyboard focus is in the Help panel? Similarly for the V1 View > Choose Dictionary menu item. And for the View > Highlight Scannos/Spelling toggles. All these choices need to be clearly associated to a single book. Hence, a context menu in the editor panel with four actions. When you have to right-click on the editor in order to choose a scanno file, you know where those scannos will be applied.

The little context menu is in and working, at least to casual testing. But I'm trying to do this shit right; and that means, an automated unit test. And that meant, I presumed, using QTest's mouse actions to simulate a control-click on the edit widget.

So I wrote up a test case that went in part like this:

ev = the_book.editv.Editor # ref to QPlainTextEdit widget
ev_w = ev.width()
ev_h = ev.height()
ev_m = QPoint(ev_w/2,ev_h/2) # the middle
QTest.mouseClick(ev, Qt.LeftButton, Qt.CtlModifier, ev_m, 1000)

...and, it didn't work. Nothing. I tried all sorts of mouse actions using both QTest's methods (mouseClick, mouseDblClick, mousePress, mouseRelease) and actually composing my own QMouseEvent objects and pushing them in with QApplication.postEvent(). Fast forward through about six hours of fiddle-faddling over three days. Sometimes I could get a double-click to work and sometimes not. Mouse presses or clicks with any button and modifier—nada. zip.

Now as it happens, I have an "event filter" on the editor. This is because I want to handle certain keystrokes, as described previously. But the edit widget is created by code generated from Qt Designer. That means it can only be a standard QPlainTextEditor. The normal way to intercept keystrokes is to subclass a widget and override its keyPressEvent() method. Don't think there's a way to get Qt Designer to plug in a custom subclass of a widget type.

However there's a way that you can install an "event filter" on any widget. That directs all events for that widget through the filter function. If it handles the event it returns True; if not, it returns False and the event is presented to the widget in the normal way. So editview puts a filter on events to the edit widget, picks off just the keyEvents it wants, and passes the rest on.

So I took advantage of this to just print out all the events passing through the edit widget so I could find out just what the heck mouse events it was getting when I clicked to bring up the context menu.

Surprise! It doesn't get any!

The Qt docs would have one believe that as a mouse moves over and clicks or drags on a widget, there's a constant flow of QMouseEvent objects to it. Nope. Not on my Macbook, anyway.

There are Enter and Leave events as the mouse pointer comes into and out of the frame of the widget. These aren't mouse events as such. There are lots of other sorts of events like Paint and Tooltip. But there are almost no mouse events posted. What does appear while the mouse is active, on every click and streaming during any drag, is QInputMethodQuery events. During a mouse click or drag, when I'd expect a stream of QMouseEvent postings, all that comes is a stream of QInputMethodQuery.

This peculiar class has only one property, a query with a not very helpful list of values. Of these possible "queries" only one is being sent in my system, the query IMEnabled meaning "The widget accepts input method input". The receiver is supposed to set something from a set of even less-interesting values in the event. Of course, my event filter doesn't see what is being set; it only sees the event on its way in.

Something nefarious is going on here. Perhaps it is only in the Mac OS; perhaps it only affects QTextEdit and derivatives (QTest mouse actions directed to other widgets seem to work). But for the editor, on my macbook, the whole mouse event architecture is effectively being ignored, replaced by something only minimally documented and not amenable to code introspection for unit-testing.

There are also a few InputMethodQueries issued just before any keyPressEvent and I am deeply suspicious that this is related to the inconsistent handling of the Mac keyboard I noted earlier.

That aside, the net from all this investigation is to realize that I don't really need to simulate the mouse at all. All I need to do is to fabricate a QContextMenuEvent with a given position in the middle of the editor. Post that; then use getChildAt that same position to get a reference to the context menu, and then I can send it keystrokes using QTest.

To be tried tomorrow.

Which Line Is It, Anyway?

The editview module is getting pretty complete. The only missing function is the dreaded syntax-highlighter to highlight scannos or spelling errors. Here's what it looks like now.

Today I added the code to highlight the current line. That's why one line has a sort of pale-lemon background. In V1, there was no current line highlight, and it was quite easy to lose sight of the cursor, and have to rattle the arrow keys to find it. (The string shown in dark gray is selected text and is actually bright yellow; the Grab utility did something to the colors.)

Qt's method of doing this was surprising to me.

In a QPlainTextEdit, there is a 1:1 correspondence between text blocks and logical lines. Each line of text is in one QTextBlock. Now, QTextBlock has a property blockFormat which is a QTextBlockFormat, which is itself a QTextCharFormat derivative, i.e. it can be used to set the font, color, background brush and so on. So when I started looking at how to make the current line a different color, I saw this and supposed it would be a matter of, each time the cursor moved:

• Get the text block containing the cursor, a single method call,
• Clear the background brush of the previous line's text block,
• Set the current text block's blockFormat to a different background brush

But in fact QTextBlock lacks anything like a setBlockFormat, so the property is read-only. And setting the background property of the returned QTextBlockFormat object was accepted but had no visible effect.

Sigh, back to the googles to find a number of places in the Qt docs, stackoverflow and the like, where the question is raised and answered.

QPlainTextEdit supports a property extraSelections, which is a list of QTextEdit::ExtraSelection objects. This is the first and I think only time I've seen a class documented as child of another class. And it's a weird little class; it has no methods (not even a constructor), just two properties, cursor and format. So it's basically the C++ version of a python tuple.

What you do is, you get a QTextCursor to select the entire line, and you build an ExtraSelection object with that cursor and the QTextCharFormat you want to use, and assign that to the edit object's list of extra selections. This is a lot of mechanism to just highlight one line. Apparently the intent is to support an IDE that, for example, wants to put a different color on each line set as a breakpoint, or such.

Note: The following is not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

Anyway for the curious, this is the code that executes every bloody time the cursor moves:

    def _cursor_moved(self):
        tc = QTextCursor(self.Editor.textCursor())
        self.ColNumber.setText(str(tc.positionInBlock()))
        tb = tc.block()
        ln = tb.blockNumber()+1 # block #s are origin-0, line #s origin-1
        if ln != self.last_line_number:
            self.last_line_number = ln
            self.LineNumber.setText(str(ln))
            tc.movePosition(QTextCursor.EndOfBlock)
            tc.movePosition(QTextCursor.StartOfBlock,QTextCursor.KeepAnchor)
            self.current_line_thing.cursor = tc
            self.Editor.setExtraSelections([self.current_line_thing])
            pn = self.page_model.page_index(tc.position())
            if pn is not None : # the page model has info on this position
                self.ImageFilename.setText(self.page_model.filename(pn))
                self.Folio.setText(self.page_model.folio_string(pn))
            else: # no image data, or positioned above page 1
                self.ImageFilename.setText('')
                self.Folio.setText('')

In sequence this does as follows:

• Get a copy of the current edit cursor. A copy because we may mess with it later.
• Set the column number in the column number widget.
• Get the QTextBlock containing the cursor's position property (note 1 below).
• Get the line number it represents.
• If this block is a change from before (note 2):
• Set the line number in the line number widget.
• Make the cursor selection be the entire line ("click" at the end, "drag" to the front)
• Set that cursor in a single ExtraSelection object we keep handy.
• Assign that object as a list of one item to the editor's extra selections.
• Get the filename of the current image file, if any; and if there is one, display it and the logical folio for that page in the image and folio widgets.

Note 1: If there's no selection, a text cursor's position is just where the cursor is. But if the user has made a selection, the position property might be at either end of it. Drag from up-left toward down-right and the position is the end of the selection. Drag the other way, it's at the start. Drag a multi-line selection that starts and ends in mid-line. One of the lines will have the faint current-line highlight: the top line if you dragged up, the bottom line if you dragged down. I don't think anyone will notice, or care if they do. I could add code to set the current line on min(tc.position(),tc.anchor())—but I won't.

Note 2: Initially, there was no "if ln != self.last_line_number" test; everything was done every time the cursor moved. And actually performance was fine. But I just could not stand the idea of all that redundant fussing about happening when it didn't have to.

Further on the Mac Option Key

The Qt Forum post I made about the Option-key problem, after 22 hours, has been viewed 32 times but drawn no responses. I also posted a respectful query on the pyqt list this morning (after obsessing about the issue some of the night).

I also spent a couple more hours delving deeply into the QCoreApplication, QGuiApplication, and QApplication docs, hoping to find some kind of magic switch to change the behavior of the key interface. I speculate that Qt5 has better Cocoa integration and as a result is getting the logical key from a higher-level interface than before.

Supposing it can't be fixed or circumvented, what I will have to do is: In constants.py where the key values and key sets are determined, check the platform and use Qt.MetaModifier instead of Qt.AltModifier when defining keys for Mac. This substitutes the actual Control shift for the Option shift.

That would be the only module with a platform dependency. Others just use the names of keys and key-sets defined in constants.py. For the user, I will have to have separate documentation about bookmarks, for Mac and non-Mac. For non-Mac, it'll remain "Press control and alt with a number 1-9 to set that bookmark." For mac it will be "Press the Control key and the Command key together with a number 1-9..." And the beautiful consistency ("where you see 'alt' think 'option'" at the front and never mention it again) is gone.

Another issue is the use of ctl-alt-M and ctl-alt-P in the Notes panel, to insert the current line or image number. Possibly I can just change the key definition in constants to whatever the mac keyboard generates for option-M and option-U (pi and mu, it seems). Or keep the directions consistent, and completely wipe out any use of Option-keys in Mac.

---

Also today I tested and committed the zoom keys, which work a treat. The unit test module buzzes up 10 points and down 15, looks great.

A Bump in the Road

Today I thought I'd add in the special keystrokes to the editview. There are three groups of them: a set that interact with the Find dialog (^f, ^g, etc), and these I'm deferring until I actually work on the Find panel; a bookmark set, (ctl-1 to 9 to jump to a bookmark, ctl-alt-1 to 9 to set one); and ctl-plus/minus to zoom. All of these were implemented and working in version 1, using the keyPressEvent() method to trap the keys.

So I messed around and tidied up the constants that define the various groups of keys as sets, so the keyPressEvent can very quickly determine if a key is one it handles, or not: if the_key in zoom_set and so on.

With the brush cleared, I copied over the keyPressEvent code from V1 and recoded it (smarter and tighter) for V2 and ran a test, and oops something is not working.

Specifically, it is no longer possible to set bookmark 2 by pressing ctl-alt-2. On a mac, that's command-option-2, which Qt delivers as the Qt.ALT_MODIFIER plus Qt.CTL_MODIFIER and the key of Qt.KEY_2.

Or rather, it used to do that. I fired up PPQT version 1 just to make sure. Yup, could set a bookmark using cmd-opt-2. But not in the new version. Put in debug printout. The key event delivered the same modifier values, ctl+alt, but the key value was... 0x2122, the ™ key? And cmd-alt-3 gave me Qt.KEY_STERLING, 0xA3. And cmd-alt-1 is a dead key.

Pondering ensued. OK, these are the glyphs that you see, if you open the Mac Keyboard viewer widget and depress the Option key. So under Qt5, the keyboard event processor is delivering the OS's logical key, but under Qt4 in the same machine at the same time it delivers the physical key.

Oh dear.

I spent several hours searching stackoverflow and the qt-project forums and bug database but nothing seemed relevant. I posted a query in the Qt forum. But I have little hope. It looks very much as if I'll have to change they key choices for bookmarks, and make them platform-dependent. In Windows and Linux they can continue to be ctl[-alt]-1 to 9, but in Mac OS this will change. The only reliable special key modifiers are control (Command) and meta (the Control key!).

In V1 it was great that I could document just once at the top of the docs, that in Mac, "ctl means cmd" and "alt means option". And that was consistent throughout. Now it won't be because the Option key is effectively dead for my purposes. I'll have to tell the mac user, "when I say control I mean command, but when I say alt, I mean control." Won't that be nice? Plus, I'll have to have code that looks at the platform and redefines the key sets for Mac at startup. Very disappointing.

Assisting the Upgradement

I got burned by one of what turned out to be quite a list of small incompatibilities between PyQt4 and PyQt5. Just fooling around I tried upgrading one of Mark Summerfield's utilities to PyQt5. It contained the following code:

        path = QFileDialog.getOpenFileName(self,
                "Make PyQt - Set Tool Path", label.text())
        if path:
            label.setText(QDir.toNativeSeparators(path))

Pretty obviously Mark expected getOpenFileName to return a path or a null string. But when executed, and I clicked Cancel in the file dialog, it caused an error in the label.setText statement. Whatever got into path evaluated to True, but wasn't a string.

It turned out to be a tuple with two strings. I documented this to the pyqt mailing list and was embarrassed when Phil just replied with the above link to the list of incompatibilities, one of which is a change to the API of the whole family of five "get..." methods supported by QFileDialog. What had happened to cause this seemingly arbitrary breaking of an existing API? It seems that PyQt4 had introduced some variant methods "to avoid the need for mutable strings". Now these extra "get...Filter" methods were being dropped and their function folded into the basic "get..." methods. And that entailed changing the return value of getOpenFileName from a simple string to a tuple of two strings.

It still seems arbitrary to me, breaking existing code in an unexpected way for no very good reason. But it's a done deal, so how to make sure that this incompatibility, and all the other subtle incompatibilities in the list, don't get overlooked? (And don't miss the fact that one item in the list is open-ended, saying "PyQt5 does not support any parts of the Qt API that are marked as deprecated or obsolete in Qt v5.0." What are those? Are they numerous?)

I decided it wouldn't be hard to write a tool to find and point out all, or anyway a lot of, these issues. In two afternoons of work I put together q45aide.py (click the link to see the Readme and get the code from Github). This is a straightforward source scanner that copies a program and inserts comments above any line that looks as if it will have an upgrade problem.

I'm particularly pleased with two features of this program. One is the way of finding out the modules that contain every Qt class. I needed this because one annoying change from Qt4 to Qt5 is that many classes moved from one import module to another. That invalidates most existing from PyQt4.module import (class-list) statements. I wanted to generate correct, minimal import statements from the class-names used in the program. But that meant having a dictionary whose keys were all the valid Qt class-names (over 880 of them, it turns out) and whose values were the module names that contain them.

I pondered quite a while over how to get such a list of class-names by module. I thought about manually or programatically scraping some pages from qt-project.org. But finally I realized, I could build a complete, accurate list dynamically in the program.

When you import a module, Python creates a namespace. And the names defined in a namespace can be interrogated by querying namespace.__dict__. So the program contains code like this:

    def load_namespace( ):
        global module_dict, import_dict
        # pick off QtXxxx from "PyQt5.QtXxxx"
        module_name = namespace.__name__.split('.')[1]
        for name in namespace.__dict__ :
            if name.startswith('Q') : # ignore e.g. __file__
                module_dict[name] = module_name

    import PyQt5.Qt as namespace ; load_namespace()
    import PyQt5.QtBluetooth as namespace ; load_namespace()
    ...

This loads module_dict with exactly the 880+ class-names related to their include modules, automatically updating should PyQt5 be updated with new or changed class-names.

The other thing that I got a kick out of writing was the way to write a list of class-names in either of two formats, in one statement. The program will generate one "from PyQt5.modulename import (class-name-list)" for each module that the input requires. A program option is -v, asking for the list to be stacked vertically. The only difference is that the class-name-list is either punctuated with comma-space, or with comma-newline-indent. And this is how it comes out:

                out_file.write('from PyQt5.{0} import\n   ('.format(mod_name))
                join_string = ',\n    ' if arg_v else ', '
                out_file.write(join_string.join(sorted(class_set)))
                out_file.write(')\n')

Badda-boom.

Using Qt Designer

Two unknowns I was fretting about have become less foggy: the work-flow that connects the graphic layout one creates with Qt Designer to the executable code; and the connection between user-visible strings defined to Qt Designer and the Qt translation mechanism.

Qt Designer to code

This part was of course laid out clearly in Summerfield's book. That book was my bible through the early days of building PPQT in 2011-12. Now it shows its age a little because there are minor differences between the PyQt4/Python 2 syntax of its examples, and the PyQt5/Python 3 syntax I'm using. But chapter 7, "Using Qt Designer", covered it all. Here's the sequence.

Run Qt Designer, select a template (in this case, just plain QWidget), and start dragging widgets onto it and laying them out. It's reasonably intuitive, especially if you know the names and uses of most of the widgets and their properties, as I do. It helps a lot to have a big screen. Qt Designer is almost unusable on the macbook because with all its various windows there's no room left for the widget you're designing. I used my desktop system with a 23-inch monitor and it was fine. It took an hour to lay out a satisfactory Edit panel as I described previously. Much of that time was spent in the Properties Editor, checking and specifying and re-specifying the many, many properties of each widget.

In the course of this I quickly discovered part of the answer to the question on translation. The Property Editor entry for any user-visible string—label text, tool-tip, status-tip, whats-this?—has a check-box "Translatable". I checked most of them. This widget has two QLabels and both will have their text filled in dynamically. But all tool-tips need to be translatable.

Having the check-box there keeps one aware that the English text you compose now will have to be translated. My experience in writing for translation, and writing tech material for people who have English as a second language, goes way back, to 1975-6 when I was at IBM World Trade in London and writing material to be read colleagues who were Brits, Swedes, Dutch and Italians. I learned then to keep going over my text to make sure it was simple, terse, and unambiguous; used no colloquialisms or metaphors; used the smallest vocabulary that would express the thought. (Which isn't a bad mindset for any expository writing.)

You save your design to a file of type .ui and invoke pyuic5, a command-line utility that reads it and writes some python source. (Summerfield invokes it by way of a version of make, makepyqt.pyw, but I don't see that in my installation. TB investigated.) The contents of this source were at first a little baffling to me. Here's the start:

class Ui_EditViewWidget(object):
    def setupUi(self, EditViewWidget):
        EditViewWidget.setObjectName("EditViewWidget")
        # ...and 140 more lines of setting-up code such as
        self.DocName = QtWidgets.QLabel(self.frame)
        # ...and the other sub-widgets...

What do I do with this? I wondered. Do I need to instantiate an object of this class? But it doesn't have an __init__; and what's this EditViewWidget being passed to this setupUi method?

Read the manual, doofus. Or in this case, keep reading in chapter 7 of Summerfield.

What I am supposed to do with this is to invoke it in—for the first time in my Pythonic career—a multiple-inheritance class definition, like this:

class EditView(QWidget, Ui_EditViewWidget):
    def __init__(self, my_book, parent=None):
        super().__init__(parent) # initialize QWidget
        self.setupUI(self) # invoke the initializer in class Ui_EditViewWidget

I am creating a QWidget subclass that also incorporates the code prepared by pyuic5. Following the call to setupUI, "self" incorporates all the widgets I defined to Qt Designer, under the object names I gave them in the Properties editor. Further initialization code is needed, for example to connect signals, set up the syntax highlighter, etc. But 150-odd lines of detailed GUI initialization are taken care of separately and more important, can be reviewed and altered at any time with Qt Designer, with no impact on the code.

Translation de-fog

To implement translation, the setupUi method ends with this:

    def retranslateUi(self, EditViewWidget):
        _translate = QtCore.QCoreApplication.translate

...which is followed by a line like this for every string that got the "Translatable" tick-mark:

        self.DocName.setToolTip(_translate("EditViewWidget", "Document filename"))

That pushes the fog of unknowns back a bit. It remains for me to learn how translation actually works.

Unknown Unknowns

Yesterday I checked off pagedata as coded and tested. That's the last remaining support or background module needed to allow the editor to run. So the next thing to tackle is editview.py, the visual face of the editable document. I imagine this as a widget containing, principally, the QPlainTextEdit, and below it a bar with five items:

• A QLabel containing the filename of the document. This will change its font style with the document's modified status, becoming perhaps bold and magenta when a save is needed (for the document or for metadata).
• A QLabel with the current folio number—a label because it isn't changeable by the user; folios are changed by modifying the folio rules in the Pages panel.
• A numeric text entry field with room for four digits, displaying the scan image number corresponding to the cursor position. Editable; you can type a new number to effect a jump.
• A numeric text entry field with room for 6 digits, displaying the current text block (line) number, updating as the cursor moves. Again, type a new number to jump in the document.
• A QLabel with the current "column" number in the current line. Not editable. (Use an arrow key or just click.)

The latter four items of course update dynamically as the cursor moves, on receipt of the cursor movement signal from the editor.

So, if I understand all this (not difficult given it's almost the same as the status bar area of the PPQT main window), where are the Unknown Unknowns of the title? Hah. Well, they are actually known, at least by category, but just the same I feel anxious about launching into this phase. They are:

1. Using Designer

The Qt Designer is a graphic tool for designing a layout. I played with it a bit a couple of years ago when starting PPQT but ended up doing all my widget layouts "by hand" with explicit code like this (one of the simplest ones)

        vbox = QVBoxLayout()
        # the image gets a high stretch and default alignment, the text
        # label hugs the bottom and doesn't stretch at all.
        vbox.addWidget(self.txLabel,0,Qt.AlignBottom)
        vbox.addWidget(self.scarea,10)
        vbox.addLayout(zhbox,0)
        self.setLayout(vbox)

It makes the __init__() rawther lengthy. With Qt Designer you supposedly separate your UI design from the code. Designer saves a file of UI info; you apply a PyQt utility to convert this into something Python can execute; you import it and execute it. Covered in Summerfield's book and in the Qt docs. But I have lots of questions, like: how are signals connected between elements; how are elements connected to the methods that update them; how do label texts set in Designer get tr() translated; just generally a fog of unknowns. But I'd like to give it a try and the editview widget should be a good test case.

I18N

Does anybody use that term any more? "I18N" was a thing back in the 80s, late 70s even at IBM. (It means "Internationalization", duh.) Anyway, I've committed to PGDP Canada that PPQT2 will be translatable. Meaning every damn user-visible text string has to be wrapped in a tr() call. And editview is the first module that has user-visible strings (log messages don't count). The tr() call is all I know about. I am anxious about the whole rest of Qt's I18N system. How do the tr'd strings get collected; how does a translator create an alternate translation; will I have to start using Qt Make; what the heck is Locale and how do I control it for testing purposes... gaaahhh. Much reading to do.

GUI Unit Tests

With this first UI module I enter the world of automated UI testing and I haven't a clue. Well, one clue: QTest. There's a multi-chapter writeup on QTest and simulation of GUI events. I presume I'll use that. There are third-party packages like FrogLogic's "Squish" but they are very expensive, at least that on is. There are open-source packages for test automation but the ones I've seen are single-platform. So I suppose I'll be rolling my own using QTest. But I really have no idea.

So: the known Unknowns are just packed with unknown unknowns. I will be learning as I go, and I will be posting what I learn to this blog. Because that's one way I have of consolidating what I've learned. You're welcome.

Getting useful info from QFontDatabase

I'll get back to the Last(goddam)Resort font issue shortly. But in learning about it I've had to look a little closer at what Qt knows about fonts, and as a result I've got a function that might be useful to others.

The QFontDatabase object contains whatever Qt knows about fonts that are "available in the underlying window system". However it makes this information available in a rather awkward way. You can call its families() method to get a list of all font names:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QFont, QFontDatabase
the_app = QApplication([])
f_db = QFontDatabase()
for family in f_db.families():
    print(family)

My macbook is rather over-endowed with fonts, and this prints a list of 238 names.

You can also ask for the recommended or default fixed and "general" fonts,

qf_fixed = f_db.systemFont(QFontDatabase.FixedFont)
print( 'Fixed font:',qf_fixed.family() )
qf_general = f_db.systemFont(QFontDatabase.GeneralFont)
print( 'General font:',qf_general.family() )

Which on my macbook prints

Fixed font: Monaco
General font: .Lucida Grande UI

Here's a surprise: what the heck is that leading dot doing in the name of the general font? There is definitely no font named ".Lucida Grande" in the list of families, nor displayed by the Fontbook app, nor (using find /System ".Lucida*') in the System folder. However, the font db will create a QFont for it:

>>> lgf = f_db.font('.Lucida Grande UI','',12)
>>> lgf.family()
'.Lucida Grande UI'
>>> lgf.pointSize()
13

So it's real in some sense, and your code can use it. Moving on...

You can ask the font db things about any single family name, for example isFixedPitch(family). But font family names aren't consistent from system to system. And often what you want is a font that meets certain criteria, such as: it's "Times" by some name, or it has a bold or italic variant, or it scales to 36pt. So I put together this little function that will return a (possibly empty) list of fonts that meet given criteria:

def font_filter( name = None, style = None, fixed = False, sizes = [], language = None ) :
    db = QFontDatabase()
    selection = db.families()
    if name :
        selection = [family for family in selection if name in family ]
    if style :
        selection = [family for family in selection if style in db.styles(family) ]
    if fixed :
        selection = [family for family in selection if db.isFixedPitch(family) ]
    if language :
        selection = [family for family in selection if language in db.writingSystems(family) ]
    if sizes :
        size_set = set(sizes)
        selection = [family for family in selection
                     if size_set.issubset( set( db.smoothSizes(family, '' ) ) ) ]
    return selection

For example,

>>> print( font_filter( name = 'Helvetica' ) )
['Helvetica', 'Helvetica CY', 'Helvetica Neue']
>>> print( font_filter( fixed = True ) )
['Anonymous Pro', 'Courier', 'Courier New', 'Inconsolata', 'PCMyungjo', 'PT Mono']
>>> print( font_filter(fixed = True, style='Bold', sizes=[12, 18]) )
['Courier', 'Courier New', 'PT Mono']
>>> font_filter(fixed=True, language=QFontDatabase.Hebrew)
['Courier New']

Next time: back to the issue of the Last(goddam)Resort font problem.

Last (goddam) Resort Font

I wrote CoBro back in 2012 with the original intent of "releasing" it for public use (i.e. making it known on the the comics subreddit) but because of QWebKit instabilities as well as problems packaging it with either PyInstaller or cx_freeze, I put that off until I could re-do it for Qt5 and Python 3.

Now I've done that it still is plagued by QWebKit issues (and still breaks cx_freeze, and PyInstaller still doesn't support Python 3). Among the instabilities is a very intermittent tendency to throw a blizzard of messages like this: 2014-03-14 15:06:59.270 Python[6570:d07] Critical failure: the LastResort font is unavailable.

This is a Python log message, not a Qt one that I could maybe stifle with the technique mentioned in the prior post. And "Critical" at that! So what's going on?

Turning to The Google I find that a lot of people encountered this message with a variety of different apps, but mostly back in 2010 or 2011 when Snow Leopard was new. But some information emerged.

What is the Last Resort font? The official explanation from Unicode.org is that it is a font to use when the system needs to print a Unicode glyph and all available fonts lack values for that code position.

if the font cannot represent any particular Unicode character, the appropriate "missing" glyph from the Last Resort font is used instead. This provides users with the ability to tell what sort of character it is, and gives them a clue as to what type of font they would need to display the characters correctly.

This explains why the message comes only rarely but in a flurry: it happens when a comic web page has a glyph that isn't in any available font. That doesn't happen all the time, but if it does, the site is likely to have a string of such glyphs; hence the multiple messages. But why can't the WebKit browser find the font? With a little poking around on my system (and absolutely no help from the Finder, whose search function insists there is nothing to match that name) I did find that /System/Library/Fonts/LastResort.ttf exists and had 644 permissions. So...

Finally I found this FontGeek post about Safari having the same problem. They claim that because of "sandboxing" the browser can't access the folder where LastResort resides. I'm dubious about the explanation; and the fix they offer isn't directly applicable to the Qt WebKit code as far as I can tell. But it's the best explanation I've seen.

What to do? One possibility: get a copy of LastResort.ttf and include it in the app. I've already got code in PPQT V1 to load a font bundled with the app and add it to the QFontDatabase. I need to think about this, also I need to find a web comic that will trigger the issue reliably.

Making a bad QTextCursor and a promising find

Working today on the unit testing and code finalization of pagedata, the module that keeps track of where the scanned OCR pages of a book each start. This module acts as the data model for several clients: every time the user moves the edit cursor, a bunch of different widgets will ask pagedata "which page is the cursor on now?" The imageview module asks it for the filename of the .png file to display, the scan# and folio# widgets under the edit window ask it for those items, and of course the Pages panel calls on it for the rows of data it displays as a table.

I thought I'd finalized the code but of course as soon as I started adding test calls into it from its unit-test module I found not only bugs but also things I hadn't thought of. It's schizophrenic, flipping back and forth between coding and testing. "What will it do if I throw it this?" the tester asks, and the coder is thinking "Oh shit, why didn't I plan for that?"

Anyway, one of the pieces of crap the test-monkey in me flang at the pagedata module opened a wonderful new prospect in Qt error control! It went down like this.

Important method in the PageData object is read_pages, which processes the page-data lines from the .meta file. When a book is saved, all the metadata goes in the bookname.meta file, including everything we know about page boundary locations. So at load time, read_pages gets called to rip through these saved lines and rebuild the page table as it was when the book was saved.

There are six items in each line, the first being the character offset to the start of the page. That gets turned into a QTextCursor so that Qt will maintain the position as it changes under user editing actions. The code is simple:

try:
    (P, fn, pfrs, rule, fmt, nbr) = line.split(' ')
    tc = QTextCursor(self.document)
    tc.setPosition(int(P))

and so forth. The test case had already flung a non-integer position P, and the failure of int(P) was caught by the try/except fine. So the next nastiness was a bad position value, first 1000000, much larger than the document, next -1. But neither of these tripped an exception! All that happened was that a message appeared on stderr, "QTextCursor::setPosition: Position '100000' out of range" and the QTextCursor was unchanged.

This opened two new questions: (1), how the heck can read_pages detect that it got a bad position?, and (B), how can we avoid having that message, about an error we've anticipated and dealt with, cluttering up stderr and getting the user all upset?

It was too late in the day to investigate (1), but (B) is a problem I've been plagued by for a long time. Qt is just full of unhelpful debugging errors. My other app, a simple web browser based on QWebKit, likes to throw out stuff like this:

QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.

And I've been wondering in a dazed sort of way if there mightn't be some way to stifle those. But PPQT2 is using proper Python logging. Every module creates its own logger and writes diagnostics, warnings and errors with the logging API. And I definitely plan not to let those log messages dribble out on stderr; they will eventually go into a file.

So it suddenly occurred to me, is there maybe some way to divert the Qt log messages into the Python logging system? I wasn't aware of anything but I started browsing in the index of the Qt Assistant and turned up this: qInstallMessageHandler. This appears to offer a way to accept and process all of Qt's messages.

I'm quite excited about this. If it is accessible under PyQt5, I see a clear path to capturing all Qt messages and converting them into Python log entries. That will let me stifle the WebKit chatter from my comics browser and also this text cursor message in PPQT. Tomorrow I spend the day at the museum but thursday afternoon I get to dig into this!

Converting PPQT: which RE lib?

I'm working on a lengthy project to make version 2 of PPQT, a large Python/Qt app. I'm documenting some of the things I learn in occasional blog posts.

PPQT 1 makes frequent use of regular expressions, mostly using Qt's QRegExp class. That has to change for two reasons. One is that QRegExp falls quite a bit short of PCRE compatibility. Qt5 includes a new class, QRegularExpression, which does claim PCRE compatibility as well as performance, so at least I want to convert the old ones to the longer-named type.

However, one big difference from PyQt4 to 5 is the "new API" that abolishes use of QString. In PyQt4 many class methods take, or return, QStrings, and PPQT uses lots of QString objects. QStrings and QRegExps work well together; QRegExp.indexIn() takes a QString, and QString.find() takes a QRegExp.

In PyQt5, all classes that (in the C++ documentation) take or return a QString, now take or return a simple Python string value, with PyQt5 doing automatic conversion. There is no "QString" class in PyQt5—at all! That means there is no way to call QString.find(), and if you call QRegExp.indexIn(string), there will be a hidden conversion from Python to QString. Which means—why use Qt regexes at all? Since all program-accessible strings are Python strings, why not use Python's own regular expression support?

Standard Python support is the "re" lib. It also is not PCRE compatible (although closer than QRegExp) and not known for speed. But there is another: the "regex" module, which intends to become the Python standard but now is an optional install. It is PCRE-compatible, with the Unicode property searches and Unicode case-folding that are lacking in QRegExp and in the re module. It actually adds more functionality, including "fuzzy" matches that could be very useful to me in PPQT. The class and method names are the same as the standard re module.

Code Changes

One design difference between Python's re/regex and QRegularExpression on one side, and the QRegExp that PPQT 1 uses so many of on the other, will cause some code changes.

An instance of QRegExp is not reentrant: when it is used, it stores information about the match position and capture groups in the regex object. Such an object shouldn't be a global or class variable shared between instances of a class, because activity in one using method could overwrite a match found from another. But based on its design, PPQT 1 had frequent uses like this:

    if 0 <= re_object.indexIn(string):
        cap1 = re_object.cap(1)

Both re/regex and QRegularExpression take a different approach: the regex object knows about the search pattern but is otherwise immutable. When you perform a search with it it, it returns a match object that encodes the positions and lengths of the matched and captured substrings. The regex object can be a global; every using method gets its private match object to work with. However, code like that above has to be rewritten (using Python re/regex) as:

    match = re_object.match(string)
    if match : # i.e. result was not None
        cap1 = match.cap(1)

Python re/relib match returns None on failure, or a match object. None evaluates as False, so "if match" is equivalent to "if a match was found." The returned value of a QRegularExpression object is always a (take a deep breath) QRegularExpressionMatch object, so the equivalent would be:

    match = re_object.match(string)
    if match.isValid() : # match succeeded
        cap1 = match.captured(1)

Not only is this many more keystrokes to write, it entails two pointless auto-conversions between Python and Qt string types: from Python to Qstring in the match() call, and from QString to Python in the captured(1). All told, the Python relib seems a better choice and I plan to use it exclusively in PPQT 2.