Funny Little Thing (Solved)

Next day second thoughts: While as noted, QTextDocument inherits from QObject and as such has no font property, it does for no obvious reason have a defaultFont property which it imposes on any Q[Plain]TextEdit to which it is connected. This completely violates the Model/View scheme, the Model imposing a presentation feature on the View, but there it is. Furthermore, it can't be overridden! As I note below, when I interrogate the edit widget for its font().family and font().pointSize, it happily reports the values I'd set. But what it displays is its document's default font, if the document is connected second. So fie on ye, Qt designer.

---

OK, here's an oddity in Qt. I'm working on the editview, a panel mainly containing a QPlainTextEdit that is pretty much the heart of PPQT. But the editor is the "view" and its "model" or repository of data is a QTextDocument. Most of the GUI initialization is handled by the code generated from the Designer. But the __init__ for the editview has to do two key things. Well, there will be lots more, hooking up signals, setting up the syntax highlighter, blah blah, but in its rudimentary state, two things:

• Set its fonts
• Set its document to the edit model

Item two is pretty simple. The parent Book is the repository of all knowledge, so it just does self.Editor.setDocument(self.my_book.get_edit_model()).

Setting the fonts is slightly more work. This is looking ahead to when the user will be able to tell the main window, I want to choose a different UI font, the default font used in most labels and buttons, or I want to choose a different edit font, the had-better-be-monospaced font used only in the editview(s) and one or two other places like the Find text string.

So there can be multiple Books open each with its editview, and up in the main window the user says, "Let's use Courier!" Main window will emit a signal, and any widget that cares better catch it and change fonts.

Also, at open time when setting up a new editview, we want to restore the font size the user had last time the book was open. In version 1, with only one book, that was a global setting, but now there can be multiple documents with potentially each editview zoomed to a different font size! Just one of the many, many features affected by multiple documents.

Anyway, while initializing, the editview needs to ask its Book for the font size and also ask the global font module for the proper font, and set that family/size combo in the QPlainTextEdit. No biggie, a simple method that is called in init. or by the font-change signal:

    def set_fonts(self):
        general = fonts.get_general() # UI font at default size
        self.setFont(general) # set self, propogates to children
        mono = fonts.get_fixed(self.my_book.get_font_size())
        self.Editor.setFont(mono) # the editor is monospaced

This story is going somewhere, really. OK, so the editview initialized like this:

        self.set_fonts()
        self.Editor.setDocument(self.my_book.get_edit_model())

And it didn't work. When it displayed, all the labels would be in the system default font (Lucida Grande 13pt) but so would the edit widget be. I put in a debug print to query the edit widget's font and display it. It happily reported, "I'm using Liberation Mono 16 just like you said, boss", but when I typed in the edit window, it came out in Lucida. I set the parent widget to the mono font, and all the labels displayed in mono but the damn edit text was still Lucida!

After wasting a couple of hours on this, it occurred to me the QTextDocument might be the villain. I reversed the init to

        self.Editor.setDocument(self.document)
        self.set_fonts()

And it worked, the edit text is now in the chosen mono font. WTF? Reviewing the doc for QTextDocument, it inherits from QObject not QWidget, so it doesn't even have a font property. Yet somehow, the call to setDocument() undid the work of a preceding call to setFont(). Maybe changing the document is so basic to the edit widget that it treats it as a reset. Whatever. In my application, I only set the document once. As long as it happens first, all is well.

A First Look at Linguist

So, where were we? I've been away nearly a week, visiting wonderful Ames, Iowa. Not by choice, but by the whimsy of the NCAA Selection Committee, which in its wisdom chose to make the Stanford Women's Basketball team not only a number-2 seed, but make them play the first two rounds in Ames, on the campus of the University of Iowa, where the host school, the 4th-seeded Iowa Cyclones, draw 10,000 screaming fans to every home game.

Well, fortunately the Cyclones faded to a zephyr before the defense of the FSU Seminoles, so for the second game the Cardinal played in front of a half-empty and subdued arena, and won comfortably. Meanwhile we dealt with snow flurries and the difficulties of passing time among the limited amusements of Ames and Des Moines. If you'd rather know about that versus Qt Linguist, check the pictures.

A couple posts back I described the process of designing a widget with Qt Designer and how any string in the design could be designated "Translatable", and how that left distinct code in the generated Python of the widget class. With the result that, when the widget initializes itself, every translatable string will pass through the bowels of the QtCore.QCoreApplication.translate method before being assigned to its QLabel, push button, menu item or whatever its use.

The output of that method—usually just written tr() in the Qt documentation, but for an arcane reason having to do with the relationship of Python classes to C++ classes, PyQt5 needs to always call the Core version not the one inherited by every QObject—the method's output is either the original, or a translated string—if there exists a translation for that string for the current Locale.

But that leaves the question, where do translations come from? From work done by a Translator (a human) using Qt Linguist. I pursued the link between the widget code and Linguist a little further.

The bridge is the PyQt5 utility, pylupdate5. Its use is described in the PyQt5 online docs. One must create a minimal Qt project description file, in this case ppqt2.pro. Actually a make file for the Qt Make program, this file lists the relevant source files and the name of the translation file. Here is what I used:

SOURCES = editview_uic.py
TRANSLATIONS = ppqt2.ts

Listing just one source file now; later there would be many on that line.

Then you turn pylupdate5 loose on the .pro file and it fills up ppqt2.ts with a bunch of XML items like this:

    <message>
        <location filename="editview_uic.py" line="153"/>
        <source>Document filename</source>
        <translation type="unfinished"></translation>
    </message>

Now I could launch Qt Linguist from the Qt distribution, and use it to open the ppqt2.ts file. It presents me with a window whose top is like this:

Every string from every widget (just one widget for now) is shown. Click on one and prepare a translation for it in the bottom part of the window.

For some reason, spaces are shown as gray dots in this part of the window. There exist Qt "phrase books" for many languages, and the French one is open in the above image. It is offering "document" as a translation for "document". Fair enough, but I would have thought "document name" would be a common phrase. Apparently not. I typed in nom du document.

Anyway, that's what the Translator person works with. The texts for the given language would be saved back into the ppqt2.ts file. And somehow become available via the Core Translate method at run-time.

I'm not going to worry further about that last step, for now. I can see how translation would be done. I don't mean to actually do any translations (or request anyone to do any) until the whole app is in near-final state. But at least I know how it all works, I've seen it can work on my system, so that's one Unknown that's Known and I can relax about it.

Assisting the Upgradement

I got burned by one of what turned out to be quite a list of small incompatibilities between PyQt4 and PyQt5. Just fooling around I tried upgrading one of Mark Summerfield's utilities to PyQt5. It contained the following code:

        path = QFileDialog.getOpenFileName(self,
                "Make PyQt - Set Tool Path", label.text())
        if path:
            label.setText(QDir.toNativeSeparators(path))

Pretty obviously Mark expected getOpenFileName to return a path or a null string. But when executed, and I clicked Cancel in the file dialog, it caused an error in the label.setText statement. Whatever got into path evaluated to True, but wasn't a string.

It turned out to be a tuple with two strings. I documented this to the pyqt mailing list and was embarrassed when Phil just replied with the above link to the list of incompatibilities, one of which is a change to the API of the whole family of five "get..." methods supported by QFileDialog. What had happened to cause this seemingly arbitrary breaking of an existing API? It seems that PyQt4 had introduced some variant methods "to avoid the need for mutable strings". Now these extra "get...Filter" methods were being dropped and their function folded into the basic "get..." methods. And that entailed changing the return value of getOpenFileName from a simple string to a tuple of two strings.

It still seems arbitrary to me, breaking existing code in an unexpected way for no very good reason. But it's a done deal, so how to make sure that this incompatibility, and all the other subtle incompatibilities in the list, don't get overlooked? (And don't miss the fact that one item in the list is open-ended, saying "PyQt5 does not support any parts of the Qt API that are marked as deprecated or obsolete in Qt v5.0." What are those? Are they numerous?)

I decided it wouldn't be hard to write a tool to find and point out all, or anyway a lot of, these issues. In two afternoons of work I put together q45aide.py (click the link to see the Readme and get the code from Github). This is a straightforward source scanner that copies a program and inserts comments above any line that looks as if it will have an upgrade problem.

I'm particularly pleased with two features of this program. One is the way of finding out the modules that contain every Qt class. I needed this because one annoying change from Qt4 to Qt5 is that many classes moved from one import module to another. That invalidates most existing from PyQt4.module import (class-list) statements. I wanted to generate correct, minimal import statements from the class-names used in the program. But that meant having a dictionary whose keys were all the valid Qt class-names (over 880 of them, it turns out) and whose values were the module names that contain them.

I pondered quite a while over how to get such a list of class-names by module. I thought about manually or programatically scraping some pages from qt-project.org. But finally I realized, I could build a complete, accurate list dynamically in the program.

When you import a module, Python creates a namespace. And the names defined in a namespace can be interrogated by querying namespace.__dict__. So the program contains code like this:

    def load_namespace( ):
        global module_dict, import_dict
        # pick off QtXxxx from "PyQt5.QtXxxx"
        module_name = namespace.__name__.split('.')[1]
        for name in namespace.__dict__ :
            if name.startswith('Q') : # ignore e.g. __file__
                module_dict[name] = module_name

    import PyQt5.Qt as namespace ; load_namespace()
    import PyQt5.QtBluetooth as namespace ; load_namespace()
    ...

This loads module_dict with exactly the 880+ class-names related to their include modules, automatically updating should PyQt5 be updated with new or changed class-names.

The other thing that I got a kick out of writing was the way to write a list of class-names in either of two formats, in one statement. The program will generate one "from PyQt5.modulename import (class-name-list)" for each module that the input requires. A program option is -v, asking for the list to be stacked vertically. The only difference is that the class-name-list is either punctuated with comma-space, or with comma-newline-indent. And this is how it comes out:

                out_file.write('from PyQt5.{0} import\n   ('.format(mod_name))
                join_string = ',\n    ' if arg_v else ', '
                out_file.write(join_string.join(sorted(class_set)))
                out_file.write(')\n')

Badda-boom.

Using Qt Designer

Two unknowns I was fretting about have become less foggy: the work-flow that connects the graphic layout one creates with Qt Designer to the executable code; and the connection between user-visible strings defined to Qt Designer and the Qt translation mechanism.

Qt Designer to code

This part was of course laid out clearly in Summerfield's book. That book was my bible through the early days of building PPQT in 2011-12. Now it shows its age a little because there are minor differences between the PyQt4/Python 2 syntax of its examples, and the PyQt5/Python 3 syntax I'm using. But chapter 7, "Using Qt Designer", covered it all. Here's the sequence.

Run Qt Designer, select a template (in this case, just plain QWidget), and start dragging widgets onto it and laying them out. It's reasonably intuitive, especially if you know the names and uses of most of the widgets and their properties, as I do. It helps a lot to have a big screen. Qt Designer is almost unusable on the macbook because with all its various windows there's no room left for the widget you're designing. I used my desktop system with a 23-inch monitor and it was fine. It took an hour to lay out a satisfactory Edit panel as I described previously. Much of that time was spent in the Properties Editor, checking and specifying and re-specifying the many, many properties of each widget.

In the course of this I quickly discovered part of the answer to the question on translation. The Property Editor entry for any user-visible string—label text, tool-tip, status-tip, whats-this?—has a check-box "Translatable". I checked most of them. This widget has two QLabels and both will have their text filled in dynamically. But all tool-tips need to be translatable.

Having the check-box there keeps one aware that the English text you compose now will have to be translated. My experience in writing for translation, and writing tech material for people who have English as a second language, goes way back, to 1975-6 when I was at IBM World Trade in London and writing material to be read colleagues who were Brits, Swedes, Dutch and Italians. I learned then to keep going over my text to make sure it was simple, terse, and unambiguous; used no colloquialisms or metaphors; used the smallest vocabulary that would express the thought. (Which isn't a bad mindset for any expository writing.)

You save your design to a file of type .ui and invoke pyuic5, a command-line utility that reads it and writes some python source. (Summerfield invokes it by way of a version of make, makepyqt.pyw, but I don't see that in my installation. TB investigated.) The contents of this source were at first a little baffling to me. Here's the start:

class Ui_EditViewWidget(object):
    def setupUi(self, EditViewWidget):
        EditViewWidget.setObjectName("EditViewWidget")
        # ...and 140 more lines of setting-up code such as
        self.DocName = QtWidgets.QLabel(self.frame)
        # ...and the other sub-widgets...

What do I do with this? I wondered. Do I need to instantiate an object of this class? But it doesn't have an __init__; and what's this EditViewWidget being passed to this setupUi method?

Read the manual, doofus. Or in this case, keep reading in chapter 7 of Summerfield.

What I am supposed to do with this is to invoke it in—for the first time in my Pythonic career—a multiple-inheritance class definition, like this:

class EditView(QWidget, Ui_EditViewWidget):
    def __init__(self, my_book, parent=None):
        super().__init__(parent) # initialize QWidget
        self.setupUI(self) # invoke the initializer in class Ui_EditViewWidget

I am creating a QWidget subclass that also incorporates the code prepared by pyuic5. Following the call to setupUI, "self" incorporates all the widgets I defined to Qt Designer, under the object names I gave them in the Properties editor. Further initialization code is needed, for example to connect signals, set up the syntax highlighter, etc. But 150-odd lines of detailed GUI initialization are taken care of separately and more important, can be reviewed and altered at any time with Qt Designer, with no impact on the code.

Translation de-fog

To implement translation, the setupUi method ends with this:

    def retranslateUi(self, EditViewWidget):
        _translate = QtCore.QCoreApplication.translate

...which is followed by a line like this for every string that got the "Translatable" tick-mark:

        self.DocName.setToolTip(_translate("EditViewWidget", "Document filename"))

That pushes the fog of unknowns back a bit. It remains for me to learn how translation actually works.

Unknown Unknowns

Yesterday I checked off pagedata as coded and tested. That's the last remaining support or background module needed to allow the editor to run. So the next thing to tackle is editview.py, the visual face of the editable document. I imagine this as a widget containing, principally, the QPlainTextEdit, and below it a bar with five items:

• A QLabel containing the filename of the document. This will change its font style with the document's modified status, becoming perhaps bold and magenta when a save is needed (for the document or for metadata).
• A QLabel with the current folio number—a label because it isn't changeable by the user; folios are changed by modifying the folio rules in the Pages panel.
• A numeric text entry field with room for four digits, displaying the scan image number corresponding to the cursor position. Editable; you can type a new number to effect a jump.
• A numeric text entry field with room for 6 digits, displaying the current text block (line) number, updating as the cursor moves. Again, type a new number to jump in the document.
• A QLabel with the current "column" number in the current line. Not editable. (Use an arrow key or just click.)

The latter four items of course update dynamically as the cursor moves, on receipt of the cursor movement signal from the editor.

So, if I understand all this (not difficult given it's almost the same as the status bar area of the PPQT main window), where are the Unknown Unknowns of the title? Hah. Well, they are actually known, at least by category, but just the same I feel anxious about launching into this phase. They are:

1. Using Designer

The Qt Designer is a graphic tool for designing a layout. I played with it a bit a couple of years ago when starting PPQT but ended up doing all my widget layouts "by hand" with explicit code like this (one of the simplest ones)

        vbox = QVBoxLayout()
        # the image gets a high stretch and default alignment, the text
        # label hugs the bottom and doesn't stretch at all.
        vbox.addWidget(self.txLabel,0,Qt.AlignBottom)
        vbox.addWidget(self.scarea,10)
        vbox.addLayout(zhbox,0)
        self.setLayout(vbox)

It makes the __init__() rawther lengthy. With Qt Designer you supposedly separate your UI design from the code. Designer saves a file of UI info; you apply a PyQt utility to convert this into something Python can execute; you import it and execute it. Covered in Summerfield's book and in the Qt docs. But I have lots of questions, like: how are signals connected between elements; how are elements connected to the methods that update them; how do label texts set in Designer get tr() translated; just generally a fog of unknowns. But I'd like to give it a try and the editview widget should be a good test case.

I18N

Does anybody use that term any more? "I18N" was a thing back in the 80s, late 70s even at IBM. (It means "Internationalization", duh.) Anyway, I've committed to PGDP Canada that PPQT2 will be translatable. Meaning every damn user-visible text string has to be wrapped in a tr() call. And editview is the first module that has user-visible strings (log messages don't count). The tr() call is all I know about. I am anxious about the whole rest of Qt's I18N system. How do the tr'd strings get collected; how does a translator create an alternate translation; will I have to start using Qt Make; what the heck is Locale and how do I control it for testing purposes... gaaahhh. Much reading to do.

GUI Unit Tests

With this first UI module I enter the world of automated UI testing and I haven't a clue. Well, one clue: QTest. There's a multi-chapter writeup on QTest and simulation of GUI events. I presume I'll use that. There are third-party packages like FrogLogic's "Squish" but they are very expensive, at least that on is. There are open-source packages for test automation but the ones I've seen are single-platform. So I suppose I'll be rolling my own using QTest. But I really have no idea.

So: the known Unknowns are just packed with unknown unknowns. I will be learning as I go, and I will be posting what I learn to this blog. Because that's one way I have of consolidating what I've learned. You're welcome.

Bypassing the LastResort font error

As I noted previously, my mini web browser Cobro had begun occasionally dumping flurries of a message Python[6570:d07] Critical failure: the LastResort font is unavailable. I found that it is refers to a need to find a font literally named LastResort.ttf that Apple distributes. This font contains a category glyph for every valid Unicode character, so a reasonable "missing character" glyph can be displayed when the active font lacks one.

Apparently the web browser cannot access this font, even though it is present in /System/Library/Fonts. I haven't found out why. Also it doesn't appear in the list of font families you can fetch from the Qt font database (see previous post); and it doesn't appear in the list of fonts presented by the very useful PopChar product, nor in the list presented by PhotoShop's Text tool. So there's a conspiracy here to hide its name. Is the conspiracy at the Mac OS level? Probably, but I don't know.

I wanted a reliable failure example, but I walked back through the last half-dozen comic pages for every comic I browse, and none triggered the message. Rats! OK, what else?

Thinking about it, the problem can only occur if the default font lacks some Unicode glyph. US-based comics are unlikely to stray outside the Latin-1 set. I look at only a few Europe-based comics and even they haven't much reason to use non-Latin-1 letters except possibly the Euro symbol. Perhaps there was some ad copy that had a Euro sign, or an n-acute or -tilde, or a brand name in Cyrillic, and that ad isn't currently being included. But most fonts these days have at least the Latin-1 extended-A and the Greek and Cyrillic blocks covered. So why would the Webkit default font not have...

Oh, wait.

A couple weeks ago, I changed the list-of-comics widget to display in Comic Sans. You know, because it would be cute for a comic browser to list comic names in... And I think maybe...

Yurp. I also put in a couple of lines in the initialization of the QWebPage to set its default sans-serif font to the same Comic Sans. I even left in a comment "#Needed? Desirable?" Could this be a problem?

Actually, no, it shouldn't be. Checking the repertoire with Font Book I see that what I have on this machine, "Comic Sans MS", has a fairly complete set of glyphs including Cyrillic and a Euro, certainly everything that an ad might call upon.

Nevertheless, I took that out, letting the web page display default to whatever font it wants (it appears to be Times Roman), and now I just have to wait a while and see if the Last Resort message pops up again.

Later: It hasn't.

Getting useful info from QFontDatabase

I'll get back to the Last(goddam)Resort font issue shortly. But in learning about it I've had to look a little closer at what Qt knows about fonts, and as a result I've got a function that might be useful to others.

The QFontDatabase object contains whatever Qt knows about fonts that are "available in the underlying window system". However it makes this information available in a rather awkward way. You can call its families() method to get a list of all font names:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QFont, QFontDatabase
the_app = QApplication([])
f_db = QFontDatabase()
for family in f_db.families():
    print(family)

My macbook is rather over-endowed with fonts, and this prints a list of 238 names.

You can also ask for the recommended or default fixed and "general" fonts,

qf_fixed = f_db.systemFont(QFontDatabase.FixedFont)
print( 'Fixed font:',qf_fixed.family() )
qf_general = f_db.systemFont(QFontDatabase.GeneralFont)
print( 'General font:',qf_general.family() )

Which on my macbook prints

Fixed font: Monaco
General font: .Lucida Grande UI

Here's a surprise: what the heck is that leading dot doing in the name of the general font? There is definitely no font named ".Lucida Grande" in the list of families, nor displayed by the Fontbook app, nor (using find /System ".Lucida*') in the System folder. However, the font db will create a QFont for it:

>>> lgf = f_db.font('.Lucida Grande UI','',12)
>>> lgf.family()
'.Lucida Grande UI'
>>> lgf.pointSize()
13

So it's real in some sense, and your code can use it. Moving on...

You can ask the font db things about any single family name, for example isFixedPitch(family). But font family names aren't consistent from system to system. And often what you want is a font that meets certain criteria, such as: it's "Times" by some name, or it has a bold or italic variant, or it scales to 36pt. So I put together this little function that will return a (possibly empty) list of fonts that meet given criteria:

def font_filter( name = None, style = None, fixed = False, sizes = [], language = None ) :
    db = QFontDatabase()
    selection = db.families()
    if name :
        selection = [family for family in selection if name in family ]
    if style :
        selection = [family for family in selection if style in db.styles(family) ]
    if fixed :
        selection = [family for family in selection if db.isFixedPitch(family) ]
    if language :
        selection = [family for family in selection if language in db.writingSystems(family) ]
    if sizes :
        size_set = set(sizes)
        selection = [family for family in selection
                     if size_set.issubset( set( db.smoothSizes(family, '' ) ) ) ]
    return selection

For example,

>>> print( font_filter( name = 'Helvetica' ) )
['Helvetica', 'Helvetica CY', 'Helvetica Neue']
>>> print( font_filter( fixed = True ) )
['Anonymous Pro', 'Courier', 'Courier New', 'Inconsolata', 'PCMyungjo', 'PT Mono']
>>> print( font_filter(fixed = True, style='Bold', sizes=[12, 18]) )
['Courier', 'Courier New', 'PT Mono']
>>> font_filter(fixed=True, language=QFontDatabase.Hebrew)
['Courier New']

Next time: back to the issue of the Last(goddam)Resort font problem.

Last (goddam) Resort Font

I wrote CoBro back in 2012 with the original intent of "releasing" it for public use (i.e. making it known on the the comics subreddit) but because of QWebKit instabilities as well as problems packaging it with either PyInstaller or cx_freeze, I put that off until I could re-do it for Qt5 and Python 3.

Now I've done that it still is plagued by QWebKit issues (and still breaks cx_freeze, and PyInstaller still doesn't support Python 3). Among the instabilities is a very intermittent tendency to throw a blizzard of messages like this: 2014-03-14 15:06:59.270 Python[6570:d07] Critical failure: the LastResort font is unavailable.

This is a Python log message, not a Qt one that I could maybe stifle with the technique mentioned in the prior post. And "Critical" at that! So what's going on?

Turning to The Google I find that a lot of people encountered this message with a variety of different apps, but mostly back in 2010 or 2011 when Snow Leopard was new. But some information emerged.

What is the Last Resort font? The official explanation from Unicode.org is that it is a font to use when the system needs to print a Unicode glyph and all available fonts lack values for that code position.

if the font cannot represent any particular Unicode character, the appropriate "missing" glyph from the Last Resort font is used instead. This provides users with the ability to tell what sort of character it is, and gives them a clue as to what type of font they would need to display the characters correctly.

This explains why the message comes only rarely but in a flurry: it happens when a comic web page has a glyph that isn't in any available font. That doesn't happen all the time, but if it does, the site is likely to have a string of such glyphs; hence the multiple messages. But why can't the WebKit browser find the font? With a little poking around on my system (and absolutely no help from the Finder, whose search function insists there is nothing to match that name) I did find that /System/Library/Fonts/LastResort.ttf exists and had 644 permissions. So...

Finally I found this FontGeek post about Safari having the same problem. They claim that because of "sandboxing" the browser can't access the folder where LastResort resides. I'm dubious about the explanation; and the fix they offer isn't directly applicable to the Qt WebKit code as far as I can tell. But it's the best explanation I've seen.

What to do? One possibility: get a copy of LastResort.ttf and include it in the app. I've already got code in PPQT V1 to load a font bundled with the app and add it to the QFontDatabase. I need to think about this, also I need to find a web comic that will trigger the issue reliably.

Trapping Qt log messages

Using PyQt5, you can install your own handler for Qt's log messages and do with them as you wish, for example diverting them to a Python log file. There are (as usual) some surprises, but by and large, it works.

The context is CoBro, my little web-comic browser. It uses QWebKit to display single HTML pages. After displaying one particular comic (Two Guys and Guy) the webkit code likes to emit a couple of log messages like "error: Internal problem, this method must only be called once." Annoying since there is nothing you can do about it, and it doesn't seem to cause any harm. (BTW this is a known problem, see Qt Bug #30298.)

So I add the following code to Cobro, after creating the App and the main window and everything and we are just about ready to show the main window and enter the event loop:

    from PyQt5.QtCore import qInstallMessageHandler, QMessageLogContext
    from PyQt5.Qt import QtMsgType

    def myQtMsgHandler( msg_type, msg_log_context, msg_string ) :
        print('file:', msg_log_context.file)
        print('function:', msg_log_context.function)
        print('line:', msg_log_context.line)
        print('  txt:', msg_string)

    qInstallMessageHandler(myQtMsgHandler)

Now what comes out on stderr is this:

file: access/qnetworkreplyhttpimpl.cpp
function: void QNetworkReplyHttpImplPrivate::error(QNetworkReplyImpl::NetworkError, const QString &)
line: 1929
  txt: QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.

Ta-daaaa! We have intercepted a Qt log message and analyzed it to show where in the Qt code it originated. One surprise is that the "function" member of the QMessageLogContext object is not a simple function name, but the full C++ signature. Another surprise is that, when I stop on this code in a debugger and look at the msg_log_context item, its members are not strings but "sip reference" objects. Nevertheless by the magic of PyQt5 they print as strings.

Well, printing this bumf isn't a lot of use. What would be more useful, is to divert it into the Python log stream, like this:

    def myQtMsgHandler( msg_type, msg_log_context, msg_string ) :
        # Convert Qt msg type to logging level
        log_level = [logging.DEBUG,
                     logging.WARN,
                     logging.ERROR,
                     logging.FATAL] [ int(msg_type) ]
        logging.log(logging.DEBUG,
                    'Qt context file is '+msg_log_context.file
                    )
        logging.log(logging.DEBUG,
                    'Qt context line and function: {0} {1}'.format(
                        msg_log_context.line, msg_log_context.function)
                    )
        logging.log(log_level, 'Qt message: '+msg_string)

In other words, log the gritty details of the QMessageLogContext at the DEBUG level, but log its actual text at its own self-assigned severity, as translated into Python logging's values. The above code works and now I can redirect QWebKit's annoying messages into CoBro's log file.

Making a bad QTextCursor and a promising find

Working today on the unit testing and code finalization of pagedata, the module that keeps track of where the scanned OCR pages of a book each start. This module acts as the data model for several clients: every time the user moves the edit cursor, a bunch of different widgets will ask pagedata "which page is the cursor on now?" The imageview module asks it for the filename of the .png file to display, the scan# and folio# widgets under the edit window ask it for those items, and of course the Pages panel calls on it for the rows of data it displays as a table.

I thought I'd finalized the code but of course as soon as I started adding test calls into it from its unit-test module I found not only bugs but also things I hadn't thought of. It's schizophrenic, flipping back and forth between coding and testing. "What will it do if I throw it this?" the tester asks, and the coder is thinking "Oh shit, why didn't I plan for that?"

Anyway, one of the pieces of crap the test-monkey in me flang at the pagedata module opened a wonderful new prospect in Qt error control! It went down like this.

Important method in the PageData object is read_pages, which processes the page-data lines from the .meta file. When a book is saved, all the metadata goes in the bookname.meta file, including everything we know about page boundary locations. So at load time, read_pages gets called to rip through these saved lines and rebuild the page table as it was when the book was saved.

There are six items in each line, the first being the character offset to the start of the page. That gets turned into a QTextCursor so that Qt will maintain the position as it changes under user editing actions. The code is simple:

try:
    (P, fn, pfrs, rule, fmt, nbr) = line.split(' ')
    tc = QTextCursor(self.document)
    tc.setPosition(int(P))

and so forth. The test case had already flung a non-integer position P, and the failure of int(P) was caught by the try/except fine. So the next nastiness was a bad position value, first 1000000, much larger than the document, next -1. But neither of these tripped an exception! All that happened was that a message appeared on stderr, "QTextCursor::setPosition: Position '100000' out of range" and the QTextCursor was unchanged.

This opened two new questions: (1), how the heck can read_pages detect that it got a bad position?, and (B), how can we avoid having that message, about an error we've anticipated and dealt with, cluttering up stderr and getting the user all upset?

It was too late in the day to investigate (1), but (B) is a problem I've been plagued by for a long time. Qt is just full of unhelpful debugging errors. My other app, a simple web browser based on QWebKit, likes to throw out stuff like this:

QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.

And I've been wondering in a dazed sort of way if there mightn't be some way to stifle those. But PPQT2 is using proper Python logging. Every module creates its own logger and writes diagnostics, warnings and errors with the logging API. And I definitely plan not to let those log messages dribble out on stderr; they will eventually go into a file.

So it suddenly occurred to me, is there maybe some way to divert the Qt log messages into the Python logging system? I wasn't aware of anything but I started browsing in the index of the Qt Assistant and turned up this: qInstallMessageHandler. This appears to offer a way to accept and process all of Qt's messages.

I'm quite excited about this. If it is accessible under PyQt5, I see a clear path to capturing all Qt messages and converting them into Python log entries. That will let me stifle the WebKit chatter from my comics browser and also this text cursor message in PPQT. Tomorrow I spend the day at the museum but thursday afternoon I get to dig into this!