A First Look at Linguist

So, where were we? I've been away nearly a week, visiting wonderful Ames, Iowa. Not by choice, but by the whimsy of the NCAA Selection Committee, which in its wisdom chose to make the Stanford Women's Basketball team not only a number-2 seed, but make them play the first two rounds in Ames, on the campus of the University of Iowa, where the host school, the 4th-seeded Iowa Cyclones, draw 10,000 screaming fans to every home game.

Well, fortunately the Cyclones faded to a zephyr before the defense of the FSU Seminoles, so for the second game the Cardinal played in front of a half-empty and subdued arena, and won comfortably. Meanwhile we dealt with snow flurries and the difficulties of passing time among the limited amusements of Ames and Des Moines. If you'd rather know about that versus Qt Linguist, check the pictures.

A couple posts back I described the process of designing a widget with Qt Designer and how any string in the design could be designated "Translatable", and how that left distinct code in the generated Python of the widget class. With the result that, when the widget initializes itself, every translatable string will pass through the bowels of the QtCore.QCoreApplication.translate method before being assigned to its QLabel, push button, menu item or whatever its use.

The output of that method—usually just written tr() in the Qt documentation, but for an arcane reason having to do with the relationship of Python classes to C++ classes, PyQt5 needs to always call the Core version not the one inherited by every QObject—the method's output is either the original, or a translated string—if there exists a translation for that string for the current Locale.

But that leaves the question, where do translations come from? From work done by a Translator (a human) using Qt Linguist. I pursued the link between the widget code and Linguist a little further.

The bridge is the PyQt5 utility, pylupdate5. Its use is described in the PyQt5 online docs. One must create a minimal Qt project description file, in this case ppqt2.pro. Actually a make file for the Qt Make program, this file lists the relevant source files and the name of the translation file. Here is what I used:

SOURCES = editview_uic.py
TRANSLATIONS = ppqt2.ts

Listing just one source file now; later there would be many on that line.

Then you turn pylupdate5 loose on the .pro file and it fills up ppqt2.ts with a bunch of XML items like this:

    <message>
        <location filename="editview_uic.py" line="153"/>
        <source>Document filename</source>
        <translation type="unfinished"></translation>
    </message>

Now I could launch Qt Linguist from the Qt distribution, and use it to open the ppqt2.ts file. It presents me with a window whose top is like this:

Every string from every widget (just one widget for now) is shown. Click on one and prepare a translation for it in the bottom part of the window.

For some reason, spaces are shown as gray dots in this part of the window. There exist Qt "phrase books" for many languages, and the French one is open in the above image. It is offering "document" as a translation for "document". Fair enough, but I would have thought "document name" would be a common phrase. Apparently not. I typed in nom du document.

Anyway, that's what the Translator person works with. The texts for the given language would be saved back into the ppqt2.ts file. And somehow become available via the Core Translate method at run-time.

I'm not going to worry further about that last step, for now. I can see how translation would be done. I don't mean to actually do any translations (or request anyone to do any) until the whole app is in near-final state. But at least I know how it all works, I've seen it can work on my system, so that's one Unknown that's Known and I can relax about it.

Assisting the Upgradement

I got burned by one of what turned out to be quite a list of small incompatibilities between PyQt4 and PyQt5. Just fooling around I tried upgrading one of Mark Summerfield's utilities to PyQt5. It contained the following code:

        path = QFileDialog.getOpenFileName(self,
                "Make PyQt - Set Tool Path", label.text())
        if path:
            label.setText(QDir.toNativeSeparators(path))

Pretty obviously Mark expected getOpenFileName to return a path or a null string. But when executed, and I clicked Cancel in the file dialog, it caused an error in the label.setText statement. Whatever got into path evaluated to True, but wasn't a string.

It turned out to be a tuple with two strings. I documented this to the pyqt mailing list and was embarrassed when Phil just replied with the above link to the list of incompatibilities, one of which is a change to the API of the whole family of five "get..." methods supported by QFileDialog. What had happened to cause this seemingly arbitrary breaking of an existing API? It seems that PyQt4 had introduced some variant methods "to avoid the need for mutable strings". Now these extra "get...Filter" methods were being dropped and their function folded into the basic "get..." methods. And that entailed changing the return value of getOpenFileName from a simple string to a tuple of two strings.

It still seems arbitrary to me, breaking existing code in an unexpected way for no very good reason. But it's a done deal, so how to make sure that this incompatibility, and all the other subtle incompatibilities in the list, don't get overlooked? (And don't miss the fact that one item in the list is open-ended, saying "PyQt5 does not support any parts of the Qt API that are marked as deprecated or obsolete in Qt v5.0." What are those? Are they numerous?)

I decided it wouldn't be hard to write a tool to find and point out all, or anyway a lot of, these issues. In two afternoons of work I put together q45aide.py (click the link to see the Readme and get the code from Github). This is a straightforward source scanner that copies a program and inserts comments above any line that looks as if it will have an upgrade problem.

I'm particularly pleased with two features of this program. One is the way of finding out the modules that contain every Qt class. I needed this because one annoying change from Qt4 to Qt5 is that many classes moved from one import module to another. That invalidates most existing from PyQt4.module import (class-list) statements. I wanted to generate correct, minimal import statements from the class-names used in the program. But that meant having a dictionary whose keys were all the valid Qt class-names (over 880 of them, it turns out) and whose values were the module names that contain them.

I pondered quite a while over how to get such a list of class-names by module. I thought about manually or programatically scraping some pages from qt-project.org. But finally I realized, I could build a complete, accurate list dynamically in the program.

When you import a module, Python creates a namespace. And the names defined in a namespace can be interrogated by querying namespace.__dict__. So the program contains code like this:

    def load_namespace( ):
        global module_dict, import_dict
        # pick off QtXxxx from "PyQt5.QtXxxx"
        module_name = namespace.__name__.split('.')[1]
        for name in namespace.__dict__ :
            if name.startswith('Q') : # ignore e.g. __file__
                module_dict[name] = module_name

    import PyQt5.Qt as namespace ; load_namespace()
    import PyQt5.QtBluetooth as namespace ; load_namespace()
    ...

This loads module_dict with exactly the 880+ class-names related to their include modules, automatically updating should PyQt5 be updated with new or changed class-names.

The other thing that I got a kick out of writing was the way to write a list of class-names in either of two formats, in one statement. The program will generate one "from PyQt5.modulename import (class-name-list)" for each module that the input requires. A program option is -v, asking for the list to be stacked vertically. The only difference is that the class-name-list is either punctuated with comma-space, or with comma-newline-indent. And this is how it comes out:

                out_file.write('from PyQt5.{0} import\n   ('.format(mod_name))
                join_string = ',\n    ' if arg_v else ', '
                out_file.write(join_string.join(sorted(class_set)))
                out_file.write(')\n')

Badda-boom.

Using Qt Designer

Two unknowns I was fretting about have become less foggy: the work-flow that connects the graphic layout one creates with Qt Designer to the executable code; and the connection between user-visible strings defined to Qt Designer and the Qt translation mechanism.

Qt Designer to code

This part was of course laid out clearly in Summerfield's book. That book was my bible through the early days of building PPQT in 2011-12. Now it shows its age a little because there are minor differences between the PyQt4/Python 2 syntax of its examples, and the PyQt5/Python 3 syntax I'm using. But chapter 7, "Using Qt Designer", covered it all. Here's the sequence.

Run Qt Designer, select a template (in this case, just plain QWidget), and start dragging widgets onto it and laying them out. It's reasonably intuitive, especially if you know the names and uses of most of the widgets and their properties, as I do. It helps a lot to have a big screen. Qt Designer is almost unusable on the macbook because with all its various windows there's no room left for the widget you're designing. I used my desktop system with a 23-inch monitor and it was fine. It took an hour to lay out a satisfactory Edit panel as I described previously. Much of that time was spent in the Properties Editor, checking and specifying and re-specifying the many, many properties of each widget.

In the course of this I quickly discovered part of the answer to the question on translation. The Property Editor entry for any user-visible string—label text, tool-tip, status-tip, whats-this?—has a check-box "Translatable". I checked most of them. This widget has two QLabels and both will have their text filled in dynamically. But all tool-tips need to be translatable.

Having the check-box there keeps one aware that the English text you compose now will have to be translated. My experience in writing for translation, and writing tech material for people who have English as a second language, goes way back, to 1975-6 when I was at IBM World Trade in London and writing material to be read colleagues who were Brits, Swedes, Dutch and Italians. I learned then to keep going over my text to make sure it was simple, terse, and unambiguous; used no colloquialisms or metaphors; used the smallest vocabulary that would express the thought. (Which isn't a bad mindset for any expository writing.)

You save your design to a file of type .ui and invoke pyuic5, a command-line utility that reads it and writes some python source. (Summerfield invokes it by way of a version of make, makepyqt.pyw, but I don't see that in my installation. TB investigated.) The contents of this source were at first a little baffling to me. Here's the start:

class Ui_EditViewWidget(object):
    def setupUi(self, EditViewWidget):
        EditViewWidget.setObjectName("EditViewWidget")
        # ...and 140 more lines of setting-up code such as
        self.DocName = QtWidgets.QLabel(self.frame)
        # ...and the other sub-widgets...

What do I do with this? I wondered. Do I need to instantiate an object of this class? But it doesn't have an __init__; and what's this EditViewWidget being passed to this setupUi method?

Read the manual, doofus. Or in this case, keep reading in chapter 7 of Summerfield.

What I am supposed to do with this is to invoke it in—for the first time in my Pythonic career—a multiple-inheritance class definition, like this:

class EditView(QWidget, Ui_EditViewWidget):
    def __init__(self, my_book, parent=None):
        super().__init__(parent) # initialize QWidget
        self.setupUI(self) # invoke the initializer in class Ui_EditViewWidget

I am creating a QWidget subclass that also incorporates the code prepared by pyuic5. Following the call to setupUI, "self" incorporates all the widgets I defined to Qt Designer, under the object names I gave them in the Properties editor. Further initialization code is needed, for example to connect signals, set up the syntax highlighter, etc. But 150-odd lines of detailed GUI initialization are taken care of separately and more important, can be reviewed and altered at any time with Qt Designer, with no impact on the code.

Translation de-fog

To implement translation, the setupUi method ends with this:

    def retranslateUi(self, EditViewWidget):
        _translate = QtCore.QCoreApplication.translate

...which is followed by a line like this for every string that got the "Translatable" tick-mark:

        self.DocName.setToolTip(_translate("EditViewWidget", "Document filename"))

That pushes the fog of unknowns back a bit. It remains for me to learn how translation actually works.

Unknown Unknowns

Yesterday I checked off pagedata as coded and tested. That's the last remaining support or background module needed to allow the editor to run. So the next thing to tackle is editview.py, the visual face of the editable document. I imagine this as a widget containing, principally, the QPlainTextEdit, and below it a bar with five items:

• A QLabel containing the filename of the document. This will change its font style with the document's modified status, becoming perhaps bold and magenta when a save is needed (for the document or for metadata).
• A QLabel with the current folio number—a label because it isn't changeable by the user; folios are changed by modifying the folio rules in the Pages panel.
• A numeric text entry field with room for four digits, displaying the scan image number corresponding to the cursor position. Editable; you can type a new number to effect a jump.
• A numeric text entry field with room for 6 digits, displaying the current text block (line) number, updating as the cursor moves. Again, type a new number to jump in the document.
• A QLabel with the current "column" number in the current line. Not editable. (Use an arrow key or just click.)

The latter four items of course update dynamically as the cursor moves, on receipt of the cursor movement signal from the editor.

So, if I understand all this (not difficult given it's almost the same as the status bar area of the PPQT main window), where are the Unknown Unknowns of the title? Hah. Well, they are actually known, at least by category, but just the same I feel anxious about launching into this phase. They are:

1. Using Designer

The Qt Designer is a graphic tool for designing a layout. I played with it a bit a couple of years ago when starting PPQT but ended up doing all my widget layouts "by hand" with explicit code like this (one of the simplest ones)

        vbox = QVBoxLayout()
        # the image gets a high stretch and default alignment, the text
        # label hugs the bottom and doesn't stretch at all.
        vbox.addWidget(self.txLabel,0,Qt.AlignBottom)
        vbox.addWidget(self.scarea,10)
        vbox.addLayout(zhbox,0)
        self.setLayout(vbox)

It makes the __init__() rawther lengthy. With Qt Designer you supposedly separate your UI design from the code. Designer saves a file of UI info; you apply a PyQt utility to convert this into something Python can execute; you import it and execute it. Covered in Summerfield's book and in the Qt docs. But I have lots of questions, like: how are signals connected between elements; how are elements connected to the methods that update them; how do label texts set in Designer get tr() translated; just generally a fog of unknowns. But I'd like to give it a try and the editview widget should be a good test case.

I18N

Does anybody use that term any more? "I18N" was a thing back in the 80s, late 70s even at IBM. (It means "Internationalization", duh.) Anyway, I've committed to PGDP Canada that PPQT2 will be translatable. Meaning every damn user-visible text string has to be wrapped in a tr() call. And editview is the first module that has user-visible strings (log messages don't count). The tr() call is all I know about. I am anxious about the whole rest of Qt's I18N system. How do the tr'd strings get collected; how does a translator create an alternate translation; will I have to start using Qt Make; what the heck is Locale and how do I control it for testing purposes... gaaahhh. Much reading to do.

GUI Unit Tests

With this first UI module I enter the world of automated UI testing and I haven't a clue. Well, one clue: QTest. There's a multi-chapter writeup on QTest and simulation of GUI events. I presume I'll use that. There are third-party packages like FrogLogic's "Squish" but they are very expensive, at least that on is. There are open-source packages for test automation but the ones I've seen are single-platform. So I suppose I'll be rolling my own using QTest. But I really have no idea.

So: the known Unknowns are just packed with unknown unknowns. I will be learning as I go, and I will be posting what I learn to this blog. Because that's one way I have of consolidating what I've learned. You're welcome.

Bypassing the LastResort font error

As I noted previously, my mini web browser Cobro had begun occasionally dumping flurries of a message Python[6570:d07] Critical failure: the LastResort font is unavailable. I found that it is refers to a need to find a font literally named LastResort.ttf that Apple distributes. This font contains a category glyph for every valid Unicode character, so a reasonable "missing character" glyph can be displayed when the active font lacks one.

Apparently the web browser cannot access this font, even though it is present in /System/Library/Fonts. I haven't found out why. Also it doesn't appear in the list of font families you can fetch from the Qt font database (see previous post); and it doesn't appear in the list of fonts presented by the very useful PopChar product, nor in the list presented by PhotoShop's Text tool. So there's a conspiracy here to hide its name. Is the conspiracy at the Mac OS level? Probably, but I don't know.

I wanted a reliable failure example, but I walked back through the last half-dozen comic pages for every comic I browse, and none triggered the message. Rats! OK, what else?

Thinking about it, the problem can only occur if the default font lacks some Unicode glyph. US-based comics are unlikely to stray outside the Latin-1 set. I look at only a few Europe-based comics and even they haven't much reason to use non-Latin-1 letters except possibly the Euro symbol. Perhaps there was some ad copy that had a Euro sign, or an n-acute or -tilde, or a brand name in Cyrillic, and that ad isn't currently being included. But most fonts these days have at least the Latin-1 extended-A and the Greek and Cyrillic blocks covered. So why would the Webkit default font not have...

Oh, wait.

A couple weeks ago, I changed the list-of-comics widget to display in Comic Sans. You know, because it would be cute for a comic browser to list comic names in... And I think maybe...

Yurp. I also put in a couple of lines in the initialization of the QWebPage to set its default sans-serif font to the same Comic Sans. I even left in a comment "#Needed? Desirable?" Could this be a problem?

Actually, no, it shouldn't be. Checking the repertoire with Font Book I see that what I have on this machine, "Comic Sans MS", has a fairly complete set of glyphs including Cyrillic and a Euro, certainly everything that an ad might call upon.

Nevertheless, I took that out, letting the web page display default to whatever font it wants (it appears to be Times Roman), and now I just have to wait a while and see if the Last Resort message pops up again.

Later: It hasn't.

Getting useful info from QFontDatabase

I'll get back to the Last(goddam)Resort font issue shortly. But in learning about it I've had to look a little closer at what Qt knows about fonts, and as a result I've got a function that might be useful to others.

The QFontDatabase object contains whatever Qt knows about fonts that are "available in the underlying window system". However it makes this information available in a rather awkward way. You can call its families() method to get a list of all font names:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QFont, QFontDatabase
the_app = QApplication([])
f_db = QFontDatabase()
for family in f_db.families():
    print(family)

My macbook is rather over-endowed with fonts, and this prints a list of 238 names.

You can also ask for the recommended or default fixed and "general" fonts,

qf_fixed = f_db.systemFont(QFontDatabase.FixedFont)
print( 'Fixed font:',qf_fixed.family() )
qf_general = f_db.systemFont(QFontDatabase.GeneralFont)
print( 'General font:',qf_general.family() )

Which on my macbook prints

Fixed font: Monaco
General font: .Lucida Grande UI

Here's a surprise: what the heck is that leading dot doing in the name of the general font? There is definitely no font named ".Lucida Grande" in the list of families, nor displayed by the Fontbook app, nor (using find /System ".Lucida*') in the System folder. However, the font db will create a QFont for it:

>>> lgf = f_db.font('.Lucida Grande UI','',12)
>>> lgf.family()
'.Lucida Grande UI'
>>> lgf.pointSize()
13

So it's real in some sense, and your code can use it. Moving on...

You can ask the font db things about any single family name, for example isFixedPitch(family). But font family names aren't consistent from system to system. And often what you want is a font that meets certain criteria, such as: it's "Times" by some name, or it has a bold or italic variant, or it scales to 36pt. So I put together this little function that will return a (possibly empty) list of fonts that meet given criteria:

def font_filter( name = None, style = None, fixed = False, sizes = [], language = None ) :
    db = QFontDatabase()
    selection = db.families()
    if name :
        selection = [family for family in selection if name in family ]
    if style :
        selection = [family for family in selection if style in db.styles(family) ]
    if fixed :
        selection = [family for family in selection if db.isFixedPitch(family) ]
    if language :
        selection = [family for family in selection if language in db.writingSystems(family) ]
    if sizes :
        size_set = set(sizes)
        selection = [family for family in selection
                     if size_set.issubset( set( db.smoothSizes(family, '' ) ) ) ]
    return selection

For example,

>>> print( font_filter( name = 'Helvetica' ) )
['Helvetica', 'Helvetica CY', 'Helvetica Neue']
>>> print( font_filter( fixed = True ) )
['Anonymous Pro', 'Courier', 'Courier New', 'Inconsolata', 'PCMyungjo', 'PT Mono']
>>> print( font_filter(fixed = True, style='Bold', sizes=[12, 18]) )
['Courier', 'Courier New', 'PT Mono']
>>> font_filter(fixed=True, language=QFontDatabase.Hebrew)
['Courier New']

Next time: back to the issue of the Last(goddam)Resort font problem.

Last (goddam) Resort Font

I wrote CoBro back in 2012 with the original intent of "releasing" it for public use (i.e. making it known on the the comics subreddit) but because of QWebKit instabilities as well as problems packaging it with either PyInstaller or cx_freeze, I put that off until I could re-do it for Qt5 and Python 3.

Now I've done that it still is plagued by QWebKit issues (and still breaks cx_freeze, and PyInstaller still doesn't support Python 3). Among the instabilities is a very intermittent tendency to throw a blizzard of messages like this: 2014-03-14 15:06:59.270 Python[6570:d07] Critical failure: the LastResort font is unavailable.

This is a Python log message, not a Qt one that I could maybe stifle with the technique mentioned in the prior post. And "Critical" at that! So what's going on?

Turning to The Google I find that a lot of people encountered this message with a variety of different apps, but mostly back in 2010 or 2011 when Snow Leopard was new. But some information emerged.

What is the Last Resort font? The official explanation from Unicode.org is that it is a font to use when the system needs to print a Unicode glyph and all available fonts lack values for that code position.

if the font cannot represent any particular Unicode character, the appropriate "missing" glyph from the Last Resort font is used instead. This provides users with the ability to tell what sort of character it is, and gives them a clue as to what type of font they would need to display the characters correctly.

This explains why the message comes only rarely but in a flurry: it happens when a comic web page has a glyph that isn't in any available font. That doesn't happen all the time, but if it does, the site is likely to have a string of such glyphs; hence the multiple messages. But why can't the WebKit browser find the font? With a little poking around on my system (and absolutely no help from the Finder, whose search function insists there is nothing to match that name) I did find that /System/Library/Fonts/LastResort.ttf exists and had 644 permissions. So...

Finally I found this FontGeek post about Safari having the same problem. They claim that because of "sandboxing" the browser can't access the folder where LastResort resides. I'm dubious about the explanation; and the fix they offer isn't directly applicable to the Qt WebKit code as far as I can tell. But it's the best explanation I've seen.

What to do? One possibility: get a copy of LastResort.ttf and include it in the app. I've already got code in PPQT V1 to load a font bundled with the app and add it to the QFontDatabase. I need to think about this, also I need to find a web comic that will trigger the issue reliably.