Not Dead, the Blog

Apologies for the gap between posts. A couple of good coding sessions and I nearly have the Book and Main Window modules under control. A bit of tidying and I will have something to show in a day or so.

In the meantime, while working with the tedious and complex business of opening a file, I updated my improved QTextStream class. This started as a simple work-around of the annoying problem that a QTextStream, when based on a QFile, does not take ownership of the QFile. If the Python programmer carelessly lets the QFile go out of scope, the next attempt to use the QTextStream: what? produces a log message? returns an error code? Oh no no! It seg-faults the Python interpreter.

But in the course of using the class several times and ways in the main window code I found it expedient to keep adding simple functions to it, so as to avoid having to create nonce QFileInfo objects. Here's it's latest incarnation.

class FileBasedTextStream(QTextStream):
    def __init__(self, qfile):
        super().__init__(qfile)
        self.saved_file = qfile
        self.qfi = None # may never need this
    def rewind(self):
        self.seek(0)
    def writeLine(self, str):
        self << str
        self << '\n'
    def basename(self):
        if self.qfi is None:
            self.qfi = QFileInfo(self.saved_file)
        return self.qfi.baseName()
    def filename(self):
        if self.qfi is None:
            self.qfi = QFileInfo(self.saved_file)
        return self.qfi.fileName()
    def filepath(self):
        if self.qfi is None:
            self.qfi = QFileInfo(self.saved_file)
        return self.qfi.absolutePath()

Joys of Parenthood

So I was doing some test spadework before launching into a lot of code in the mainwindow module, just to make sure I actually could actually accomplish what I want to accomplish. And I found something that puzzled me a lot, and taught me something.

The basic UI for PPQT2 is a refinement of the previous version: a single window divided into two horizontal panes by a splitter. In V.1 the left pane is the single edit window. The right pane contains a tab-set with a bunch of activity panels. There's the image view panel that shows the OCR scan images, the notes panel where the user can jot notes about the project, the find/replace panel, etc. etc.

So in V.2, the left pane will be not just one edit widget, but a tabset that might contain one edit widget, or two, or however many documents the user wants to open. But what about the right pane? The activities there are used with all documents, but they contain document-unique items. The image view should show the scan images for the document that has the focus. The notes panel should show the notes for the document that has the focus. And so on.

Whenever the user selects a different edit widget from the tabset on the left, all the panels on the right need to change their contents to match. (Only one of eight needn't change when a different document comes into focus, the Help panel.) I thought a bit about how to make this happen.

One way not to do it, would be to design the different view objects, imageview and pageview and notesview and so on, to have some way to switch to a different model object. "Hey, image view, use the pagedata from this other book; hey words view, use the word counts from this other book, etc." Although conceptually a reasonable use of the model/view paradigm, this would complicate the coding of those view classes, and worse, has a performance implication. Some of the views (word view especially) consist of QTables with thousands of rows, and these can take a long time to repopulate. Switching from one 15,000-word vocabulary to a different 15,000 word vocabulary (even if the word lists were substantially the same, as in switching between two versions of the same book), could cause an unpleasant delay. Plus, there are more things than just the data contents that have to change. Perhaps in one book the user has sorted the words table ascending, case-sensitive, and in another book, descending and ignore-case and filtered to show only spellcheck errors. All the settings of these incidental controls need to switch, smoothly and quickly, when the user moves from one document to another.

So what I'd planned to do was to represent each open document with a Book object; and have that Book be the owner of a complete set of objects related to that document, models and views alike. It will own a notes view, an image view, everything—even its own Find/Replace panel, because the user will have set different options and have different lists of recent find/replace strings, for that document.

When the main window finds out that the edit view of a given document has received the focus, the main window needs to go to that document's book, and replace all the activity tabs on the right with the corresponding objects for the book where the user just clicked. They would be sitting in memory, all initialized and ready to show, so this should not take much time. But is this kind of wholesale replacement of tab contents feasible? I wrote a little test program to find out. And here it is in its entirety. Read it; copy it and run it if you like.

from PyQt5.QtCore import Qt, QPoint, QSize
from PyQt5.QtWidgets import QLabel, QMainWindow, QSplitter, QTabWidget

class mypanel(QLabel):
    def __init__(self, parent, text):
        super().__init__(text,None) #<-- see discussion
        self.owner = parent
        self.setFocusPolicy(Qt.StrongFocus)

    def focusInEvent(self,event):
        event.accept()
        self.owner.i_got_focus(int(self.text()))

class MainWindow(QMainWindow):
    def __init__(self, settings=None):
        super().__init__()
        self.editview_tabset = QTabWidget()
        self.panel_tabset = QTabWidget()
        # Create the splitter that contains the above two parts.
        self.splitter = QSplitter(Qt.Horizontal)
        self.splitter.addWidget(self.editview_tabset)
        self.splitter.addWidget(self.panel_tabset)
        # Set that splitter as the main window central (and only) widget
        self.setCentralWidget(self.splitter)
        self.resize(QSize(600, 600)) # make a reasonable size
        # Make some widgies
        self.pinks = []
        for j in range(4):
            self.pinks.append( mypanel(self, str(j)) )
            self.pinks[j].setStyleSheet("color:'#000080'; background:'#e0c0c0';")
        self.blues = []
        for j in range(4):
            self.blues.append( mypanel(self, str(j+4)) )
            self.blues[j].setStyleSheet("color:'#600000'; background:'#00e0d0';")
        # On the left, one pink and one blue with pink one active
        self.editview_tabset.addTab(self.pinks[0],self.pinks[0].text())
        self.editview_tabset.addTab(self.blues[0],self.blues[0].text())
        self.editview_tabset.setCurrentIndex(0)
        # On the right, two pinks to match the focussed one on the left
        self.panel_tabset.addTab(self.pinks[2],self.pinks[2].text())
        self.panel_tabset.addTab(self.pinks[3],self.pinks[3].text())

    def i_got_focus(self,num):
        print('num=',num) # document what's happening
        self.panel_tabset.setUpdatesEnabled(False)
        if num == 0 : # pink left widget got focus, make right side pink
            self.panel_tabset.removeTab(0)
            self.panel_tabset.insertTab(0,self.pinks[2],self.pinks[2].text())
            self.panel_tabset.removeTab(1)
            self.panel_tabset.insertTab(1,self.pinks[3],self.pinks[3].text())
        elif num == 4 : # blue left widget got focus, make right side blue
            self.panel_tabset.removeTab(0)
            self.panel_tabset.insertTab(0,self.blues[2],self.blues[2].text())
            self.panel_tabset.removeTab(1)
            self.panel_tabset.insertTab(1,self.blues[3],self.blues[3].text())
        self.panel_tabset.setUpdatesEnabled(True)

from PyQt5.QtWidgets import QApplication
app = QApplication([])
mw = MainWindow()
mw.show()
app.exec_()

If you run this you see a window divided in half. If you click on the left, nothing much happens (except "num=0" prints on stdout). If you select the other tab on the left, though, the panels on the right turn blue. Select the first tab on the left, the right turns pink. So this small-scale version works: when one of the two tabs on the left receive a focusInEvent, they call the main window's i_got_focus() method, which switches all two of the tabs on the right.

There are some oddities that you can discover if you experiment (put the focus on the right then click on a tab, not a window, on the left) but I think I see how to fix them. What I want to talk about in this post is a bug that had me puzzled for an hour.

Look at the initialization of the mypanel class where it says "see discussion". Originally it read:

        super().__init__(text,parent)

That's automatic Qt boilerplate that I wrote without even thinking. Change the above program to read that way, and run it. What do you see?

In the upper left corner of the window, overlapping the corner of the splitter, is a small blue rectangle with the number 7. What?!?

Real Qt mavens know immediately what has happened, but it took me a bit of head-scratching. In the main window initialization, when each of eight instances of mypane are created, the main window passes self. The panel widgets need that reference if they are to call the main window's i_got_focus method. But I was also passing it as the "parent" argument to the QLabel initializer. So each of the eight Qlabels was a child of the main window.

Being a parent of a widget means, among other things, that you display that widget. Four of the eight widgets are added to the tabsets, and that re-parents them to their tab widget. But pink 3 and blues 5, 6 and 7 remain parented by the main window. It displays them in the default location, upper left. Blue 7 is created last so it goes in front, hiding the others. Passing no-parent to QLabel fixed the problem.

So just as in life, things go better if you choose the right parents.

Remembering PEEK and POKE

Remember PEEK and POKE? Back in the Precambrian (or pre-PC) era of computing, the primary way to program the Apple and TRS-80 and Commodore machines was with BASIC. And really the only way to do the cool stuff was to get in and diddle system memory and hardware registers. The way to read memory was PEEK int, which returned the value of the byte at int. And with POKE you could store a value into a byte of memory. Ahhh... good times...

Well, crazily enough, PyQt offers an equivalent: a way to get direct access to data in memory. It's called a voidptr and is a type defined by SIP, PyQt's interface to the C++ world of Qt. You can get a voidptr from PyQt wherever the Qt documentation says a method returns uchar *. Such places are not common, but one emerged when I first implemented PPQT's scan image display.

A major feature of PPQT is that it shows the scan image from which the book's text was OCR'd, alongside the text itself in the editor window. As you move the edit cursor from page to page, the image viewer tracks it, flipping from scan image to image. It's one feature that PPQT has over its predecessor Guiguts, which requires the user to have a separate image-display app.

Here is the V2 imageview undergoing unit test. As you see it offers an adjustable zoom. Sometimes, especially when proofing Greek or a small-print footnote, the user needs to zoom in and peer closely. But usually you just want the whole page visible so you can scan for italics and bold, or check hyphenations.

The To-Width and To-Height buttons are supposed to set the zoom automatically so that the printed part of the page just fills the window side-to-side or top-to-bottom. When I first implemented these buttons back in V.1 I found it quite the coding challenge.

Here's what To-Width has to do:

1. Scan the pixels of the image to find the width of the nonwhite area.
2. Get the ratio of that width to the image's viewport width.
3. Set that ratio as the zoom factor and redraw the image.
4. Set the scroll position of the scroll area so as to center the nonwhite block.

Implementing these steps took me into back-alleys of Qt where I'd never been before, and introduced me to the SIP voidptr. In order to do step 1, I have to inspect the pixels of the image row by row, looking for the left and right edges of the text. I make sure the image is in the 8-bit Indexed Color mode, so that each pixel is one byte. Then the method QImage.bytes() returns a uchar * pointing to the bytes/pixels that comprise the image. The PyQt translation of uchar * is SIP.voidptr.

You can't use the voidptr as returned. First you must define the amount of memory it represents:

    vp = my_qimage.bytes()
    vp.setsize(my_qimage.width() * my_qimage.height())

Now that PyQt knows the bounds of the addressed memory—and one wonders: does it check the size; or would it let you define a very large size and potentially examine things you shouldn't?—anyway once you have set the bounds, you can index the voidptr as if it were a Python bytes string:

    if vp[j] < b'\xff': # nonwhite pixel

Which is exactly PEEK j! plus ça change, plus c'est la même chose...

Next post: performance tuning, or how fast can you finger the pixels?

Case of the Shredded Data

A persistent source of PyQt bugs is the problem that as soon as a variable "goes out of scope"—that is, cannot be referenced from any statement—it gets garbage-collected and either re-used or made unreachable by the virtual memory hardware. Newbies to PyQt get bitten by this early, often, and hard. It usually shows up as a segmentation fault that takes down the Python interpreter and your app. And there's no obvious bread-crumb path back to the problem.

The usual problem is that you build one object A based on another object B. Then you pass object A around and try to use it in another part of the program. Meanwhile, object B, which was just input material to making A, has gone out of scope and been shredded. Then any use of A references non-existent memory and segfaults, or produces weird results because it is accessing memory that doesn't contain what it should.

Case in point: QTextStream. This is a useful class, very handy for reading and writing all kinds of files. You could use the Python file class instead, but you need to standardize on one paradigm or the other, Python's files or Qt's QTextStreams. And I've gone with the latter, but they have this little problem of segfaulting if you are not careful with them.

A QTextStream is built upon some source of data, either a QFile or an in-memory QByteArray. The class constructor takes that source object as its only argument, as in this perfectly innocuous function:

def get_a_stream(path_string):
    '''Return a QTextStream based on a path, or None if invalid path'''
    if not QFile.exists(path_string):
        return None
    a_file = QFile(path_string)
    if not a_file.open(QIODevice.ReadOnly):
        return None
    return QTextStream(a_file)

Lovely, what? Simple, clear—and wrong. Because a_file goes out of scope as soon as the function returns. Many statements away in another part of the program, the next use of the returned stream crashes the program. This is (in my oh-so-humble opinion) a stupid design error in Qt (it affects C++ users too) but fortunately it is easy to work around. You just use the following instead of QTextStream:

class FileBasedTextStream(QTextStream):
    def __init__(self, qfile):
        super().__init__(qfile)
        self.save_the_goddam_file_from_garbage_collection = qfile

That's it! An object of this class FileBasedTextStream can be used anywhere a QTextStream would be used, but it does not require you to find some way to save the QFile from the garbage collector. The single reference to the QFile keeps it alive until the stream object itself is freed.

I solved this same issue earlier, for memory-based text streams. These are very handy for nonce files, and my unit test code builds lots of them.

class MemoryStream(QTextStream):
    def __init__(self):
        # Create a byte array that stays in scope as long as we do
        self.buffer = QByteArray()
        # Initialize the "real" QTextStream with a ByteArray buffer.
        super().__init__(self.buffer)
        # The default codec is codecForLocale, which might vary with
        # the platform, so set a codec here for consistency. UTF-16
        # should entail minimal or no conversion on input or output.
        self.setCodec( QTextCodec.codecForName('UTF-16') )
    def rewind(self):
        self.seek(0)
    def writeLine(self, str):
        self << str
        self << '\n'

It's just a QTextStream based on an in-memory buffer, but the buffer can't go out of scope as long as the object exists. It adds a couple of minor features that QTextStream lacks.

Bypassing the LastResort font error

As I noted previously, my mini web browser Cobro had begun occasionally dumping flurries of a message Python[6570:d07] Critical failure: the LastResort font is unavailable. I found that it is refers to a need to find a font literally named LastResort.ttf that Apple distributes. This font contains a category glyph for every valid Unicode character, so a reasonable "missing character" glyph can be displayed when the active font lacks one.

Apparently the web browser cannot access this font, even though it is present in /System/Library/Fonts. I haven't found out why. Also it doesn't appear in the list of font families you can fetch from the Qt font database (see previous post); and it doesn't appear in the list of fonts presented by the very useful PopChar product, nor in the list presented by PhotoShop's Text tool. So there's a conspiracy here to hide its name. Is the conspiracy at the Mac OS level? Probably, but I don't know.

I wanted a reliable failure example, but I walked back through the last half-dozen comic pages for every comic I browse, and none triggered the message. Rats! OK, what else?

Thinking about it, the problem can only occur if the default font lacks some Unicode glyph. US-based comics are unlikely to stray outside the Latin-1 set. I look at only a few Europe-based comics and even they haven't much reason to use non-Latin-1 letters except possibly the Euro symbol. Perhaps there was some ad copy that had a Euro sign, or an n-acute or -tilde, or a brand name in Cyrillic, and that ad isn't currently being included. But most fonts these days have at least the Latin-1 extended-A and the Greek and Cyrillic blocks covered. So why would the Webkit default font not have...

Oh, wait.

A couple weeks ago, I changed the list-of-comics widget to display in Comic Sans. You know, because it would be cute for a comic browser to list comic names in... And I think maybe...

Yurp. I also put in a couple of lines in the initialization of the QWebPage to set its default sans-serif font to the same Comic Sans. I even left in a comment "#Needed? Desirable?" Could this be a problem?

Actually, no, it shouldn't be. Checking the repertoire with Font Book I see that what I have on this machine, "Comic Sans MS", has a fairly complete set of glyphs including Cyrillic and a Euro, certainly everything that an ad might call upon.

Nevertheless, I took that out, letting the web page display default to whatever font it wants (it appears to be Times Roman), and now I just have to wait a while and see if the Last Resort message pops up again.

Later: It hasn't.

Getting useful info from QFontDatabase

I'll get back to the Last(goddam)Resort font issue shortly. But in learning about it I've had to look a little closer at what Qt knows about fonts, and as a result I've got a function that might be useful to others.

The QFontDatabase object contains whatever Qt knows about fonts that are "available in the underlying window system". However it makes this information available in a rather awkward way. You can call its families() method to get a list of all font names:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QFont, QFontDatabase
the_app = QApplication([])
f_db = QFontDatabase()
for family in f_db.families():
    print(family)

My macbook is rather over-endowed with fonts, and this prints a list of 238 names.

You can also ask for the recommended or default fixed and "general" fonts,

qf_fixed = f_db.systemFont(QFontDatabase.FixedFont)
print( 'Fixed font:',qf_fixed.family() )
qf_general = f_db.systemFont(QFontDatabase.GeneralFont)
print( 'General font:',qf_general.family() )

Which on my macbook prints

Fixed font: Monaco
General font: .Lucida Grande UI

Here's a surprise: what the heck is that leading dot doing in the name of the general font? There is definitely no font named ".Lucida Grande" in the list of families, nor displayed by the Fontbook app, nor (using find /System ".Lucida*') in the System folder. However, the font db will create a QFont for it:

>>> lgf = f_db.font('.Lucida Grande UI','',12)
>>> lgf.family()
'.Lucida Grande UI'
>>> lgf.pointSize()
13

So it's real in some sense, and your code can use it. Moving on...

You can ask the font db things about any single family name, for example isFixedPitch(family). But font family names aren't consistent from system to system. And often what you want is a font that meets certain criteria, such as: it's "Times" by some name, or it has a bold or italic variant, or it scales to 36pt. So I put together this little function that will return a (possibly empty) list of fonts that meet given criteria:

def font_filter( name = None, style = None, fixed = False, sizes = [], language = None ) :
    db = QFontDatabase()
    selection = db.families()
    if name :
        selection = [family for family in selection if name in family ]
    if style :
        selection = [family for family in selection if style in db.styles(family) ]
    if fixed :
        selection = [family for family in selection if db.isFixedPitch(family) ]
    if language :
        selection = [family for family in selection if language in db.writingSystems(family) ]
    if sizes :
        size_set = set(sizes)
        selection = [family for family in selection
                     if size_set.issubset( set( db.smoothSizes(family, '' ) ) ) ]
    return selection

For example,

>>> print( font_filter( name = 'Helvetica' ) )
['Helvetica', 'Helvetica CY', 'Helvetica Neue']
>>> print( font_filter( fixed = True ) )
['Anonymous Pro', 'Courier', 'Courier New', 'Inconsolata', 'PCMyungjo', 'PT Mono']
>>> print( font_filter(fixed = True, style='Bold', sizes=[12, 18]) )
['Courier', 'Courier New', 'PT Mono']
>>> font_filter(fixed=True, language=QFontDatabase.Hebrew)
['Courier New']

Next time: back to the issue of the Last(goddam)Resort font problem.

Last (goddam) Resort Font

I wrote CoBro back in 2012 with the original intent of "releasing" it for public use (i.e. making it known on the the comics subreddit) but because of QWebKit instabilities as well as problems packaging it with either PyInstaller or cx_freeze, I put that off until I could re-do it for Qt5 and Python 3.

Now I've done that it still is plagued by QWebKit issues (and still breaks cx_freeze, and PyInstaller still doesn't support Python 3). Among the instabilities is a very intermittent tendency to throw a blizzard of messages like this: 2014-03-14 15:06:59.270 Python[6570:d07] Critical failure: the LastResort font is unavailable.

This is a Python log message, not a Qt one that I could maybe stifle with the technique mentioned in the prior post. And "Critical" at that! So what's going on?

Turning to The Google I find that a lot of people encountered this message with a variety of different apps, but mostly back in 2010 or 2011 when Snow Leopard was new. But some information emerged.

What is the Last Resort font? The official explanation from Unicode.org is that it is a font to use when the system needs to print a Unicode glyph and all available fonts lack values for that code position.

if the font cannot represent any particular Unicode character, the appropriate "missing" glyph from the Last Resort font is used instead. This provides users with the ability to tell what sort of character it is, and gives them a clue as to what type of font they would need to display the characters correctly.

This explains why the message comes only rarely but in a flurry: it happens when a comic web page has a glyph that isn't in any available font. That doesn't happen all the time, but if it does, the site is likely to have a string of such glyphs; hence the multiple messages. But why can't the WebKit browser find the font? With a little poking around on my system (and absolutely no help from the Finder, whose search function insists there is nothing to match that name) I did find that /System/Library/Fonts/LastResort.ttf exists and had 644 permissions. So...

Finally I found this FontGeek post about Safari having the same problem. They claim that because of "sandboxing" the browser can't access the folder where LastResort resides. I'm dubious about the explanation; and the fix they offer isn't directly applicable to the Qt WebKit code as far as I can tell. But it's the best explanation I've seen.

What to do? One possibility: get a copy of LastResort.ttf and include it in the app. I've already got code in PPQT V1 to load a font bundled with the app and add it to the QFontDatabase. I need to think about this, also I need to find a web comic that will trigger the issue reliably.

Trapping Qt log messages

Using PyQt5, you can install your own handler for Qt's log messages and do with them as you wish, for example diverting them to a Python log file. There are (as usual) some surprises, but by and large, it works.

The context is CoBro, my little web-comic browser. It uses QWebKit to display single HTML pages. After displaying one particular comic (Two Guys and Guy) the webkit code likes to emit a couple of log messages like "error: Internal problem, this method must only be called once." Annoying since there is nothing you can do about it, and it doesn't seem to cause any harm. (BTW this is a known problem, see Qt Bug #30298.)

So I add the following code to Cobro, after creating the App and the main window and everything and we are just about ready to show the main window and enter the event loop:

    from PyQt5.QtCore import qInstallMessageHandler, QMessageLogContext
    from PyQt5.Qt import QtMsgType

    def myQtMsgHandler( msg_type, msg_log_context, msg_string ) :
        print('file:', msg_log_context.file)
        print('function:', msg_log_context.function)
        print('line:', msg_log_context.line)
        print('  txt:', msg_string)

    qInstallMessageHandler(myQtMsgHandler)

Now what comes out on stderr is this:

file: access/qnetworkreplyhttpimpl.cpp
function: void QNetworkReplyHttpImplPrivate::error(QNetworkReplyImpl::NetworkError, const QString &)
line: 1929
  txt: QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.

Ta-daaaa! We have intercepted a Qt log message and analyzed it to show where in the Qt code it originated. One surprise is that the "function" member of the QMessageLogContext object is not a simple function name, but the full C++ signature. Another surprise is that, when I stop on this code in a debugger and look at the msg_log_context item, its members are not strings but "sip reference" objects. Nevertheless by the magic of PyQt5 they print as strings.

Well, printing this bumf isn't a lot of use. What would be more useful, is to divert it into the Python log stream, like this:

    def myQtMsgHandler( msg_type, msg_log_context, msg_string ) :
        # Convert Qt msg type to logging level
        log_level = [logging.DEBUG,
                     logging.WARN,
                     logging.ERROR,
                     logging.FATAL] [ int(msg_type) ]
        logging.log(logging.DEBUG,
                    'Qt context file is '+msg_log_context.file
                    )
        logging.log(logging.DEBUG,
                    'Qt context line and function: {0} {1}'.format(
                        msg_log_context.line, msg_log_context.function)
                    )
        logging.log(log_level, 'Qt message: '+msg_string)

In other words, log the gritty details of the QMessageLogContext at the DEBUG level, but log its actual text at its own self-assigned severity, as translated into Python logging's values. The above code works and now I can redirect QWebKit's annoying messages into CoBro's log file.

Making a bad QTextCursor and a promising find

Working today on the unit testing and code finalization of pagedata, the module that keeps track of where the scanned OCR pages of a book each start. This module acts as the data model for several clients: every time the user moves the edit cursor, a bunch of different widgets will ask pagedata "which page is the cursor on now?" The imageview module asks it for the filename of the .png file to display, the scan# and folio# widgets under the edit window ask it for those items, and of course the Pages panel calls on it for the rows of data it displays as a table.

I thought I'd finalized the code but of course as soon as I started adding test calls into it from its unit-test module I found not only bugs but also things I hadn't thought of. It's schizophrenic, flipping back and forth between coding and testing. "What will it do if I throw it this?" the tester asks, and the coder is thinking "Oh shit, why didn't I plan for that?"

Anyway, one of the pieces of crap the test-monkey in me flang at the pagedata module opened a wonderful new prospect in Qt error control! It went down like this.

Important method in the PageData object is read_pages, which processes the page-data lines from the .meta file. When a book is saved, all the metadata goes in the bookname.meta file, including everything we know about page boundary locations. So at load time, read_pages gets called to rip through these saved lines and rebuild the page table as it was when the book was saved.

There are six items in each line, the first being the character offset to the start of the page. That gets turned into a QTextCursor so that Qt will maintain the position as it changes under user editing actions. The code is simple:

try:
    (P, fn, pfrs, rule, fmt, nbr) = line.split(' ')
    tc = QTextCursor(self.document)
    tc.setPosition(int(P))

and so forth. The test case had already flung a non-integer position P, and the failure of int(P) was caught by the try/except fine. So the next nastiness was a bad position value, first 1000000, much larger than the document, next -1. But neither of these tripped an exception! All that happened was that a message appeared on stderr, "QTextCursor::setPosition: Position '100000' out of range" and the QTextCursor was unchanged.

This opened two new questions: (1), how the heck can read_pages detect that it got a bad position?, and (B), how can we avoid having that message, about an error we've anticipated and dealt with, cluttering up stderr and getting the user all upset?

It was too late in the day to investigate (1), but (B) is a problem I've been plagued by for a long time. Qt is just full of unhelpful debugging errors. My other app, a simple web browser based on QWebKit, likes to throw out stuff like this:

QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.

And I've been wondering in a dazed sort of way if there mightn't be some way to stifle those. But PPQT2 is using proper Python logging. Every module creates its own logger and writes diagnostics, warnings and errors with the logging API. And I definitely plan not to let those log messages dribble out on stderr; they will eventually go into a file.

So it suddenly occurred to me, is there maybe some way to divert the Qt log messages into the Python logging system? I wasn't aware of anything but I started browsing in the index of the Qt Assistant and turned up this: qInstallMessageHandler. This appears to offer a way to accept and process all of Qt's messages.

I'm quite excited about this. If it is accessible under PyQt5, I see a clear path to capturing all Qt messages and converting them into Python log entries. That will let me stifle the WebKit chatter from my comics browser and also this text cursor message in PPQT. Tomorrow I spend the day at the museum but thursday afternoon I get to dig into this!

Model-View design and user expectations of performance

PPQT presents

• A table of the words in the document with their properties such as uppercase, numeric, misspelled.
• A table of the characters in the document with their counts and Unicode categories
• A table of the book pages, derived from the original PGDP page-separator lines

Each of these tables is derived from a "census" in which every line, word-token, and character in the document is counted. In v.1 this census is done the first time a book is opened, and any time after when the user needs to "refresh" the display of word or character counts. It's very time-consuming, 5 to 20 seconds for a large book. Getting the time down for v.2 would be a good thing. So would avoiding a big delay during first opening of a new book.

In v.1 the census is done in one rather massive block of code that fetches each line from the QTextDocument in turn as a QString and parses each, counting the characters and using a massive regex to pick out wordlike tokens. This process is complicated by:

• The need to handle the PG codes for non-Latin-1 chars such as [oe].
• The need to recognize HTML-like productions: some like <i> and <sc> are common from the start, and later in the book-production process there might be thousands of HTML codes; we count them for characters but not for "words".
• But also the need to spot the lang=code property embedded in HTML codes, to signal use of an alternate spelling dictionary.

For v.2 I want to break up the management of all these metadata along MVC lines, with a "data" module and a "view" module for each type, so worddata.py manages the list of words while wordview.py contains the code to present that data using a QTableView and assorted buttons. Similarly for chardata/charview and pagedata/pageview. But will this complicate the census process? Will it slow it down?

Complicate it? Not exactly; more like "distribute" it. I will move each type of census to its data model: worddata will take a word census, chardata a char census, pagedata a page census. So a full census could potentially entail three passes over the document.

However, when this separation is done, it becomes clear that the only census that really needs to be done the first time a book is opened, is the page census. That's because the module that displays the matching page scan image as the user moves through the text, needs to know the position of each page's start. In other words, pagedata is the data model for both the page table and the image-display panel. Images need to be displayed immediately, so the page data needs to be censused the first time a book is opened.

The word and char censii, however, can wait. The char data is the model only for the Char panel. If that panel is showing an empty table, the user knows to click its "Refresh" button to make a char census happen, so the table updates.

The word data is the model for the Word panel, and again, if the user opens a new book and goes to the Word panel, and sees an empty table, it's a no-brainer to click Refresh and update the table. In either case, the user knows they've asked for something, and should be content to wait while the progress bar turns and the census finishes.

The word data is also the model, however, for the display of misspelled words with a red underline, and the display of "scannos", highlighted document words that appear in a file of likely OCR errors. These features of the editor are turned on with a menu choice (? or perhaps a check box in v.2? TBS). If either highlighter is set ON when a new book is opened, the highlights won't happen because the word data isn't known until a census is taken.

Easy solution: we know when we are opening a new book (we don't see a matching metadata file from a prior save), and in that case we force OFF the spellcheck and scanno highlight choices. Then if/when the user clicks spelling or scanno highlights ON, we can run a census at that time. Again the potentially slow process is initiated by an explicit user action.

What about (perceived) performance? It should be snappier. If you Refresh the Chars panel it will rip through the document counting characters, but not spend time on the big word-token regex html-skipping process. Refresh the Words panel and its census will at least not be slowed by counting characters.

Great, but I already started coding worddata on the assumption it would base both chars and words. Now I have to split it up.

Converting PPQT: which RE lib?

I'm working on a lengthy project to make version 2 of PPQT, a large Python/Qt app. I'm documenting some of the things I learn in occasional blog posts.

PPQT 1 makes frequent use of regular expressions, mostly using Qt's QRegExp class. That has to change for two reasons. One is that QRegExp falls quite a bit short of PCRE compatibility. Qt5 includes a new class, QRegularExpression, which does claim PCRE compatibility as well as performance, so at least I want to convert the old ones to the longer-named type.

However, one big difference from PyQt4 to 5 is the "new API" that abolishes use of QString. In PyQt4 many class methods take, or return, QStrings, and PPQT uses lots of QString objects. QStrings and QRegExps work well together; QRegExp.indexIn() takes a QString, and QString.find() takes a QRegExp.

In PyQt5, all classes that (in the C++ documentation) take or return a QString, now take or return a simple Python string value, with PyQt5 doing automatic conversion. There is no "QString" class in PyQt5—at all! That means there is no way to call QString.find(), and if you call QRegExp.indexIn(string), there will be a hidden conversion from Python to QString. Which means—why use Qt regexes at all? Since all program-accessible strings are Python strings, why not use Python's own regular expression support?

Standard Python support is the "re" lib. It also is not PCRE compatible (although closer than QRegExp) and not known for speed. But there is another: the "regex" module, which intends to become the Python standard but now is an optional install. It is PCRE-compatible, with the Unicode property searches and Unicode case-folding that are lacking in QRegExp and in the re module. It actually adds more functionality, including "fuzzy" matches that could be very useful to me in PPQT. The class and method names are the same as the standard re module.

Code Changes

One design difference between Python's re/regex and QRegularExpression on one side, and the QRegExp that PPQT 1 uses so many of on the other, will cause some code changes.

An instance of QRegExp is not reentrant: when it is used, it stores information about the match position and capture groups in the regex object. Such an object shouldn't be a global or class variable shared between instances of a class, because activity in one using method could overwrite a match found from another. But based on its design, PPQT 1 had frequent uses like this:

    if 0 <= re_object.indexIn(string):
        cap1 = re_object.cap(1)

Both re/regex and QRegularExpression take a different approach: the regex object knows about the search pattern but is otherwise immutable. When you perform a search with it it, it returns a match object that encodes the positions and lengths of the matched and captured substrings. The regex object can be a global; every using method gets its private match object to work with. However, code like that above has to be rewritten (using Python re/regex) as:

    match = re_object.match(string)
    if match : # i.e. result was not None
        cap1 = match.cap(1)

Python re/relib match returns None on failure, or a match object. None evaluates as False, so "if match" is equivalent to "if a match was found." The returned value of a QRegularExpression object is always a (take a deep breath) QRegularExpressionMatch object, so the equivalent would be:

    match = re_object.match(string)
    if match.isValid() : # match succeeded
        cap1 = match.captured(1)

Not only is this many more keystrokes to write, it entails two pointless auto-conversions between Python and Qt string types: from Python to Qstring in the match() call, and from QString to Python in the captured(1). All told, the Python relib seems a better choice and I plan to use it exclusively in PPQT 2.