Current Line Revisited

A few days ago I described my progress on editview, but just today I stumbled on a big improvement. Here's how it looks now.

If you click through you find that's a quite large image. The reason is, it's from a retina macbook so what looks like quite a modest window, when captured, comes out 1500px wide. Here are the improvements from the prior version.

• The current-line highlight now extends the full width of the window. Before it was only as long as the text on that line.
• Scanno highlighting (the lilac highlights) is implemented. You can load a file of common OCR errors and they are marked wherever they appear.
• Spellcheck highlighting (wiggly magenta underlines) is implemented, including alternate dictionaries. Note the line with <span lang='fr_FR'>; those words get checked against the french dictionary instead of the default one.

Pretty much all that remains is to finish an automated unit test of these features. I have one simple unit test driver now that uses QTest to automate a number of keystrokes, but I need to also automate exercising a pop-up context menu. That'll be an adventure I'm sure.

In the previous post I kvetched about how, although a QTextBlock has a format (QTextBlockFormat), you could only interrogate it, and modifying it didn't change the format. As a result, what I expected would be a simple way to set a current-line highlight, by setting the background brush of the current text block, didn't work.

Then today, browsing around the QTextCursor documentation, what should my eye fall upon but a setBlockFormat method! You can ask a QTextBlock for its format, but in order to set it, you have to aim a QTextCursor at that block, and then tell the cursor to set the block's format.

Bizarre.

Well, at any rate, not how I'd have designed it. But I didn't, so...

So I realized that my previous method of highlighting the current line using the extraSelections mechanism was over-complicated. I changed the logic to set a background brush on the current block. The cursor-moved logic now reads as follows:

Note: The following is still not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

    def _cursor_moved(self):
        tc = QTextCursor(self.Editor.textCursor())
        self.ColNumber.setText(str(tc.positionInBlock()))
        tb = tc.block()
        if tb == self.last_text_block:
            return # still on same line, nothing more to do
        # Fill in line-number widget, line #s are origin-1
        self.LineNumber.setText(str(tb.blockNumber()+1))
        # Fill in the image name and folio widgets
        pn = self.page_model.page_index(tc.position())
        if pn is not None : # the page model has info on this position
            self.ImageFilename.setText(self.page_model.filename(pn))
            self.Folio.setText(self.page_model.folio_string(pn))
        else: # no image data, or cursor is above page 1
            self.ImageFilename.setText('')
            self.Folio.setText('')
        # clear any highlight on the previous current line
        self.last_cursor.setBlockFormat(self.normal_line_fmt)
        # remember this new current line
        self.last_cursor = tc
        self.last_text_block = tb
        # and set its highlight
        tc.setBlockFormat(self.current_line_fmt)

Case of the Shredded Data

A persistent source of PyQt bugs is the problem that as soon as a variable "goes out of scope"—that is, cannot be referenced from any statement—it gets garbage-collected and either re-used or made unreachable by the virtual memory hardware. Newbies to PyQt get bitten by this early, often, and hard. It usually shows up as a segmentation fault that takes down the Python interpreter and your app. And there's no obvious bread-crumb path back to the problem.

The usual problem is that you build one object A based on another object B. Then you pass object A around and try to use it in another part of the program. Meanwhile, object B, which was just input material to making A, has gone out of scope and been shredded. Then any use of A references non-existent memory and segfaults, or produces weird results because it is accessing memory that doesn't contain what it should.

Case in point: QTextStream. This is a useful class, very handy for reading and writing all kinds of files. You could use the Python file class instead, but you need to standardize on one paradigm or the other, Python's files or Qt's QTextStreams. And I've gone with the latter, but they have this little problem of segfaulting if you are not careful with them.

A QTextStream is built upon some source of data, either a QFile or an in-memory QByteArray. The class constructor takes that source object as its only argument, as in this perfectly innocuous function:

def get_a_stream(path_string):
    '''Return a QTextStream based on a path, or None if invalid path'''
    if not QFile.exists(path_string):
        return None
    a_file = QFile(path_string)
    if not a_file.open(QIODevice.ReadOnly):
        return None
    return QTextStream(a_file)

Lovely, what? Simple, clear—and wrong. Because a_file goes out of scope as soon as the function returns. Many statements away in another part of the program, the next use of the returned stream crashes the program. This is (in my oh-so-humble opinion) a stupid design error in Qt (it affects C++ users too) but fortunately it is easy to work around. You just use the following instead of QTextStream:

class FileBasedTextStream(QTextStream):
    def __init__(self, qfile):
        super().__init__(qfile)
        self.save_the_goddam_file_from_garbage_collection = qfile

That's it! An object of this class FileBasedTextStream can be used anywhere a QTextStream would be used, but it does not require you to find some way to save the QFile from the garbage collector. The single reference to the QFile keeps it alive until the stream object itself is freed.

I solved this same issue earlier, for memory-based text streams. These are very handy for nonce files, and my unit test code builds lots of them.

class MemoryStream(QTextStream):
    def __init__(self):
        # Create a byte array that stays in scope as long as we do
        self.buffer = QByteArray()
        # Initialize the "real" QTextStream with a ByteArray buffer.
        super().__init__(self.buffer)
        # The default codec is codecForLocale, which might vary with
        # the platform, so set a codec here for consistency. UTF-16
        # should entail minimal or no conversion on input or output.
        self.setCodec( QTextCodec.codecForName('UTF-16') )
    def rewind(self):
        self.seek(0)
    def writeLine(self, str):
        self << str
        self << '\n'

It's just a QTextStream based on an in-memory buffer, but the buffer can't go out of scope as long as the object exists. It adds a couple of minor features that QTextStream lacks.

Why Can't Huns Spell?

Lordy but I hate the kind of work described below. It's really stressful. (I know, kvetch, kvetch, kvetch.)

I'm to the point where I want to test the ability to mark words that fail spellcheck. To do that, I need the ability to check spelling, duh!

PPQT version 1 went through stages of spell-checking, each representing many hours of effort. First, trying to send words over a pipe to Aspell running as a subprocess. Then I wrote my own all-Python spell-checker to use the Myspell/OpenOffice dictionary format. That was a useful learning exercise. I learned:

• All about the format and content of the .dic/.aff dictionary files.
• That German is a damned hard language to spell-check.
• That there are a lot of subtleties to the spell-check algorithms.

In the end German defeated my code. I just couldn't get it to handle multiple affixes properly. In the nick of time I found this Python wrapper for Hunspell. Like a lot of FOSS, it was created by someone who needed it a few years ago, and that person has apparently moved on and left it dangling unmaintained. But it can be made to work, with effort—for Python 2.x. And it was suuu-weet once I got it going, blazing fast and reliable. I made it work for Mac OS and for Linux, but blew many hours failing to make it work for my Windows distribution. Eventually I went on ELance and paid a dude $150 to make the Hunspell wrapper work on Windows. Money well spent.

But PPQT2 is built on Python 3.3 (well, probably 3.6 by the time it's done) and the Hunspell wrapper doesn't work for that. However, another user posted a diff file that, he claimed, made it work with Python 3. So I spent some hours today getting it compiled and installed.

It should be a one-liner, python setup.py install, but of course that don't work because there are things in the setup.py script that assume Linux, and a prior release of Hunspell. So you tweak that a while. Reviewing my notes from last fall, you run python setup.py build and it fails, then you manually run a compile command that works to actually create the module, then python setup.py install will install it. The compile command that worked for 2.7 (contributed to that wiki by another user, bless her heart) was:

gcc -fno-strict-aliasing -fno-common -g -fwrapv\
 -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64\
 -pipe -D_LINUX -I/usr/include/hunspell\
 -I/usr/include/python2.7 -lpython -lhunspell-1.2\
 -shared hunspell.c -o build/hunspell.so

But that doesn't work after the diff was applied for Python 3. It coughed up an unresolved symbol _PyModule_Create2 for no apparent reason. So, what's a search engine for if not to find obscure error messages? And Da Google turned up many people with this problem dating back to 2010. A stackoverflow response, although not directly responsive, pointed to lack of inclusion of the python3.3 library, and that was it. Here's the command that actually compiles and links hunspell for Python 3:

P=/Library/Frameworks/Python.framework/Versions/Current
gcc -fno-strict-aliasing -fno-common -g -fwrapv\
 -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64\
 -pipe -D_LINUX -I/usr/local/include/hunspell\
 -I$P/include/python3.3m\
 -L$P/lib -lpython3.3 -lhunspell-1.3 -shared\
 hunspell.c -o build/hunspell.so

So I now have a working Hunspell that I can start playing with, on Mac OS at least. Such a relief! OK, Ghu alone knows what it will take to make it work on other platforms, but that's months away. For now, my Huns can spell.

Which Line Is It, Anyway?

The editview module is getting pretty complete. The only missing function is the dreaded syntax-highlighter to highlight scannos or spelling errors. Here's what it looks like now.

Today I added the code to highlight the current line. That's why one line has a sort of pale-lemon background. In V1, there was no current line highlight, and it was quite easy to lose sight of the cursor, and have to rattle the arrow keys to find it. (The string shown in dark gray is selected text and is actually bright yellow; the Grab utility did something to the colors.)

Qt's method of doing this was surprising to me.

In a QPlainTextEdit, there is a 1:1 correspondence between text blocks and logical lines. Each line of text is in one QTextBlock. Now, QTextBlock has a property blockFormat which is a QTextBlockFormat, which is itself a QTextCharFormat derivative, i.e. it can be used to set the font, color, background brush and so on. So when I started looking at how to make the current line a different color, I saw this and supposed it would be a matter of, each time the cursor moved:

• Get the text block containing the cursor, a single method call,
• Clear the background brush of the previous line's text block,
• Set the current text block's blockFormat to a different background brush

But in fact QTextBlock lacks anything like a setBlockFormat, so the property is read-only. And setting the background property of the returned QTextBlockFormat object was accepted but had no visible effect.

Sigh, back to the googles to find a number of places in the Qt docs, stackoverflow and the like, where the question is raised and answered.

QPlainTextEdit supports a property extraSelections, which is a list of QTextEdit::ExtraSelection objects. This is the first and I think only time I've seen a class documented as child of another class. And it's a weird little class; it has no methods (not even a constructor), just two properties, cursor and format. So it's basically the C++ version of a python tuple.

What you do is, you get a QTextCursor to select the entire line, and you build an ExtraSelection object with that cursor and the QTextCharFormat you want to use, and assign that to the edit object's list of extra selections. This is a lot of mechanism to just highlight one line. Apparently the intent is to support an IDE that, for example, wants to put a different color on each line set as a breakpoint, or such.

Note: The following is not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

Anyway for the curious, this is the code that executes every bloody time the cursor moves:

    def _cursor_moved(self):
        tc = QTextCursor(self.Editor.textCursor())
        self.ColNumber.setText(str(tc.positionInBlock()))
        tb = tc.block()
        ln = tb.blockNumber()+1 # block #s are origin-0, line #s origin-1
        if ln != self.last_line_number:
            self.last_line_number = ln
            self.LineNumber.setText(str(ln))
            tc.movePosition(QTextCursor.EndOfBlock)
            tc.movePosition(QTextCursor.StartOfBlock,QTextCursor.KeepAnchor)
            self.current_line_thing.cursor = tc
            self.Editor.setExtraSelections([self.current_line_thing])
            pn = self.page_model.page_index(tc.position())
            if pn is not None : # the page model has info on this position
                self.ImageFilename.setText(self.page_model.filename(pn))
                self.Folio.setText(self.page_model.folio_string(pn))
            else: # no image data, or positioned above page 1
                self.ImageFilename.setText('')
                self.Folio.setText('')

In sequence this does as follows:

• Get a copy of the current edit cursor. A copy because we may mess with it later.
• Set the column number in the column number widget.
• Get the QTextBlock containing the cursor's position property (note 1 below).
• Get the line number it represents.
• If this block is a change from before (note 2):
• Set the line number in the line number widget.
• Make the cursor selection be the entire line ("click" at the end, "drag" to the front)
• Set that cursor in a single ExtraSelection object we keep handy.
• Assign that object as a list of one item to the editor's extra selections.
• Get the filename of the current image file, if any; and if there is one, display it and the logical folio for that page in the image and folio widgets.

Note 1: If there's no selection, a text cursor's position is just where the cursor is. But if the user has made a selection, the position property might be at either end of it. Drag from up-left toward down-right and the position is the end of the selection. Drag the other way, it's at the start. Drag a multi-line selection that starts and ends in mid-line. One of the lines will have the faint current-line highlight: the top line if you dragged up, the bottom line if you dragged down. I don't think anyone will notice, or care if they do. I could add code to set the current line on min(tc.position(),tc.anchor())—but I won't.

Note 2: Initially, there was no "if ln != self.last_line_number" test; everything was done every time the cursor moved. And actually performance was fine. But I just could not stand the idea of all that redundant fussing about happening when it didn't have to.

Further on the Mac Option Key

The Qt Forum post I made about the Option-key problem, after 22 hours, has been viewed 32 times but drawn no responses. I also posted a respectful query on the pyqt list this morning (after obsessing about the issue some of the night).

I also spent a couple more hours delving deeply into the QCoreApplication, QGuiApplication, and QApplication docs, hoping to find some kind of magic switch to change the behavior of the key interface. I speculate that Qt5 has better Cocoa integration and as a result is getting the logical key from a higher-level interface than before.

Supposing it can't be fixed or circumvented, what I will have to do is: In constants.py where the key values and key sets are determined, check the platform and use Qt.MetaModifier instead of Qt.AltModifier when defining keys for Mac. This substitutes the actual Control shift for the Option shift.

That would be the only module with a platform dependency. Others just use the names of keys and key-sets defined in constants.py. For the user, I will have to have separate documentation about bookmarks, for Mac and non-Mac. For non-Mac, it'll remain "Press control and alt with a number 1-9 to set that bookmark." For mac it will be "Press the Control key and the Command key together with a number 1-9..." And the beautiful consistency ("where you see 'alt' think 'option'" at the front and never mention it again) is gone.

Another issue is the use of ctl-alt-M and ctl-alt-P in the Notes panel, to insert the current line or image number. Possibly I can just change the key definition in constants to whatever the mac keyboard generates for option-M and option-U (pi and mu, it seems). Or keep the directions consistent, and completely wipe out any use of Option-keys in Mac.

---

Also today I tested and committed the zoom keys, which work a treat. The unit test module buzzes up 10 points and down 15, looks great.

A Bump in the Road

Today I thought I'd add in the special keystrokes to the editview. There are three groups of them: a set that interact with the Find dialog (^f, ^g, etc), and these I'm deferring until I actually work on the Find panel; a bookmark set, (ctl-1 to 9 to jump to a bookmark, ctl-alt-1 to 9 to set one); and ctl-plus/minus to zoom. All of these were implemented and working in version 1, using the keyPressEvent() method to trap the keys.

So I messed around and tidied up the constants that define the various groups of keys as sets, so the keyPressEvent can very quickly determine if a key is one it handles, or not: if the_key in zoom_set and so on.

With the brush cleared, I copied over the keyPressEvent code from V1 and recoded it (smarter and tighter) for V2 and ran a test, and oops something is not working.

Specifically, it is no longer possible to set bookmark 2 by pressing ctl-alt-2. On a mac, that's command-option-2, which Qt delivers as the Qt.ALT_MODIFIER plus Qt.CTL_MODIFIER and the key of Qt.KEY_2.

Or rather, it used to do that. I fired up PPQT version 1 just to make sure. Yup, could set a bookmark using cmd-opt-2. But not in the new version. Put in debug printout. The key event delivered the same modifier values, ctl+alt, but the key value was... 0x2122, the ™ key? And cmd-alt-3 gave me Qt.KEY_STERLING, 0xA3. And cmd-alt-1 is a dead key.

Pondering ensued. OK, these are the glyphs that you see, if you open the Mac Keyboard viewer widget and depress the Option key. So under Qt5, the keyboard event processor is delivering the OS's logical key, but under Qt4 in the same machine at the same time it delivers the physical key.

Oh dear.

I spent several hours searching stackoverflow and the qt-project forums and bug database but nothing seemed relevant. I posted a query in the Qt forum. But I have little hope. It looks very much as if I'll have to change they key choices for bookmarks, and make them platform-dependent. In Windows and Linux they can continue to be ctl[-alt]-1 to 9, but in Mac OS this will change. The only reliable special key modifiers are control (Command) and meta (the Control key!).

In V1 it was great that I could document just once at the top of the docs, that in Mac, "ctl means cmd" and "alt means option". And that was consistent throughout. Now it won't be because the Option key is effectively dead for my purposes. I'll have to tell the mac user, "when I say control I mean command, but when I say alt, I mean control." Won't that be nice? Plus, I'll have to have code that looks at the platform and redefines the key sets for Mac at startup. Very disappointing.

Funny Little Thing (Solved)

Next day second thoughts: While as noted, QTextDocument inherits from QObject and as such has no font property, it does for no obvious reason have a defaultFont property which it imposes on any Q[Plain]TextEdit to which it is connected. This completely violates the Model/View scheme, the Model imposing a presentation feature on the View, but there it is. Furthermore, it can't be overridden! As I note below, when I interrogate the edit widget for its font().family and font().pointSize, it happily reports the values I'd set. But what it displays is its document's default font, if the document is connected second. So fie on ye, Qt designer.

---

OK, here's an oddity in Qt. I'm working on the editview, a panel mainly containing a QPlainTextEdit that is pretty much the heart of PPQT. But the editor is the "view" and its "model" or repository of data is a QTextDocument. Most of the GUI initialization is handled by the code generated from the Designer. But the __init__ for the editview has to do two key things. Well, there will be lots more, hooking up signals, setting up the syntax highlighter, blah blah, but in its rudimentary state, two things:

• Set its fonts
• Set its document to the edit model

Item two is pretty simple. The parent Book is the repository of all knowledge, so it just does self.Editor.setDocument(self.my_book.get_edit_model()).

Setting the fonts is slightly more work. This is looking ahead to when the user will be able to tell the main window, I want to choose a different UI font, the default font used in most labels and buttons, or I want to choose a different edit font, the had-better-be-monospaced font used only in the editview(s) and one or two other places like the Find text string.

So there can be multiple Books open each with its editview, and up in the main window the user says, "Let's use Courier!" Main window will emit a signal, and any widget that cares better catch it and change fonts.

Also, at open time when setting up a new editview, we want to restore the font size the user had last time the book was open. In version 1, with only one book, that was a global setting, but now there can be multiple documents with potentially each editview zoomed to a different font size! Just one of the many, many features affected by multiple documents.

Anyway, while initializing, the editview needs to ask its Book for the font size and also ask the global font module for the proper font, and set that family/size combo in the QPlainTextEdit. No biggie, a simple method that is called in init. or by the font-change signal:

    def set_fonts(self):
        general = fonts.get_general() # UI font at default size
        self.setFont(general) # set self, propogates to children
        mono = fonts.get_fixed(self.my_book.get_font_size())
        self.Editor.setFont(mono) # the editor is monospaced

This story is going somewhere, really. OK, so the editview initialized like this:

        self.set_fonts()
        self.Editor.setDocument(self.my_book.get_edit_model())

And it didn't work. When it displayed, all the labels would be in the system default font (Lucida Grande 13pt) but so would the edit widget be. I put in a debug print to query the edit widget's font and display it. It happily reported, "I'm using Liberation Mono 16 just like you said, boss", but when I typed in the edit window, it came out in Lucida. I set the parent widget to the mono font, and all the labels displayed in mono but the damn edit text was still Lucida!

After wasting a couple of hours on this, it occurred to me the QTextDocument might be the villain. I reversed the init to

        self.Editor.setDocument(self.document)
        self.set_fonts()

And it worked, the edit text is now in the chosen mono font. WTF? Reviewing the doc for QTextDocument, it inherits from QObject not QWidget, so it doesn't even have a font property. Yet somehow, the call to setDocument() undid the work of a preceding call to setFont(). Maybe changing the document is so basic to the edit widget that it treats it as a reset. Whatever. In my application, I only set the document once. As long as it happens first, all is well.