Why Can't Huns Spell?

Lordy but I hate the kind of work described below. It's really stressful. (I know, kvetch, kvetch, kvetch.)

I'm to the point where I want to test the ability to mark words that fail spellcheck. To do that, I need the ability to check spelling, duh!

PPQT version 1 went through stages of spell-checking, each representing many hours of effort. First, trying to send words over a pipe to Aspell running as a subprocess. Then I wrote my own all-Python spell-checker to use the Myspell/OpenOffice dictionary format. That was a useful learning exercise. I learned:

• All about the format and content of the .dic/.aff dictionary files.
• That German is a damned hard language to spell-check.
• That there are a lot of subtleties to the spell-check algorithms.

In the end German defeated my code. I just couldn't get it to handle multiple affixes properly. In the nick of time I found this Python wrapper for Hunspell. Like a lot of FOSS, it was created by someone who needed it a few years ago, and that person has apparently moved on and left it dangling unmaintained. But it can be made to work, with effort—for Python 2.x. And it was suuu-weet once I got it going, blazing fast and reliable. I made it work for Mac OS and for Linux, but blew many hours failing to make it work for my Windows distribution. Eventually I went on ELance and paid a dude $150 to make the Hunspell wrapper work on Windows. Money well spent.

But PPQT2 is built on Python 3.3 (well, probably 3.6 by the time it's done) and the Hunspell wrapper doesn't work for that. However, another user posted a diff file that, he claimed, made it work with Python 3. So I spent some hours today getting it compiled and installed.

It should be a one-liner, python setup.py install, but of course that don't work because there are things in the setup.py script that assume Linux, and a prior release of Hunspell. So you tweak that a while. Reviewing my notes from last fall, you run python setup.py build and it fails, then you manually run a compile command that works to actually create the module, then python setup.py install will install it. The compile command that worked for 2.7 (contributed to that wiki by another user, bless her heart) was:

gcc -fno-strict-aliasing -fno-common -g -fwrapv\
 -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64\
 -pipe -D_LINUX -I/usr/include/hunspell\
 -I/usr/include/python2.7 -lpython -lhunspell-1.2\
 -shared hunspell.c -o build/hunspell.so

But that doesn't work after the diff was applied for Python 3. It coughed up an unresolved symbol _PyModule_Create2 for no apparent reason. So, what's a search engine for if not to find obscure error messages? And Da Google turned up many people with this problem dating back to 2010. A stackoverflow response, although not directly responsive, pointed to lack of inclusion of the python3.3 library, and that was it. Here's the command that actually compiles and links hunspell for Python 3:

P=/Library/Frameworks/Python.framework/Versions/Current
gcc -fno-strict-aliasing -fno-common -g -fwrapv\
 -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64\
 -pipe -D_LINUX -I/usr/local/include/hunspell\
 -I$P/include/python3.3m\
 -L$P/lib -lpython3.3 -lhunspell-1.3 -shared\
 hunspell.c -o build/hunspell.so

So I now have a working Hunspell that I can start playing with, on Mac OS at least. Such a relief! OK, Ghu alone knows what it will take to make it work on other platforms, but that's months away. For now, my Huns can spell.

Which Line Is It, Anyway?

The editview module is getting pretty complete. The only missing function is the dreaded syntax-highlighter to highlight scannos or spelling errors. Here's what it looks like now.

Today I added the code to highlight the current line. That's why one line has a sort of pale-lemon background. In V1, there was no current line highlight, and it was quite easy to lose sight of the cursor, and have to rattle the arrow keys to find it. (The string shown in dark gray is selected text and is actually bright yellow; the Grab utility did something to the colors.)

Qt's method of doing this was surprising to me.

In a QPlainTextEdit, there is a 1:1 correspondence between text blocks and logical lines. Each line of text is in one QTextBlock. Now, QTextBlock has a property blockFormat which is a QTextBlockFormat, which is itself a QTextCharFormat derivative, i.e. it can be used to set the font, color, background brush and so on. So when I started looking at how to make the current line a different color, I saw this and supposed it would be a matter of, each time the cursor moved:

• Get the text block containing the cursor, a single method call,
• Clear the background brush of the previous line's text block,
• Set the current text block's blockFormat to a different background brush

But in fact QTextBlock lacks anything like a setBlockFormat, so the property is read-only. And setting the background property of the returned QTextBlockFormat object was accepted but had no visible effect.

Sigh, back to the googles to find a number of places in the Qt docs, stackoverflow and the like, where the question is raised and answered.

QPlainTextEdit supports a property extraSelections, which is a list of QTextEdit::ExtraSelection objects. This is the first and I think only time I've seen a class documented as child of another class. And it's a weird little class; it has no methods (not even a constructor), just two properties, cursor and format. So it's basically the C++ version of a python tuple.

What you do is, you get a QTextCursor to select the entire line, and you build an ExtraSelection object with that cursor and the QTextCharFormat you want to use, and assign that to the edit object's list of extra selections. This is a lot of mechanism to just highlight one line. Apparently the intent is to support an IDE that, for example, wants to put a different color on each line set as a breakpoint, or such.

Note: The following is not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

Anyway for the curious, this is the code that executes every bloody time the cursor moves:

    def _cursor_moved(self):
        tc = QTextCursor(self.Editor.textCursor())
        self.ColNumber.setText(str(tc.positionInBlock()))
        tb = tc.block()
        ln = tb.blockNumber()+1 # block #s are origin-0, line #s origin-1
        if ln != self.last_line_number:
            self.last_line_number = ln
            self.LineNumber.setText(str(ln))
            tc.movePosition(QTextCursor.EndOfBlock)
            tc.movePosition(QTextCursor.StartOfBlock,QTextCursor.KeepAnchor)
            self.current_line_thing.cursor = tc
            self.Editor.setExtraSelections([self.current_line_thing])
            pn = self.page_model.page_index(tc.position())
            if pn is not None : # the page model has info on this position
                self.ImageFilename.setText(self.page_model.filename(pn))
                self.Folio.setText(self.page_model.folio_string(pn))
            else: # no image data, or positioned above page 1
                self.ImageFilename.setText('')
                self.Folio.setText('')

In sequence this does as follows:

• Get a copy of the current edit cursor. A copy because we may mess with it later.
• Set the column number in the column number widget.
• Get the QTextBlock containing the cursor's position property (note 1 below).
• Get the line number it represents.
• If this block is a change from before (note 2):
• Set the line number in the line number widget.
• Make the cursor selection be the entire line ("click" at the end, "drag" to the front)
• Set that cursor in a single ExtraSelection object we keep handy.
• Assign that object as a list of one item to the editor's extra selections.
• Get the filename of the current image file, if any; and if there is one, display it and the logical folio for that page in the image and folio widgets.

Note 1: If there's no selection, a text cursor's position is just where the cursor is. But if the user has made a selection, the position property might be at either end of it. Drag from up-left toward down-right and the position is the end of the selection. Drag the other way, it's at the start. Drag a multi-line selection that starts and ends in mid-line. One of the lines will have the faint current-line highlight: the top line if you dragged up, the bottom line if you dragged down. I don't think anyone will notice, or care if they do. I could add code to set the current line on min(tc.position(),tc.anchor())—but I won't.

Note 2: Initially, there was no "if ln != self.last_line_number" test; everything was done every time the cursor moved. And actually performance was fine. But I just could not stand the idea of all that redundant fussing about happening when it didn't have to.

Further on the Mac Option Key

The Qt Forum post I made about the Option-key problem, after 22 hours, has been viewed 32 times but drawn no responses. I also posted a respectful query on the pyqt list this morning (after obsessing about the issue some of the night).

I also spent a couple more hours delving deeply into the QCoreApplication, QGuiApplication, and QApplication docs, hoping to find some kind of magic switch to change the behavior of the key interface. I speculate that Qt5 has better Cocoa integration and as a result is getting the logical key from a higher-level interface than before.

Supposing it can't be fixed or circumvented, what I will have to do is: In constants.py where the key values and key sets are determined, check the platform and use Qt.MetaModifier instead of Qt.AltModifier when defining keys for Mac. This substitutes the actual Control shift for the Option shift.

That would be the only module with a platform dependency. Others just use the names of keys and key-sets defined in constants.py. For the user, I will have to have separate documentation about bookmarks, for Mac and non-Mac. For non-Mac, it'll remain "Press control and alt with a number 1-9 to set that bookmark." For mac it will be "Press the Control key and the Command key together with a number 1-9..." And the beautiful consistency ("where you see 'alt' think 'option'" at the front and never mention it again) is gone.

Another issue is the use of ctl-alt-M and ctl-alt-P in the Notes panel, to insert the current line or image number. Possibly I can just change the key definition in constants to whatever the mac keyboard generates for option-M and option-U (pi and mu, it seems). Or keep the directions consistent, and completely wipe out any use of Option-keys in Mac.

---

Also today I tested and committed the zoom keys, which work a treat. The unit test module buzzes up 10 points and down 15, looks great.

A Bump in the Road

Today I thought I'd add in the special keystrokes to the editview. There are three groups of them: a set that interact with the Find dialog (^f, ^g, etc), and these I'm deferring until I actually work on the Find panel; a bookmark set, (ctl-1 to 9 to jump to a bookmark, ctl-alt-1 to 9 to set one); and ctl-plus/minus to zoom. All of these were implemented and working in version 1, using the keyPressEvent() method to trap the keys.

So I messed around and tidied up the constants that define the various groups of keys as sets, so the keyPressEvent can very quickly determine if a key is one it handles, or not: if the_key in zoom_set and so on.

With the brush cleared, I copied over the keyPressEvent code from V1 and recoded it (smarter and tighter) for V2 and ran a test, and oops something is not working.

Specifically, it is no longer possible to set bookmark 2 by pressing ctl-alt-2. On a mac, that's command-option-2, which Qt delivers as the Qt.ALT_MODIFIER plus Qt.CTL_MODIFIER and the key of Qt.KEY_2.

Or rather, it used to do that. I fired up PPQT version 1 just to make sure. Yup, could set a bookmark using cmd-opt-2. But not in the new version. Put in debug printout. The key event delivered the same modifier values, ctl+alt, but the key value was... 0x2122, the ™ key? And cmd-alt-3 gave me Qt.KEY_STERLING, 0xA3. And cmd-alt-1 is a dead key.

Pondering ensued. OK, these are the glyphs that you see, if you open the Mac Keyboard viewer widget and depress the Option key. So under Qt5, the keyboard event processor is delivering the OS's logical key, but under Qt4 in the same machine at the same time it delivers the physical key.

Oh dear.

I spent several hours searching stackoverflow and the qt-project forums and bug database but nothing seemed relevant. I posted a query in the Qt forum. But I have little hope. It looks very much as if I'll have to change they key choices for bookmarks, and make them platform-dependent. In Windows and Linux they can continue to be ctl[-alt]-1 to 9, but in Mac OS this will change. The only reliable special key modifiers are control (Command) and meta (the Control key!).

In V1 it was great that I could document just once at the top of the docs, that in Mac, "ctl means cmd" and "alt means option". And that was consistent throughout. Now it won't be because the Option key is effectively dead for my purposes. I'll have to tell the mac user, "when I say control I mean command, but when I say alt, I mean control." Won't that be nice? Plus, I'll have to have code that looks at the platform and redefines the key sets for Mac at startup. Very disappointing.

Funny Little Thing (Solved)

Next day second thoughts: While as noted, QTextDocument inherits from QObject and as such has no font property, it does for no obvious reason have a defaultFont property which it imposes on any Q[Plain]TextEdit to which it is connected. This completely violates the Model/View scheme, the Model imposing a presentation feature on the View, but there it is. Furthermore, it can't be overridden! As I note below, when I interrogate the edit widget for its font().family and font().pointSize, it happily reports the values I'd set. But what it displays is its document's default font, if the document is connected second. So fie on ye, Qt designer.

---

OK, here's an oddity in Qt. I'm working on the editview, a panel mainly containing a QPlainTextEdit that is pretty much the heart of PPQT. But the editor is the "view" and its "model" or repository of data is a QTextDocument. Most of the GUI initialization is handled by the code generated from the Designer. But the __init__ for the editview has to do two key things. Well, there will be lots more, hooking up signals, setting up the syntax highlighter, blah blah, but in its rudimentary state, two things:

• Set its fonts
• Set its document to the edit model

Item two is pretty simple. The parent Book is the repository of all knowledge, so it just does self.Editor.setDocument(self.my_book.get_edit_model()).

Setting the fonts is slightly more work. This is looking ahead to when the user will be able to tell the main window, I want to choose a different UI font, the default font used in most labels and buttons, or I want to choose a different edit font, the had-better-be-monospaced font used only in the editview(s) and one or two other places like the Find text string.

So there can be multiple Books open each with its editview, and up in the main window the user says, "Let's use Courier!" Main window will emit a signal, and any widget that cares better catch it and change fonts.

Also, at open time when setting up a new editview, we want to restore the font size the user had last time the book was open. In version 1, with only one book, that was a global setting, but now there can be multiple documents with potentially each editview zoomed to a different font size! Just one of the many, many features affected by multiple documents.

Anyway, while initializing, the editview needs to ask its Book for the font size and also ask the global font module for the proper font, and set that family/size combo in the QPlainTextEdit. No biggie, a simple method that is called in init. or by the font-change signal:

    def set_fonts(self):
        general = fonts.get_general() # UI font at default size
        self.setFont(general) # set self, propogates to children
        mono = fonts.get_fixed(self.my_book.get_font_size())
        self.Editor.setFont(mono) # the editor is monospaced

This story is going somewhere, really. OK, so the editview initialized like this:

        self.set_fonts()
        self.Editor.setDocument(self.my_book.get_edit_model())

And it didn't work. When it displayed, all the labels would be in the system default font (Lucida Grande 13pt) but so would the edit widget be. I put in a debug print to query the edit widget's font and display it. It happily reported, "I'm using Liberation Mono 16 just like you said, boss", but when I typed in the edit window, it came out in Lucida. I set the parent widget to the mono font, and all the labels displayed in mono but the damn edit text was still Lucida!

After wasting a couple of hours on this, it occurred to me the QTextDocument might be the villain. I reversed the init to

        self.Editor.setDocument(self.document)
        self.set_fonts()

And it worked, the edit text is now in the chosen mono font. WTF? Reviewing the doc for QTextDocument, it inherits from QObject not QWidget, so it doesn't even have a font property. Yet somehow, the call to setDocument() undid the work of a preceding call to setFont(). Maybe changing the document is so basic to the edit widget that it treats it as a reset. Whatever. In my application, I only set the document once. As long as it happens first, all is well.

A First Look at Linguist

So, where were we? I've been away nearly a week, visiting wonderful Ames, Iowa. Not by choice, but by the whimsy of the NCAA Selection Committee, which in its wisdom chose to make the Stanford Women's Basketball team not only a number-2 seed, but make them play the first two rounds in Ames, on the campus of the University of Iowa, where the host school, the 4th-seeded Iowa Cyclones, draw 10,000 screaming fans to every home game.

Well, fortunately the Cyclones faded to a zephyr before the defense of the FSU Seminoles, so for the second game the Cardinal played in front of a half-empty and subdued arena, and won comfortably. Meanwhile we dealt with snow flurries and the difficulties of passing time among the limited amusements of Ames and Des Moines. If you'd rather know about that versus Qt Linguist, check the pictures.

A couple posts back I described the process of designing a widget with Qt Designer and how any string in the design could be designated "Translatable", and how that left distinct code in the generated Python of the widget class. With the result that, when the widget initializes itself, every translatable string will pass through the bowels of the QtCore.QCoreApplication.translate method before being assigned to its QLabel, push button, menu item or whatever its use.

The output of that method—usually just written tr() in the Qt documentation, but for an arcane reason having to do with the relationship of Python classes to C++ classes, PyQt5 needs to always call the Core version not the one inherited by every QObject—the method's output is either the original, or a translated string—if there exists a translation for that string for the current Locale.

But that leaves the question, where do translations come from? From work done by a Translator (a human) using Qt Linguist. I pursued the link between the widget code and Linguist a little further.

The bridge is the PyQt5 utility, pylupdate5. Its use is described in the PyQt5 online docs. One must create a minimal Qt project description file, in this case ppqt2.pro. Actually a make file for the Qt Make program, this file lists the relevant source files and the name of the translation file. Here is what I used:

SOURCES = editview_uic.py
TRANSLATIONS = ppqt2.ts

Listing just one source file now; later there would be many on that line.

Then you turn pylupdate5 loose on the .pro file and it fills up ppqt2.ts with a bunch of XML items like this:

    <message>
        <location filename="editview_uic.py" line="153"/>
        <source>Document filename</source>
        <translation type="unfinished"></translation>
    </message>

Now I could launch Qt Linguist from the Qt distribution, and use it to open the ppqt2.ts file. It presents me with a window whose top is like this:

Every string from every widget (just one widget for now) is shown. Click on one and prepare a translation for it in the bottom part of the window.

For some reason, spaces are shown as gray dots in this part of the window. There exist Qt "phrase books" for many languages, and the French one is open in the above image. It is offering "document" as a translation for "document". Fair enough, but I would have thought "document name" would be a common phrase. Apparently not. I typed in nom du document.

Anyway, that's what the Translator person works with. The texts for the given language would be saved back into the ppqt2.ts file. And somehow become available via the Core Translate method at run-time.

I'm not going to worry further about that last step, for now. I can see how translation would be done. I don't mean to actually do any translations (or request anyone to do any) until the whole app is in near-final state. But at least I know how it all works, I've seen it can work on my system, so that's one Unknown that's Known and I can relax about it.

Assisting the Upgradement

I got burned by one of what turned out to be quite a list of small incompatibilities between PyQt4 and PyQt5. Just fooling around I tried upgrading one of Mark Summerfield's utilities to PyQt5. It contained the following code:

        path = QFileDialog.getOpenFileName(self,
                "Make PyQt - Set Tool Path", label.text())
        if path:
            label.setText(QDir.toNativeSeparators(path))

Pretty obviously Mark expected getOpenFileName to return a path or a null string. But when executed, and I clicked Cancel in the file dialog, it caused an error in the label.setText statement. Whatever got into path evaluated to True, but wasn't a string.

It turned out to be a tuple with two strings. I documented this to the pyqt mailing list and was embarrassed when Phil just replied with the above link to the list of incompatibilities, one of which is a change to the API of the whole family of five "get..." methods supported by QFileDialog. What had happened to cause this seemingly arbitrary breaking of an existing API? It seems that PyQt4 had introduced some variant methods "to avoid the need for mutable strings". Now these extra "get...Filter" methods were being dropped and their function folded into the basic "get..." methods. And that entailed changing the return value of getOpenFileName from a simple string to a tuple of two strings.

It still seems arbitrary to me, breaking existing code in an unexpected way for no very good reason. But it's a done deal, so how to make sure that this incompatibility, and all the other subtle incompatibilities in the list, don't get overlooked? (And don't miss the fact that one item in the list is open-ended, saying "PyQt5 does not support any parts of the Qt API that are marked as deprecated or obsolete in Qt v5.0." What are those? Are they numerous?)

I decided it wouldn't be hard to write a tool to find and point out all, or anyway a lot of, these issues. In two afternoons of work I put together q45aide.py (click the link to see the Readme and get the code from Github). This is a straightforward source scanner that copies a program and inserts comments above any line that looks as if it will have an upgrade problem.

I'm particularly pleased with two features of this program. One is the way of finding out the modules that contain every Qt class. I needed this because one annoying change from Qt4 to Qt5 is that many classes moved from one import module to another. That invalidates most existing from PyQt4.module import (class-list) statements. I wanted to generate correct, minimal import statements from the class-names used in the program. But that meant having a dictionary whose keys were all the valid Qt class-names (over 880 of them, it turns out) and whose values were the module names that contain them.

I pondered quite a while over how to get such a list of class-names by module. I thought about manually or programatically scraping some pages from qt-project.org. But finally I realized, I could build a complete, accurate list dynamically in the program.

When you import a module, Python creates a namespace. And the names defined in a namespace can be interrogated by querying namespace.__dict__. So the program contains code like this:

    def load_namespace( ):
        global module_dict, import_dict
        # pick off QtXxxx from "PyQt5.QtXxxx"
        module_name = namespace.__name__.split('.')[1]
        for name in namespace.__dict__ :
            if name.startswith('Q') : # ignore e.g. __file__
                module_dict[name] = module_name

    import PyQt5.Qt as namespace ; load_namespace()
    import PyQt5.QtBluetooth as namespace ; load_namespace()
    ...

This loads module_dict with exactly the 880+ class-names related to their include modules, automatically updating should PyQt5 be updated with new or changed class-names.

The other thing that I got a kick out of writing was the way to write a list of class-names in either of two formats, in one statement. The program will generate one "from PyQt5.modulename import (class-name-list)" for each module that the input requires. A program option is -v, asking for the list to be stacked vertically. The only difference is that the class-name-list is either punctuated with comma-space, or with comma-newline-indent. And this is how it comes out:

                out_file.write('from PyQt5.{0} import\n   ('.format(mod_name))
                join_string = ',\n    ' if arg_v else ', '
                out_file.write(join_string.join(sorted(class_set)))
                out_file.write(')\n')

Badda-boom.