Continuing to live-ish blog the hunspell hunt

For reference, here is where the experimental code is now.

import os
# set up path strings to a dictionary
dpath = '/Users/dcortes1/Desktop/scratch'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
print( os.access(daff,os.R_OK), os.access(ddic,os.R_OK) )
# Find the library -- I know it is in /usr/local/lib but let's use
# the platform-independent way.
import ctypes.util as CU
libpath = CU.find_library( 'hunspell-1.3.0' )
# Get an object that represents the library
import ctypes as C
hunlib = C.CDLL( libpath )
# Define the API to ctypes
hunlib.Hunspell_create.argtypes = [C.c_wchar_p, C.c_wchar_p]
hunlib.Hunspell_create.restype = C.c_void_p
hunlib.Hunspell_destroy.argtypes = [ C.c_void_p ]
hunlib.Hunspell_get_dic_encoding.argtypes = [C.c_voidp]
hunlib.Hunspell_get_dic_encoding.restype = C.c_char_p
hunlib.Hunspell_spell.argtypes = [C.c_void_p, C.c_char_p]
hunlib.Hunspell_spell.restype = C.c_uint
# Make the Hunspell object
hun_handle = hunlib.Hunspell_create( daff, ddic )
# Check encoding
print(hunlib.Hunspell_get_dic_encoding( hun_handle ))
# Check spelling
for s in [ 'a', 'the', 'asdfasdf' ] :
    b = bytes(s,'UTF-8','ignore')
    t = hunlib.Hunspell_spell( hun_handle, b )
    print(t, s)
# GCOLL the object
hunlib.Hunspell_destroy( hun_handle )

Let's see if changing the create argtypes makes a difference.

Bingo! Made the following changes. One, change the argtypes of create():

hunlib.Hunspell_create.argtypes = [C.c_char_p, C.c_char_p]

That caused a ctypes error on the call _create(daff,ddic), because a Python3 string is not compatible with c_char_p. So encode the strings:

baff = bytes(daff,'UTF-8','ignore')
bdic = bytes(ddic,'UTF-8','ignore')
hun_handle = hunlib.Hunspell_create( baff, bdic )

Et voila, the output is

b'UTF-8'
1 a
1 the
0 asdfasdf

Most excellent! I have achieved my goal of invoking Hunspell for spell-checking without use of the pyhunspell package. I am not sure if I want to change my existing dictionaries.py to do this in place of relying on the package. For sure, if I have even the slightest trouble installing the package on Windows, I will be quick to fall back on this.

It's like live-blogging, almost

Ok. I have figured out one issue. I looked at my own code that creates a dictionary and noticed that it presents the two path arguments to hunspell.Hunspell() with .dic first, .aff second. And that is documented in the hunspell package doc. Making that change, the hunspell package works. It correctly notes the Greek dictionary is UTF-8 and spells a word.

Hgr = hunspell.HunSpell(pdic,paff)
Hgr.get_dic_encoding()
'UTF-8'
Hgr.spell('α')
True

And if I present it with my en_US dictionary saved in UTF-8, it opens it correctly also. This is rather bad, in that anyone looking at the Hunspell doc at the Hunspell sourceforge page will see "Hunspell(const char *affpath, const char *dpath);" which is exactly the reverse of the hunspell package. If you present the files in the reverse order (the correct order per the man page), the Hunspell object is created, no error is reported, but it can't check spelling, calls any input misspelled.

What about my ctypes invocation? Well, that definitely uses the C-defined function which should take the .aff first, the .dic second. Checking the pyhunspell code it definitely passes the aff-path first, dic-path second, which is what my ctypes invocation is doing.

I do note a comment in the pyhunspell code, "Some versions of Hunspell_create() will succeed even if there are no dictionary files." So that's probably what's happening: for some reason it is not opening the path strings I am passing, and it silently fails and defaults to a rather useless, and undetectable, no-dictionary condition.

The likeliest cause of that is it is not getting the path strings in a form it expects. Maybe it can't handle c_wchar_p after all. Before I experiment with that, I am going to add a call to destroy the Hunspell object. Unlike a PyQt object, it isn't known to Python. I may be memory-leaking a Hunspell object every time I run my test code.

Continuing ctypes and hunspell

Right, so we have used ctypes to locate the Hunspell dylib and invoke the Hunspell_create() function returning a handle to a C++ object of class Hunspell. That demonstrated that Python 3 strings could be passed to a C function that expected const char * parameters.

Then we invoked a method of the Hunspell object by calling the C wrapper Hunspell_get_dic_encoding(), and huzzah! it returned what it was supposed to return, Hunspell's belief about the encoding of the dictionary's .dic and .aff files. It returned 'ISO8859-1' which may prove to be significant.

Next was to try to invoke the most important method, spell(). If this works, I can toss the whole hunspell.py package and just use fewer than 20 lines of ctype code (maybe 30 lines, adding code for platform dependencies). Hunspell has a dozen other methods, suggestions, stemming, etc., but all I need is spell(word) yielding 0 for bad and nonzero for good. The C header file says,

LIBHUNSPELL_DLL_EXPORTED int Hunspell_spell(Hunhandle *pHunspell, const char *);

Translating to Python,

hunlib.Hunspell_spell.argtypes = [C.c_void_p, C.c_wchar_p]
hunlib.Hunspell_spell.restype = C.c_uint

OK, let's do it!

for s in [ 'a', 'the', 'asdfasdf' ] :
    t = hunlib.Hunspell_spell( hun_handle, s )
    print(t, s)

Output:

b'ISO8859-1'
0 a
0 the
0 asdfasdf

Not good! Neither "a" nor "the" are valid words.

My diagnosis is this. I know the dictionary was opened successfully, and the words are in it. Either the word is not being passed correctly, or Hunspell is not comparing it correctly. I tried several variations on passing the argument, for example I changed the argtypes to show it as taking a c_char_p (no change), and then converted the word (b = bytes(s,'ISO-8859-1','ignore')) and passed the byte string (no change) and again encoding as UTF-8 (no change).

It stands out that Hunspell thinks the dictionary is encoded Latin-1. It lurks somewhere in my memory that I solved a similar problem by converting the dictionary to UTF-8 encoding. The encoding of the .dic file is specified in the .aff file in a SET statement. So I opened both files in BBEdit and saved them as UTF-8, also changing the .aff file to read SET UTF-8 (which is the same as the SET statement in a Greek dictionary). Tried again.

b'ISO8859-1'
0 a
0 the
0 asdfasdf

Wait, what? The SET statement says, and the actual file encodings are, UTF-8, but get_dic_encoding returns ISO8859-1? Just in case there's some kind of file caching going on, I copy the dictionary files to a different folder and change the path string to match. No change! I re-save the files as UTF-8 "with BOM". No change; it still returns ISO8859-1.

Now I doubt my prior diagnosis. Hunspell is ignoring the content of the dictionary, which possibly means it isn't reading it at all. Maybe it is failing to open the files, not reporting a failure, and returning some kind of default?

Does the hunspell package do the same?

import hunspell
import os
dpath = '/Users/dcortes1/Desktop/scratch'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
Hobj = hunspell.HunSpell(daff, ddic)
Hobj.get_dic_encoding()
'ISO8859-1'
Hobj.spell('the')
False

OK, I am officially flabbergasted. My flabber is gast. That dictionary is UTF-8 and defines "the". It is nice, in a way, that the package (that works fine inside PPQT1 and 2) is failing exactly as my ctypes experiment failes, but what the heck am I doing wrong?

I'll post again when I understand more.

Update: this much is confirmed:

Hobj = hunspell.HunSpell('aasdf/asdf/asdf.aff','aasdf/asdf/asdf.dic')
Hobj.get_dic_encoding()
'ISO8859-1'

If the Hunspell object creation cannot open the .aff/.dic files, it fails silently and uses a null dictionary with a default encoding. And I don't see any way of testing whether this has happened or not!

Exploring ctypes

Implementing spellcheck has been a constant problem for me. I solved it, awkwardly, using the pyhunspell package (see below for another link). This provides a Python interface to the Hunspell checker.

There is nothing at all wrong with Hunspell itself. It is complete, fast, and still supported. It is superior to Aspell and Myspell in several ways. Most importantly it supports UTF, and so can be used to spellcheck German, Greek, and the like.

My issues are, or were, with the pyhunspell package. It went unsupported for a long time. It didn't support Python3 until a user posted the necessary small changes as a comment on an issue. And getting it compiled and working on Windows was, for me, a huge problem. So I wanted to experiment to see if I could access the hunspell library direct from Python using ctypes, eliminating the need to compile a wrapper. Important note: I just discovered that pyhunspell was very recently picked up by a new owner, Benoît Latinier, and rehosted on github: here is its new home. Another user has posted a binary package for Windows on the old site; unfortunately it's for Python 2.7. So things are looking up for pyhunspell. Which is a good thing, because as I will now finally get around to saying, the ctypes experiments are not going super-well.

We start with getting access to the library.

# Find the library -- I know it is in /usr/local/lib but let's use
# the platform-independent way.
import ctypes.util as CU
libpath = CU.find_library( 'hunspell-1.3.0' )
# Get an object that represents the library
import ctypes as C
hunlib = C.CDLL( libpath )

To do spell-checking, one must create a Hunspell object. The C header declares:

typedef struct Hunhandle Hunhandle;
LIBHUNSPELL_DLL_EXPORTED Hunhandle *Hunspell_create(const char * affpath, const char * dpath);

Converting that to ctypes, we have:

hunlib.Hunspell_create.argtypes = [C.c_wchar_p, C.c_wchar_p]
hunlib.Hunspell_create.restype = C.c_void_p

OK, let's call it!

dpath = '/blah/blah...'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
hun_handle = hunlib.Hunspell_create( daff, ddic )

Well, nothing crashed. At this point we should be able to use methods of the Hunspell object. Back to the C header file:

LIBHUNSPELL_DLL_EXPORTED char *Hunspell_get_dic_encoding(Hunhandle *pHunspell);

In Python/ctypes:

hunlib.Hunspell_get_dic_encoding.argtypes = [C.c_voidp]
hunlib.Hunspell_get_dic_encoding.restype = C.c_char_p
print(hunlib.Hunspell_get_dic_encoding( hun_handle ))

And whoop-de-doo, it prints b'ISO8859-1'. To review: we have successfully loaded the library, created a Hunspell object, and invoked one of its methods. During creation, the object correctly loaded the dictionary that was passed. Ergo, we can pass a Python3 string into a const char * argument. This is looking great!

And this post is getting long. I will continue with actual spell-checking next time. Spoiler alert! It doesn't go well!

Preferences done, still waiting

I completed the Preferences dialog. It looks about the same as the test version shown in the prior post, plus the addition of buttons for Defaults, Cancel, Apply and OK.

Getting things to work smoothly with the "Apply" button took a bit of work. There are four highlight types the user can change: the editor current line, the text of a limited Find/Replace range, spelling error words, and scanno words. The first two use one mechanism, the second two use a completely different one.

As noted in some post I can't be arsed to look up now, the current-line highlight is done using an "extra selection", and the find-range highlight is done using another. An extra selection is a peculiar thing unlike any other class in Qt (that I know of; maybe the graphics area has similar things). It has no behaviors, no methods; it is basically a two-ple of a cursor and a format (QTextCursor and QTextCharFormat). You give your QTextDocument a list of your extra selection objects and it applies each one's format on its cursor's selection.

The current line's cursor gets updated whenever the edit cursor moves. The find-range cursor gets updated only when the user toggles the In Selection switch in the Find panel.

Their formats get updated only when the Preferences dialog calls the colors module set_current_line_format or set_find_range_format. At that time, the colors module emits a signal, ColorsChanged, which is fielded by editview, and it refreshes the formats in its list of two extra selections.

Therein lies one problem. The user can choose a new format for current line or find range, and click the Apply button in Preferences, and the the highlighting in the visible part of the Edit panel should change immediately. But it didn't, apparently because signals don't get processed while a modal dialog is up. However, I added a call to QCoreApplication.processEvents() in the Apply logic, and then, ta-daa, those two highlights changed instantly upon Apply.

The spellcheck and scanno highlights are created by a different mechanism. They are applied by a QSyntaxHighlighter. Syntax highlighting is turned on by assigning a document to the highlighter, and turned off by assigning a null document to it. But the highlights it applies don't change once they are set until the highlighted text is hidden and shown again, e.g. by paging the document in the editor.

So those two highlights didn't change even though the editor was getting control in its ColorsChanged signal slot. I had to add logic to this slot to ask, is highlighting of either scanno or spelling now active? If so, turn highlighting off, and turn it on again. That forces re-tagging of visible words. With that change, the visible highlighting changes instantly when Apply is clicked.

I am a bit concerned about this last, because in Version 1, there was a significant delay when you turned on the syntax highlighter in a large document. I assumed this was because at that time it would go over the whole document passing every text block through the highlighter. If Qt5 behaves like Qt4, clicking the Apply button in Preferences might incur a significant (1-3 second) stall when the spelling or scanno highlight has changed. That delay isn't noticeable now. There's no perceptible delay with a 25-page document. I hope that Qt5 is smarter and only invokes the highlighter for visible text. Even if there is a delay, these highlights are not something you'd change often.

For now it all works nicely. Doing Preferences was supposed to fill the time until PyQt5.4 was out. I'm done, and it isn't. Now what?

Fiddling with Fonts

This week, waiting for PyQt5.4 to be released, I've been working on the Preferences dialog, and it's coming out rather interesting. Alongside here is a test version.

It's a stack of items. I built a whole little O-O heirarchy to make this. There's a parent class, ChoiceWidget, that displays a title line and on mouse-over, changes color and puts an explanation in the explainer box at the bottom. For the path-entry items, there's a PathChooser class that implements the path line-edit and a browse button, and when focus leaves the line-edit, checks that the given path is accessible according to some criterion (R_OK or X_OK) passed to its initializer, and if not, beeps and makes the line-edit pink. For the text-format items, there's a FormatChooser class that implements the color swatch and the sample highlight.

And then there's the font-chooser. Here I wanted to use QFontComboBox, which displays the available fonts using those fonts. Unfortunately it is rigidly designed. When told to display only monospaced fonts, it displays only the fonts the QFontDatabase thinks has that property.

Unfortunately, I am including two monospaced fonts with the program, Liberation Mono and another that I only just discovered, Cousine from the Chrome Core set. Cousine is basically the same as Liberation Mono except it has even more Unicode glyphs.

Either of these is a better choice than the next-best font, Courier New. The latter has about as wide a Unicode repertoire but it has a poor contrast between 1/l and 0/O. Alas, the QFontDatabase will not recognize either of the fonts that I am loading (using QFontDatabase.addApplicationFont()) as being actually monospaced. It will not return their names when asked for a list of monospaced fonts.

Which means that if QFontComboBox is set to display only monospaced fonts, it will not display the two that I most want the user to have access to. And although it claims to support the methods of its parent QComboBox, it ignores a call to addItem(string). So I can't add them. If I don't tell it to show only monospaced fonts, it of course shows every font available in the system, and takes quite a while to open first time.

I wasted quite a bit of time today trying to remedy this, first by trying to find some way to get QFontComboBox to show all but only the fonts I wanted it to show (the known monospaced fonts plus my two); then by trying to find a way to change my fonts so that QFontDatabase would recognize them as monospaced. That entailed a very lengthy search for a free or cheap TrueType font editor that would let me verify and set what I presume is a one-bit flag in the font file format that says "really, I'm monospaced." Didn't find one. Well, one, but it costs a bundle and its 30-day free trial version will not allow saving a modified font. So fuck them.

Actually, after all that, I'm not sure whether any amount of editing would help. The Mac OS Font-Book application does know these two are monospaced. It lists them when I make a "smart search" for monospaced fonts. So it may be that QFontDatabase is just prejudiced against added application fonts.

In the end I used a plain QComboBox loaded with the family names of monospaced fonts. I set it up so that when you make a selection from the list, the "explanation" box at the bottom changes to use that font.

I meant to finish the Preference dialog today, I really did. But the fiddling with fonts killed too much time. What's left is to implement the dialog widget itself, including the important "Set Defaults", "Apply", "Cancel" and "OK" buttons. A few hours more.

Power of missing parentheses

I've completed the Footnotes panel. Its testing was rather minimal, although I believe sufficient. The footnote "model" did receive a rather thorough fnotdata_test.py module that exercises every branch and error condition. With a solid model, the view/controller piece goes together very quickly. I set up a fnotview_runner.py that starts up the app, loads a document full of footnotes of various types, and sits waiting for interaction. I used this to exercise all the functions manually, and quickly ran into a vexing problem that took a couple of hours to track down. When I knew the cause, I was even more vexed. Look at the following code fragment. Hands up all those who see the glaring, obvious, stupid error?

        self.model.beginResetModel()
        worktc = self.edit_view.get_cursor()
        worktc.beginEditBlock()
        try:
            for j in range(self.data.count()):
                # ...twenty lines of code computing a new key value
                # for footnote j...
                if new_key is not None :
                    self.data.set_key(j, new_key, worktc)
            # end of for j in range of keys
        except Exception as whatever:
            fnotview_logger.error(
                'Unexpected error renumbering footnotes: {}'.format(whatever.args)
                )
        worktc.endEditBlock
        self.model.endResetModel()

Here's what's going on. The user has clicked Renumber. The view/controller tells its QTableModel that data is changing. It obtains a QTextCursor on the document and starts an "edit macro" on it, so that all changes made using it will be a single Undo. For each existing footnote key it computes a new renumbered value, and calls the data model to set that as the new Key value in both the Anchor and Note of footnote j.

The model gets passed the working text cursor and uses it when it updates the Key value in the Anchor ([key]) and the Note ([Footnote key:...). At the end of the loop, even if there were errors, the edit macro is closed and the table model is told it can refresh itself.

When executed, this caused all sorts of flaky behavior in the editor. The new Key values would not appear unless I made the page scroll. Undo did not always undo the changes. Doing new changes compounded the problems.

I spent quite a bit of time over two days inserting print statements and tracing and... before I finally noticed that glaring error that you, dear reader, spotted five minutes ago. Fix that and suddenly everything worked "just swellegant" as my late father liked to say.

Well, live and learn.

Or not.

Looking ahead, again

Anyway, that's done, and also I fixed a serious, if obscure, bug in Version 1 and released new packages for it. A few posts back I put up the current to-do list. A month has passed; time to revisit it.

• When Qt5.4 and the matching PyQt are available, install those on the new iMac and move development to there.
• Qt5.4 is out, PyQt5.4 is expected any minute. So this should happen next week I hope.
• Then, bring CoBro up to the Qt5.4 level and replace the execrable WebKit browser with the new WebEngine one.
• Then, use Cobro as a test-bed for learning how to use pyqtdeploy to bundle an app. I am eager to find out if this is truly a way to make a self-contained executable on all 3 platforms, in place of pyinstaller.
• Presuming that works (and that the new web engine fixes the frequent crashes induced by webkit), release CoBro on all three platforms.
• Then, or right now to pass the time waiting for Qt5.4, code the footnotes module. That will go fast; most of the code can be lifted out of version 1...
• It took a bit more work than that, but it's done.
• Then, or right now if PyQt5.4 is delayed, implement the Preferences dialog.

At that point — which will in no way be reached in calendar 2014 — PPQT2 will be at what might be called an alpha state, that is, with adequate function that an experienced user could post-process a book with it. That user would have to run from source, however, until the pyqtdeploy work is complete.

The work to be done after that includes:

• Writing the translation interface module, which includes figuring out how to dynamically load translator modules.
• Writing the plain-ascii example translator
• Writing the HTML example translator
• Bringing the "keyboard palettes" of V1 forward to V2 and making them load dynamically (using the same scheme as the translators?)
• Finally going back into the UI and make panels drag-out-able, applying the drag-drop research with which I began this series of posts many months ago.
• Writing the Help file and adding the Help panel
• Rewriting the "suggested workflow" document to reflect all the changes; for this I will want to actually post-process a book myself to make sure I know the best way to use the app.
• Make some screencasts to explain PPQT and show its features.