Webkit, a never-failing source of laughs

So yesterday I wrote how the Qt5.4 Webkit seems to be behaving itself. And it is, compared to the 5.2 and 5.3 versions. It actually displays all the hard comics without screwing up the rendering. And so far it hasn't crashed in any of the common ways I summarized in the previous post.

However, the following is the slightly edited console output of running it today.

[08:10:09 CoBro] python ./cobro.py
2014-12-31 08:10:21.581 Python[68621:14259725] Cannot find executable for CFBundle 0x10858fd00
</Users/dcortesi/Library/Internet Plug-Ins/DjVu> (not loaded)
2014-12-31 08:10:21.589 Python[68621:14259725] Error loading /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin:  dlopen(/Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin, 265): no suitable image found.  Did find:
 /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin: mach-o, but wrong architecture
GVA info: Successfully connected to the Intel plugin, offline Gen75 

That was just for starters. Then as I got down toward the end of the list, after viewing a dozen comics without further messages, I clicked on Gunnerkrig Court and the console filled up with this.

[08:16:58.637] FigLimitedDiskCacheProvider_CopyProperty signalled err=-12784 (kFigBaseObjectError_PropertyNotFound) (no such property) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/FigByteStreamPrototypes/FigLimitedDiskCacheProvider.c line 947
<<<< FigByteStream >>>> FigByteStreamStatsLogOneRead: ByteStream read of 8 bytes @ 4453661 took 0.573590 sec. to complete, 1 reads >= 0.5 sec.
Dec 31 08:17:18 Silver-streak-2.local rtcreporting[68621] : logging starts...
Dec 31 08:17:18 Silver-streak-2.local rtcreporting[68621] : setMessageLoggingBlock: called
[08:17:18.935] itemasync_GetDuration signalled err=-12785 (kFigBaseObjectError_Invalidated) (invalidated item) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/Player/FigPlayer_Async.c line 2870
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : startConfigurationWithCompletionHandler: Cached 0 enabled backends
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : setUserInfoDict: enabled backends: (
 )
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : initWithSessionInfo: XPC connection invalid
Dec 31 08:23:14 Silver-streak-2.local Python[68621] : CGContextSaveGState: invalid context 0x126544940. This is a serious error. This application, or a library it uses, is using an invalid context  and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update.
Dec 31 08:23:14 Silver-streak-2.local Python[68621] : CGContextScaleCTM: invalid context 0x126544940. This is a serious error. This application, or a library it uses, is using an invalid context  and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update.

The above message repeated 15 more times.

Do I need to analyze these things? Is there a single gorram thing I can do to prevent them?

Sigh. OK, one at a time. Inserting newlines so blogger can display them.

2014-12-31 08:10:21.581 Python[68621:14259725] Cannot find executable for
CFBundle 0x10858fd00
</Users/dcortesi/Library/Internet Plug-Ins/DjVu> (not loaded)

Some webcomic wants a "DjVu" plugin. This is what I get for telling the browser, PluginsEnabled(True). Because so many comics rely on Flash, you see. What is "DjVu"? I suppose I could find out, but supposing I did, is there any way I could ensure it would always be available wherever Cobro runs? No. And why oh why cannot Webkit fail silently and just put up a broken-plugin icon? Why does it have to blabber about it on the console?

Then we have this beautiful message, let's break it down by parts.

Error loading /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin:

Another plugin the browser can't find. I don't care! Nobody cares. Just shut up.

dlopen(/Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin,
265): no suitable image found.

My heart bleeds for you.

Did find: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime
Plugin: mach-o, but wrong architecture

I'm still not caring.

GVA info: Successfully connected to the Intel plugin, offline Gen75 

Good job, Webkit! If I knew what "GVA info" was, I'd probably be very impressed. What do you suppose it means, "offline Gen75"?

Then there's this wonderful streak of completely opaque babble:

FigLimitedDiskCacheProvider_CopyProperty signalled err=-12784
(kFigBaseObjectError_PropertyNotFound) (no such property) at
/SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/FigByteStreamPrototypes
/FigLimitedDiskCacheProvider.c line 947
<<<< FigByteStream >>>> FigByteStreamStatsLogOneRead: ByteStream read of 8 bytes @ 4453661
 took 0.573590 sec. to complete, 1 reads >= 0.5 sec.

This should really be bronzed and hung in a place of honor in the Hall of Indecipherable and Utterly Useless Messages. It is telling me about an error -12784, apparently having to do with some undocumented cache not finding some arcane property. But instead of explaining what that error means or how to fix it, it natters on about how it read 8 bytes in one-half second. I can personally read a lot faster than that, so I don't think a 4-core 3.5Ghz machine should be bragging about this.

The next part is just poetry. Lightly edited for aesthetic enjoyment,

: logging starts...
: setMessageLoggingBlock: called
=itemasync_GetDuration signalled err=-12785
   (kFigBaseObjectError_Invalidated)
   (invalidated item) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/Player/FigPlayer_Async.c line 2870
: startConfigurationWithCompletionHandler: Cached 0 enabled backends
: setUserInfoDict: enabled backends: ( )
: initWithSessionInfo: XPC connection invalid

So helpful! So informative! So much not wanted by anybody... And then we have 16 repetitions of

: CGContextSaveGState: invalid context 0x126544940.
  This is a serious error.
  This application, or a library it uses, is using an invalid context
  and is thereby contributing to an overall degradation of
  system stability and reliability.
  This notice is a courtesy: please fix this problem.
  It will become a fatal error in an upcoming update.

Oh, it's a "courtesy" is it? And I am to "fix" the "problem" which is happening where and caused by what, and fix it how? Spare me your passive-aggressive manipulative bullshit and fix your own fucking bugs, thank you very much.

---

Maybe later today I'll google some of this horse-hockey. Or tomorrow; that'll be a good way to start a new year. Hope yours starts better.

Postponing WebEngine, trying Nuitka

I posted a polite & constructive note to the WebEngine development list asking for a timeline on when some of the missing features might appear. No replies as yet.

Trying Cobro under Qt5.4 I am very pleased to see that the Webkit version is now behaving itself. There were several comics that it would display incorrectly. This failure was always that some small image file on the page would be shown at 1000% zoom and overlay the rest of the rendered page. It made Penny Arcade, for example, unreadable.

In addition there were at least four failure modes I've been documenting over the past few months of daily use. In no particular order,

• Emitting a stream of a hundred or more messages "Critical failure: the LastResort font is unavailable", then a segfault.
• "QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files... Abort trap: 6"
• Segfault 11 somewhere deep in the Mac OS innards, usually with QNetworkConfiguration in the stack trace.
• Segfault nested 50 or more calls deep in Webkit rendering code.

It typically crashed in one of these ways every 3 or 4 times I ran it. That was clearly unacceptable. I didn't mind it, just sigh and restart it, but I could never distribute a program that was so unreliable. I was really looking forward to changing to WebEngine and leaving all those nasty bugs behind. Unfortunately that appears not to be, because WebEngine doesn't support several functions that, although minor, I want to have for proper operation. The ability to command "private" (uncached) browsing, for example. It would be unacceptable to have the residue of browsing comics in one's work-browser's cache. Or the ability to implement the custom context menu I wrote, that lets you right-click on a link and make it open in the default browser. Can't do that under WebEngine because it lacks the feature of Webkit that lets you find out what type of data was under the mouse, and if a URL, get its text.

However, after several uses, the 5.4 Webkit browser has displayed none of the above problems. It handles Penny Arcade and Stand Still Stay Silent perfectly, and they were unreadable with 5.3. And none of the crashes has happened yet either.

So, I'm going to shelve the conversion to WebEngine for now. I've gone ahead to put in a couple of minor feature enhancements that I had in mind for some time, and I will continue reading comics with it daily, and just maybe it will prove reliable.

Meanwhile, I downloaded Nuitka and tried to install it. Unfortunately, although it claims to support all of Python through 3.4, the code executed by its setup.py displayed a number of syntax errors that were clearly due to byte-compiling Python 2 code under Python 3.4—missing parens on a print statement, for example.

So I joined yet another goddam dev mailing list and sent a polite query about this. Maybe tomorrow I'll get a reply from one group or the other.

Tangled up in Git

Here's how I thought git worked. I start a branch, (git checkout -b newbranch) and edit a file and save it. Have not committed anything yet but changed plenty.

Suddenly realize, I need to try this thing out in its earlier version. So I thought I could just do git checkout master and any changes made on newbranch would be swept under the carpet and I'd be back to the master version.

Did that: branched cobro, made a shit-ton of changes to cobro.py to switch it to WebEngine, realized, wait, I need to check how something worked with WebKit. Did "checkout master" thing and... cobro didn't change. Didn't go back to what it was. Still had all the WebEngine changes in it. I ended up doing a git revert HEAD, which lost all the changes, made a copy cobro-webkit.py, and now I get to make those changes again.

I have no idea what went wrong, or indeed if it was wrong or expected behavior. (Possibly it happened because the file is on the dropbox folder, maybe my laptop was fighting the desktop?)

Also I discovered two new things that QWebKit offers and QWebEngine does not. One is setting the web page non-editable. The other is "link delegation" where anytime a link is clicked, it emits a signal and lets you decide if it should go through. So now I have three posts on this theme at the qt-project forums. Sad to say, only one has drawn a reply. Although that reply points to the WebEngine dev mailing list, so maybe I will learn something from that.

First steps with QWebEngine

Needing to kill some time this evening while my wife worked on her website, I sat down and began the process of converting Cobro from QWebView, QWebPage, QWebSettings to use their modern but more verbose counterparts, QWebEngineView, QWebEnginePage and QWebEngineSettings. Things look good but as usual there are some issues.

Thanks to my splendidly modular and well-organized coding style, there is really only one object affected in the whole program, a QWebView derived class. I changed its parent class from QWebView to QWebEngineView, and then began going through all the self-initializing lines I had used to set up the QWebView.

Many of them had direct equivalents. For example, self.page() returns the underlying QWebEnginePage just as it previously returned the QWebPage. One still uses self.settings().setAttribute(something, True/False) for setting properties. However, I quickly discovered that several of the settable properties of QWebSettings are not available in QWebEngineSettings.

In particular, there is no JavaEnabled, no PluginsEnabled, and no PrivateBrowsingEnabled. I put a query about these three on the qt forum. It seems strange they would go to some trouble to make the new API compatible with the old, and yet leave out settings, and fairly significant settings at that.

Also my app implements a custom context menu. In it, I use the following to check to see if the thing that was right-clicked upon is a URL. Call self.page().currentFrame() to access the QWebFrame in which the right click occurred. Ask that for hitTestContent(QPoint) to find out what was under the point of the mouse event. If it is a URL, then I can access the text of that URL, and offer the user to open it in the default browser. This is very useful; Cobro is not a full-function browser, and if there is something on a comic page besides the comic, it is handy to be able to jump to a real browser.

Unfortunately it does not appear that QWebEnginePage offers access to the current frame. And there is no QWebEngineFrame class. So although QWebEnginePage specifically documents that you can implement a custom context menu, it is not clear how one can find out what was under the right-click. I also posted a query about this in the qt forum.

However, I just commented out all the parts that I couldn't immediately translate, and ran the app, and up it came. And displayed a comic very nicely. The progress bar signals worked, the titleChanged signal worked. So that was a useful couple of hours.

Py/Qt5.4 at last

So Phil at Riverbank computing dropped a Christmas present on us all, with PyQt5.4 on December 24th. Today I installed Qt5.4, the latest SIP, and PyQt5.4 on my new iMac and started working on Cobro. This all went very well. The installations went smoothly. I spent a couple of hours reviewing the code of Cobro in Wing IDE. I tidied up some comments. I removed some unnecessary global statements. And then ran it, and was pleased to find that this version can display some comics that it cannot running in 5.2. Apparently some of the many bugs in QWebkit have been fixed. Doesn't matter; my next move (on Monday) will be replace the webkit element with the new QWebEngine ones. Following that, probably Tuesday, I have a couple of minor functional enhancements to add.

Toward the end of the week, or next week, the next step I think will be to try compiling Cobro with Nuitka. In theory, the result of compiling a program like Cobro in Nuitka should be a single, stand-alone, self-supporting executable. If so, that would bypass the need for a packager like pyinstaller.

If the Nuitka experiment doesn't work, I will, as previously planned, start trying to use pyqtdeploy on it.

Continuing to live-ish blog the hunspell hunt

For reference, here is where the experimental code is now.

import os
# set up path strings to a dictionary
dpath = '/Users/dcortes1/Desktop/scratch'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
print( os.access(daff,os.R_OK), os.access(ddic,os.R_OK) )
# Find the library -- I know it is in /usr/local/lib but let's use
# the platform-independent way.
import ctypes.util as CU
libpath = CU.find_library( 'hunspell-1.3.0' )
# Get an object that represents the library
import ctypes as C
hunlib = C.CDLL( libpath )
# Define the API to ctypes
hunlib.Hunspell_create.argtypes = [C.c_wchar_p, C.c_wchar_p]
hunlib.Hunspell_create.restype = C.c_void_p
hunlib.Hunspell_destroy.argtypes = [ C.c_void_p ]
hunlib.Hunspell_get_dic_encoding.argtypes = [C.c_voidp]
hunlib.Hunspell_get_dic_encoding.restype = C.c_char_p
hunlib.Hunspell_spell.argtypes = [C.c_void_p, C.c_char_p]
hunlib.Hunspell_spell.restype = C.c_uint
# Make the Hunspell object
hun_handle = hunlib.Hunspell_create( daff, ddic )
# Check encoding
print(hunlib.Hunspell_get_dic_encoding( hun_handle ))
# Check spelling
for s in [ 'a', 'the', 'asdfasdf' ] :
    b = bytes(s,'UTF-8','ignore')
    t = hunlib.Hunspell_spell( hun_handle, b )
    print(t, s)
# GCOLL the object
hunlib.Hunspell_destroy( hun_handle )

Let's see if changing the create argtypes makes a difference.

Bingo! Made the following changes. One, change the argtypes of create():

hunlib.Hunspell_create.argtypes = [C.c_char_p, C.c_char_p]

That caused a ctypes error on the call _create(daff,ddic), because a Python3 string is not compatible with c_char_p. So encode the strings:

baff = bytes(daff,'UTF-8','ignore')
bdic = bytes(ddic,'UTF-8','ignore')
hun_handle = hunlib.Hunspell_create( baff, bdic )

Et voila, the output is

b'UTF-8'
1 a
1 the
0 asdfasdf

Most excellent! I have achieved my goal of invoking Hunspell for spell-checking without use of the pyhunspell package. I am not sure if I want to change my existing dictionaries.py to do this in place of relying on the package. For sure, if I have even the slightest trouble installing the package on Windows, I will be quick to fall back on this.

It's like live-blogging, almost

Ok. I have figured out one issue. I looked at my own code that creates a dictionary and noticed that it presents the two path arguments to hunspell.Hunspell() with .dic first, .aff second. And that is documented in the hunspell package doc. Making that change, the hunspell package works. It correctly notes the Greek dictionary is UTF-8 and spells a word.

Hgr = hunspell.HunSpell(pdic,paff)
Hgr.get_dic_encoding()
'UTF-8'
Hgr.spell('α')
True

And if I present it with my en_US dictionary saved in UTF-8, it opens it correctly also. This is rather bad, in that anyone looking at the Hunspell doc at the Hunspell sourceforge page will see "Hunspell(const char *affpath, const char *dpath);" which is exactly the reverse of the hunspell package. If you present the files in the reverse order (the correct order per the man page), the Hunspell object is created, no error is reported, but it can't check spelling, calls any input misspelled.

What about my ctypes invocation? Well, that definitely uses the C-defined function which should take the .aff first, the .dic second. Checking the pyhunspell code it definitely passes the aff-path first, dic-path second, which is what my ctypes invocation is doing.

I do note a comment in the pyhunspell code, "Some versions of Hunspell_create() will succeed even if there are no dictionary files." So that's probably what's happening: for some reason it is not opening the path strings I am passing, and it silently fails and defaults to a rather useless, and undetectable, no-dictionary condition.

The likeliest cause of that is it is not getting the path strings in a form it expects. Maybe it can't handle c_wchar_p after all. Before I experiment with that, I am going to add a call to destroy the Hunspell object. Unlike a PyQt object, it isn't known to Python. I may be memory-leaking a Hunspell object every time I run my test code.

Continuing ctypes and hunspell

Right, so we have used ctypes to locate the Hunspell dylib and invoke the Hunspell_create() function returning a handle to a C++ object of class Hunspell. That demonstrated that Python 3 strings could be passed to a C function that expected const char * parameters.

Then we invoked a method of the Hunspell object by calling the C wrapper Hunspell_get_dic_encoding(), and huzzah! it returned what it was supposed to return, Hunspell's belief about the encoding of the dictionary's .dic and .aff files. It returned 'ISO8859-1' which may prove to be significant.

Next was to try to invoke the most important method, spell(). If this works, I can toss the whole hunspell.py package and just use fewer than 20 lines of ctype code (maybe 30 lines, adding code for platform dependencies). Hunspell has a dozen other methods, suggestions, stemming, etc., but all I need is spell(word) yielding 0 for bad and nonzero for good. The C header file says,

LIBHUNSPELL_DLL_EXPORTED int Hunspell_spell(Hunhandle *pHunspell, const char *);

Translating to Python,

hunlib.Hunspell_spell.argtypes = [C.c_void_p, C.c_wchar_p]
hunlib.Hunspell_spell.restype = C.c_uint

OK, let's do it!

for s in [ 'a', 'the', 'asdfasdf' ] :
    t = hunlib.Hunspell_spell( hun_handle, s )
    print(t, s)

Output:

b'ISO8859-1'
0 a
0 the
0 asdfasdf

Not good! Neither "a" nor "the" are valid words.

My diagnosis is this. I know the dictionary was opened successfully, and the words are in it. Either the word is not being passed correctly, or Hunspell is not comparing it correctly. I tried several variations on passing the argument, for example I changed the argtypes to show it as taking a c_char_p (no change), and then converted the word (b = bytes(s,'ISO-8859-1','ignore')) and passed the byte string (no change) and again encoding as UTF-8 (no change).

It stands out that Hunspell thinks the dictionary is encoded Latin-1. It lurks somewhere in my memory that I solved a similar problem by converting the dictionary to UTF-8 encoding. The encoding of the .dic file is specified in the .aff file in a SET statement. So I opened both files in BBEdit and saved them as UTF-8, also changing the .aff file to read SET UTF-8 (which is the same as the SET statement in a Greek dictionary). Tried again.

b'ISO8859-1'
0 a
0 the
0 asdfasdf

Wait, what? The SET statement says, and the actual file encodings are, UTF-8, but get_dic_encoding returns ISO8859-1? Just in case there's some kind of file caching going on, I copy the dictionary files to a different folder and change the path string to match. No change! I re-save the files as UTF-8 "with BOM". No change; it still returns ISO8859-1.

Now I doubt my prior diagnosis. Hunspell is ignoring the content of the dictionary, which possibly means it isn't reading it at all. Maybe it is failing to open the files, not reporting a failure, and returning some kind of default?

Does the hunspell package do the same?

import hunspell
import os
dpath = '/Users/dcortes1/Desktop/scratch'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
Hobj = hunspell.HunSpell(daff, ddic)
Hobj.get_dic_encoding()
'ISO8859-1'
Hobj.spell('the')
False

OK, I am officially flabbergasted. My flabber is gast. That dictionary is UTF-8 and defines "the". It is nice, in a way, that the package (that works fine inside PPQT1 and 2) is failing exactly as my ctypes experiment failes, but what the heck am I doing wrong?

I'll post again when I understand more.

Update: this much is confirmed:

Hobj = hunspell.HunSpell('aasdf/asdf/asdf.aff','aasdf/asdf/asdf.dic')
Hobj.get_dic_encoding()
'ISO8859-1'

If the Hunspell object creation cannot open the .aff/.dic files, it fails silently and uses a null dictionary with a default encoding. And I don't see any way of testing whether this has happened or not!

Exploring ctypes

Implementing spellcheck has been a constant problem for me. I solved it, awkwardly, using the pyhunspell package (see below for another link). This provides a Python interface to the Hunspell checker.

There is nothing at all wrong with Hunspell itself. It is complete, fast, and still supported. It is superior to Aspell and Myspell in several ways. Most importantly it supports UTF, and so can be used to spellcheck German, Greek, and the like.

My issues are, or were, with the pyhunspell package. It went unsupported for a long time. It didn't support Python3 until a user posted the necessary small changes as a comment on an issue. And getting it compiled and working on Windows was, for me, a huge problem. So I wanted to experiment to see if I could access the hunspell library direct from Python using ctypes, eliminating the need to compile a wrapper. Important note: I just discovered that pyhunspell was very recently picked up by a new owner, Benoît Latinier, and rehosted on github: here is its new home. Another user has posted a binary package for Windows on the old site; unfortunately it's for Python 2.7. So things are looking up for pyhunspell. Which is a good thing, because as I will now finally get around to saying, the ctypes experiments are not going super-well.

We start with getting access to the library.

# Find the library -- I know it is in /usr/local/lib but let's use
# the platform-independent way.
import ctypes.util as CU
libpath = CU.find_library( 'hunspell-1.3.0' )
# Get an object that represents the library
import ctypes as C
hunlib = C.CDLL( libpath )

To do spell-checking, one must create a Hunspell object. The C header declares:

typedef struct Hunhandle Hunhandle;
LIBHUNSPELL_DLL_EXPORTED Hunhandle *Hunspell_create(const char * affpath, const char * dpath);

Converting that to ctypes, we have:

hunlib.Hunspell_create.argtypes = [C.c_wchar_p, C.c_wchar_p]
hunlib.Hunspell_create.restype = C.c_void_p

OK, let's call it!

dpath = '/blah/blah...'
daff = os.path.join(dpath, 'en_US.aff')
ddic = os.path.join(dpath, 'en_US.dic')
hun_handle = hunlib.Hunspell_create( daff, ddic )

Well, nothing crashed. At this point we should be able to use methods of the Hunspell object. Back to the C header file:

LIBHUNSPELL_DLL_EXPORTED char *Hunspell_get_dic_encoding(Hunhandle *pHunspell);

In Python/ctypes:

hunlib.Hunspell_get_dic_encoding.argtypes = [C.c_voidp]
hunlib.Hunspell_get_dic_encoding.restype = C.c_char_p
print(hunlib.Hunspell_get_dic_encoding( hun_handle ))

And whoop-de-doo, it prints b'ISO8859-1'. To review: we have successfully loaded the library, created a Hunspell object, and invoked one of its methods. During creation, the object correctly loaded the dictionary that was passed. Ergo, we can pass a Python3 string into a const char * argument. This is looking great!

And this post is getting long. I will continue with actual spell-checking next time. Spoiler alert! It doesn't go well!

Preferences done, still waiting

I completed the Preferences dialog. It looks about the same as the test version shown in the prior post, plus the addition of buttons for Defaults, Cancel, Apply and OK.

Getting things to work smoothly with the "Apply" button took a bit of work. There are four highlight types the user can change: the editor current line, the text of a limited Find/Replace range, spelling error words, and scanno words. The first two use one mechanism, the second two use a completely different one.

As noted in some post I can't be arsed to look up now, the current-line highlight is done using an "extra selection", and the find-range highlight is done using another. An extra selection is a peculiar thing unlike any other class in Qt (that I know of; maybe the graphics area has similar things). It has no behaviors, no methods; it is basically a two-ple of a cursor and a format (QTextCursor and QTextCharFormat). You give your QTextDocument a list of your extra selection objects and it applies each one's format on its cursor's selection.

The current line's cursor gets updated whenever the edit cursor moves. The find-range cursor gets updated only when the user toggles the In Selection switch in the Find panel.

Their formats get updated only when the Preferences dialog calls the colors module set_current_line_format or set_find_range_format. At that time, the colors module emits a signal, ColorsChanged, which is fielded by editview, and it refreshes the formats in its list of two extra selections.

Therein lies one problem. The user can choose a new format for current line or find range, and click the Apply button in Preferences, and the the highlighting in the visible part of the Edit panel should change immediately. But it didn't, apparently because signals don't get processed while a modal dialog is up. However, I added a call to QCoreApplication.processEvents() in the Apply logic, and then, ta-daa, those two highlights changed instantly upon Apply.

The spellcheck and scanno highlights are created by a different mechanism. They are applied by a QSyntaxHighlighter. Syntax highlighting is turned on by assigning a document to the highlighter, and turned off by assigning a null document to it. But the highlights it applies don't change once they are set until the highlighted text is hidden and shown again, e.g. by paging the document in the editor.

So those two highlights didn't change even though the editor was getting control in its ColorsChanged signal slot. I had to add logic to this slot to ask, is highlighting of either scanno or spelling now active? If so, turn highlighting off, and turn it on again. That forces re-tagging of visible words. With that change, the visible highlighting changes instantly when Apply is clicked.

I am a bit concerned about this last, because in Version 1, there was a significant delay when you turned on the syntax highlighter in a large document. I assumed this was because at that time it would go over the whole document passing every text block through the highlighter. If Qt5 behaves like Qt4, clicking the Apply button in Preferences might incur a significant (1-3 second) stall when the spelling or scanno highlight has changed. That delay isn't noticeable now. There's no perceptible delay with a 25-page document. I hope that Qt5 is smarter and only invokes the highlighter for visible text. Even if there is a delay, these highlights are not something you'd change often.

For now it all works nicely. Doing Preferences was supposed to fill the time until PyQt5.4 was out. I'm done, and it isn't. Now what?

Fiddling with Fonts

This week, waiting for PyQt5.4 to be released, I've been working on the Preferences dialog, and it's coming out rather interesting. Alongside here is a test version.

It's a stack of items. I built a whole little O-O heirarchy to make this. There's a parent class, ChoiceWidget, that displays a title line and on mouse-over, changes color and puts an explanation in the explainer box at the bottom. For the path-entry items, there's a PathChooser class that implements the path line-edit and a browse button, and when focus leaves the line-edit, checks that the given path is accessible according to some criterion (R_OK or X_OK) passed to its initializer, and if not, beeps and makes the line-edit pink. For the text-format items, there's a FormatChooser class that implements the color swatch and the sample highlight.

And then there's the font-chooser. Here I wanted to use QFontComboBox, which displays the available fonts using those fonts. Unfortunately it is rigidly designed. When told to display only monospaced fonts, it displays only the fonts the QFontDatabase thinks has that property.

Unfortunately, I am including two monospaced fonts with the program, Liberation Mono and another that I only just discovered, Cousine from the Chrome Core set. Cousine is basically the same as Liberation Mono except it has even more Unicode glyphs.

Either of these is a better choice than the next-best font, Courier New. The latter has about as wide a Unicode repertoire but it has a poor contrast between 1/l and 0/O. Alas, the QFontDatabase will not recognize either of the fonts that I am loading (using QFontDatabase.addApplicationFont()) as being actually monospaced. It will not return their names when asked for a list of monospaced fonts.

Which means that if QFontComboBox is set to display only monospaced fonts, it will not display the two that I most want the user to have access to. And although it claims to support the methods of its parent QComboBox, it ignores a call to addItem(string). So I can't add them. If I don't tell it to show only monospaced fonts, it of course shows every font available in the system, and takes quite a while to open first time.

I wasted quite a bit of time today trying to remedy this, first by trying to find some way to get QFontComboBox to show all but only the fonts I wanted it to show (the known monospaced fonts plus my two); then by trying to find a way to change my fonts so that QFontDatabase would recognize them as monospaced. That entailed a very lengthy search for a free or cheap TrueType font editor that would let me verify and set what I presume is a one-bit flag in the font file format that says "really, I'm monospaced." Didn't find one. Well, one, but it costs a bundle and its 30-day free trial version will not allow saving a modified font. So fuck them.

Actually, after all that, I'm not sure whether any amount of editing would help. The Mac OS Font-Book application does know these two are monospaced. It lists them when I make a "smart search" for monospaced fonts. So it may be that QFontDatabase is just prejudiced against added application fonts.

In the end I used a plain QComboBox loaded with the family names of monospaced fonts. I set it up so that when you make a selection from the list, the "explanation" box at the bottom changes to use that font.

I meant to finish the Preference dialog today, I really did. But the fiddling with fonts killed too much time. What's left is to implement the dialog widget itself, including the important "Set Defaults", "Apply", "Cancel" and "OK" buttons. A few hours more.

Power of missing parentheses

I've completed the Footnotes panel. Its testing was rather minimal, although I believe sufficient. The footnote "model" did receive a rather thorough fnotdata_test.py module that exercises every branch and error condition. With a solid model, the view/controller piece goes together very quickly. I set up a fnotview_runner.py that starts up the app, loads a document full of footnotes of various types, and sits waiting for interaction. I used this to exercise all the functions manually, and quickly ran into a vexing problem that took a couple of hours to track down. When I knew the cause, I was even more vexed. Look at the following code fragment. Hands up all those who see the glaring, obvious, stupid error?

        self.model.beginResetModel()
        worktc = self.edit_view.get_cursor()
        worktc.beginEditBlock()
        try:
            for j in range(self.data.count()):
                # ...twenty lines of code computing a new key value
                # for footnote j...
                if new_key is not None :
                    self.data.set_key(j, new_key, worktc)
            # end of for j in range of keys
        except Exception as whatever:
            fnotview_logger.error(
                'Unexpected error renumbering footnotes: {}'.format(whatever.args)
                )
        worktc.endEditBlock
        self.model.endResetModel()

Here's what's going on. The user has clicked Renumber. The view/controller tells its QTableModel that data is changing. It obtains a QTextCursor on the document and starts an "edit macro" on it, so that all changes made using it will be a single Undo. For each existing footnote key it computes a new renumbered value, and calls the data model to set that as the new Key value in both the Anchor and Note of footnote j.

The model gets passed the working text cursor and uses it when it updates the Key value in the Anchor ([key]) and the Note ([Footnote key:...). At the end of the loop, even if there were errors, the edit macro is closed and the table model is told it can refresh itself.

When executed, this caused all sorts of flaky behavior in the editor. The new Key values would not appear unless I made the page scroll. Undo did not always undo the changes. Doing new changes compounded the problems.

I spent quite a bit of time over two days inserting print statements and tracing and... before I finally noticed that glaring error that you, dear reader, spotted five minutes ago. Fix that and suddenly everything worked "just swellegant" as my late father liked to say.

Well, live and learn.

Or not.

Looking ahead, again

Anyway, that's done, and also I fixed a serious, if obscure, bug in Version 1 and released new packages for it. A few posts back I put up the current to-do list. A month has passed; time to revisit it.

• When Qt5.4 and the matching PyQt are available, install those on the new iMac and move development to there.
• Qt5.4 is out, PyQt5.4 is expected any minute. So this should happen next week I hope.
• Then, bring CoBro up to the Qt5.4 level and replace the execrable WebKit browser with the new WebEngine one.
• Then, use Cobro as a test-bed for learning how to use pyqtdeploy to bundle an app. I am eager to find out if this is truly a way to make a self-contained executable on all 3 platforms, in place of pyinstaller.
• Presuming that works (and that the new web engine fixes the frequent crashes induced by webkit), release CoBro on all three platforms.
• Then, or right now to pass the time waiting for Qt5.4, code the footnotes module. That will go fast; most of the code can be lifted out of version 1...
• It took a bit more work than that, but it's done.
• Then, or right now if PyQt5.4 is delayed, implement the Preferences dialog.

At that point — which will in no way be reached in calendar 2014 — PPQT2 will be at what might be called an alpha state, that is, with adequate function that an experienced user could post-process a book with it. That user would have to run from source, however, until the pyqtdeploy work is complete.

The work to be done after that includes:

• Writing the translation interface module, which includes figuring out how to dynamically load translator modules.
• Writing the plain-ascii example translator
• Writing the HTML example translator
• Bringing the "keyboard palettes" of V1 forward to V2 and making them load dynamically (using the same scheme as the translators?)
• Finally going back into the UI and make panels drag-out-able, applying the drag-drop research with which I began this series of posts many months ago.
• Writing the Help file and adding the Help panel
• Rewriting the "suggested workflow" document to reflect all the changes; for this I will want to actually post-process a book myself to make sure I know the best way to use the app.
• Make some screencasts to explain PPQT and show its features.