Monday, February 23, 2015

PyInstaller and QtWebEngine

While PyInstaller can bundle PPQT2, it cannot bundle Cobro. The difference is QtWebEngine. I've noted before that Cobro, which is basically a browser, was horribly unreliable when it used QtWebKit. I was eager to get Qt5.4 because it came with QtWebEngine, based on the Chrome browser. And executing Cobro from source as a Python app, the difference is striking. I've been running Cobro from the command line daily for a month and it has yet to crash in any way.

Web Engine did come with some problems, as I've noted before. I basically had to give up implementing a custom context menu, and keystrokes for "back" and "forward". And the binaries are huge, totalling close to 100MB when all related modules are counted. But it works! So going back to QWebKit is not an option.

Unfortunately the Web Engine people just can't fit in to the standard Qt ecosystem. In a Mac OS app, the QtWebEngineCore module cannot be dropped into the MacOS folder along with all the QtThisAndThat modules. It has to be placed as a "framework" in the Frameworks folder. If it isn't there, the app won't start; it terminates with the message "Couldn't mmap icudtl.dat". This simple message is the tip of a large, deep iceberg. I've put the ugly details in a PyInstaller issue and won't repeat it all here.

Bottom line, it looks to me as if PyInstaller will never be able to bundle an app using QtWebEngine. This kind of complexity can't be shoe-horned into a "hook", I think.

Which means, Cobro can't be bundled by PyInstaller. I am confident that cx_freeze will have the identical problem. Maybe... just maybe... perhaps when [Py]Qt5.5 comes out, I will pull out and reinstall everything and see if maybe I can get pyqtdeploy to work on it then. Until then, Cobro is just a personal project. I can't hope to distribute it more widely.

Giving up on pyqtdeploy; breakthrough with PyInstaller

My very helpful correspondent Lloyd built Cobro on his system with pyqtdeploy and it ran. I have what appears to be the same setup, but when I build it, it doesn't. After days of struggle, I wrote pyqtdeploy off and turned back to PyInstaller.

PyInstaller is only a beta under Python3. More than a year ago, I contributed quite a bit of time and code to making it work, not only with Python3 but using ModuleGraph as its analysis tool in place of its previous method. My code was integrated but the Python3 branch got only some attention from the primary developers. Also relevant here: PyInstaller uses a system of "hooks" for module-specific work. There are a number of PyQt-related hooks. A hook receives as its argument a "mod" object, which was formerly a Module instance as created by the old import analyzer. The hook could add or delete binary dependencies from the mod and do other things to it.

When I changed the code to use ModuleGraph, that interface had to change. There was no more Module object. But rather than change all the hooks, I crafted a FakeModule class. Before calling a hook, PyInstaller makes a FakeModule and passes it to the hook. The fake mod argument can be interrogated the way the old mod could be.

Unfortunately the most common thing a hook does to its mod is to add or delete some dependencies by use of an expression like

    mod.binaries.append( list-of-descriptive-tuples )

Unfortunately the binaries member of a fake module had to remain read-only. The hook could examine it to find out if some binary was already included. But to add or delete binaries, the hook now has to have

    mod.add_binaries( list-of-descriptive-tuples )

When my changes were merged into the Python3 branch, all of the then-existing hooks were updated to use this API.

When a few weeks ago, I tried PyInstaller for Cobro, the resulting Mac OS app would immediately crash for lack of a platform plug-in. Eventually some hours of effort revealed that the PyQt5 hooks were putting the platform plugins in a nonstandard place, and playing a naughty game with the QApplication to make that happen. But at least for other people, the platform plugins did get put in that place. For me, they weren't.

So now I went back to PyInstaller and examined more carefully what was going on. Gradually the work I'd done last year came back to me and in tracing through the code, I discovered that the set of PyQt5 hooks were not using my Python3, Modulegraph, API. They were probably created when Qt5 first came out, which was after I'd stopped work on it. They were probably just copied from the current version of the PyQt4 hooks, and merged into the Python3 branch unchanged.

So all the PyQt5 hooks were trying to do mod.binaries.append(), which had absolutely no effect on the final bundle. All I had to do was update them to use the correct API and bingo, it copied the plugin files to where they should be. So I cloned the Python3 branch and made those changes and issued a pull request.

Cobro still didn't work as bundled by PyInstaller, for a reason I'll cover in the next post. But here's the big news: PPQT2 does work! PyInstaller, under Python3, with the changes I just made, correctly bundles PPQT2 as a Mac OS app bundle! This suggests that in the not too-distant future, after I've installed all the needful bits into my Ubuntu and Win7 VMs, I'll be able to roll out a PPQT2 alpha.

That will have to wait at least 2 more weeks. Thursday we head off for a ten-day run up to Portland and Seattle. But not long after I get back, hopefully before the end of March, I'll be able to push out that alpha. In my spare time on this trip I plan to start work on the Translator interface, which is the one remaining feature of PPQT2 that is not coded.

Tuesday, February 17, 2015

Yet another deployment hitch, cleaning up PPQT2

So at Lloyd Konneker's suggestion, I put a symlink to the Qt distribution in the SYSROOT that is at the heart of the pyqtdeploy process. This makes perfect sense; it is essential that the QMake/Make steps have access to the target version of Qt. I should have realized the need for it before. I'm going to blame my mistake at least in part on the confusing and arcane documentation. Anyway.

With that done, the QMake;make steps run aaaaaalmost to completion. But not quite. The very last, and crucial, step is a huge link that should end up putting all the executable resources into the .app folder. It gets partway; it makes Cobro.app/Contents/MacOS/Cobro, an 18MB executable. But none of the other things, the PyQt and Qt frameworks, make it.

The reason seems to be this "warning" message (not really a warning, it clearly causes the process to abort):

ld: warning: could not create compact unwind for _ffi_call_unix64: does not use RBP or RSP based frame

Searching on that reveals a fair number of people were having problems with "compact unwind" a couple of years ago. Around XCode 4 time one could add an option -no-compact-unwind to get around it. I tried adding that to the LFLAGS in the Makefile but it was rejected as unknown.

Turning from banging my head on pyqtdeploy, I fixed the issue of invalid regex replace strings. I just put try/except around the actual replace operation, and put up a warning dialog for the user to enjoy. Then I had to put another except clause, because there are two classes of exceptions that can come from a regex replace. If there is a syntax error, for example using \g< (an incomplete group name reference) it raises a regex.core exception. But using \2 (when there is no group 2) causes an IndexError. So the try statement has to have two, near-identical except clause suites.

After that I spent some time browsing techs on elance and odesk, looking for people with strong PyQt chops who I might hire to set up deployments for me. I'm looking ahead: supposing I can struggle through and make it work on OSX, then I still have to go through making pyqtdeploy work on Ubuntu and then, even more daunting, Win7. Based on my experience so far, I could spend another 3 or 4 months exasperating myself on that.

So I am thinking I can hire somebody to build me three complete, provisioned, virtual machines, one for each OS, that I can run when I need to deploy a new version. I'd happily spend a few hundred to avoid that pain.

Oh, yesterday I said "I've never butted my head so hard against a problem, for so long, with so little to show for it." Today I remembered there was one other time...

Back in 1986 I wrote a book about OS/2. Heh heh... Amazon never forgets! Anyone in dire need of a copy? I still have a few.

So this was before the thing was released. I had a developer's license and got very occasional pre-releases from Microsoft, but the documentation that came with it was just abysmally bad: badly written, incomplete, and inaccurate. I was trying to write a programming book, a careful guide to the APIs. Thanks to the shitty work Microsoft did, I had to basically reverse-engineer every API call. It was horrible. I hated Microsoft so much for the way they treated me (and all the other developers). After I finally delivered the book, I remember telling my patient spouse that I had been continuously angry for months.

I guess Riverbank and pyqtdeploy are pretty decent, compared to Microsoft and OS/2.

Monday, February 16, 2015

Deploy depression, PPQT work

Yesterday I described how I'd found that QWebEngineWidgets was not mentioned in the config file left by pyqtdeploycli. I put it in manually and to my delight, the make of Cobro now completed. It left an incomplete app structure, which I may have a handle on fixing (tomorrow). But, assuming that the absence of QW.E.W. was a mistake in pyqtdeploycli, I posted the info to the PyQt mailing list.

Here's the reply from Phil:

The configuration files generated by pyqtdeploycli should be considered starting points which you should review and possibly update to meet your specific needs. The reason why those modules are not enabled by default is that they are not available on all platforms supported by Python.

This made me quite angry. First, the documentation says this:

The package configuration files created by the configure action of pyqtdeploycli assume a default Qt installation, i.e. with features that would only be enabled by default on the particular platform. For example, SSL support is disabled for Windows and Android targets. If you have configured your Qt installation differently then you may need to modify the configuration files appropriately.

Is there anything there to suggest that a normal piece of Qt for all desktop platforms would be omitted? And as for "not available", probably QWebEngine is not supported on some mobile platforms, but pyqtdeploycli is told on the command line the target is "osx_64" and should not have any question about whether the feature is supported there.

Well, what's the point of bitching? I got over that hump.

And then I noticed this gem:

pyqtdeploy itself uses rcc to embed all the files that make up the applications and does not support the use of the output of pyrcc5 and pyrcc4 in a deployed application.

So? So, I am using pyrcc5 to embed two monospaced fonts into PPQT. The above means that, if I use pyqtdeploy on PPQT, I have to change the logic of how those fonts are delivered. Yes, the change will affect only one module (fonts.py) and some documentation (I'll have to explain why there are a couple of fonts in the Extras folder, which is I guess where I'll have to put them).

All this left me quite depressed about this whole thing. It's been almost two months I've been fucking around with bundlers, and every fucking one of them has failed. I am not stupid or inexperienced but I have to say I've never butted my head so hard against a problem, for so long, with so little to show for it. I keep thinking I'm close to a breakthrough with pyqtdeploy, but every time I get over one hurdle there's another ahead. It is just an exceedingly complex, poorly-documented affair based on many hidden assumptions that I don't know about. And just a bugger to work with.

For a while I thought seriously about tossing the whole project, both projects, just saying, fuck it, here's the github source, enjoy yourself. And I thought seriously about going on elance and finding a contractor and solving the problem by paying somebody to solve it. I did that when I just could not get hunspell to work on Windows; it was $150 well spent.

Well, so. In the course of writing my PPQT2 help file (which is done) I found several things that needed fixing. And this afternoon I fixed all but one of them. That remaining one is a bit of a poser.

In V1 and also V2, the Find panel can be used for regex search and replace. When the Regex switch is checked, every time the user edits the Find string I quickly do a regex.compile() of it, inside a try/except block. The only exception would be a regex error; and if one is triggered, I turn the background of the string pink and put the regex diagnostic message in the toolTip string for the field.

While testing things I was writing into the help, I happened to trigger a different error. I put something invalid in a Replace field, and did a Replace. When the Regex switch is on, a Replace means doing match.expand() on the match object resulting from the most recent Find. Turns out, if the replace string is invalid, that can throw an exception. Down in the guts of the do_replace where no exceptions were anticipated.

So clearly, I thought, I need to also syntax-check the Replace string as the user edits it, similar to how I'm checking the Find expression. Well, not so simple. First, the main error that a replace can have is a reference to a non-existent match group: \2 when the regex didn't have two paren groups, say. But that means the correctness of a replace expression depends on the content of the match expression. Which means (a) the replace expression can't be valid if the find expression is invalid, (b) if the replace expression is valid now, but the user changes the find expression in a valid way, the replace expression can become invalid.

And then, my scheme for checking replace validity fell apart. I'd supposed I could test it in this way. Suppose that the current, valid Find expression is compiled as find_rx. I supposed I could do something like,

try:
    find_rx.sub(rep_text, '' )
except:
    handle the error

Nope. Because the find_rx matches itself against the target string (the null string), says, "I don't match, thanks, finished, bye." And never gets around to compiling the rep_text. The only way to get it to actually look at the rep_text is to have a valid match. But no match is available at the time the user is still editing the replace string.

All this says that there is no way to pre-validate the replace expression. I will have to put the actual replace operation in a try block, and when it throws an error, then I can turn the replace string pink and put an error text in its toolTip. But then, when will I know to clear the error indication? Whenever the user edits that field or the Find expression, I suppose.

Sunday, February 15, 2015

Major step in pyqt deployment

Yesterday I brought the pyqtdeploy build process to a new make failure: a module named _sqlite didn't compile because a variable MODULE_NAME was not defined. I thought this looked like a generic problem. I could see lots of uses of that name in various modules and no definitions of it anywhere. But I reported it to the PyQt list and quickly a fix specific to that one module was posted.

Meanwhile I figured out that there was no reason to include that module in Cobro anyway. I'd turned it on in the list of modules because cx_freeze had included it. Well, maybe that means it is imported at some level by something, but for now I just turned off the switch in the pyqtdeploy list.

Then the make failed in a new way: by failing to find a library for QtWebEngineWidgets. That is a PyQt module that Cobro definitely needs. So tonight (Sunday night, 8pm) while Marian is working on the Stanford WBB web site ("Bonnie Bombs the Bruins" is the headline from this afternoon's game), I started to look at that.

There exists a QtWebEngineWidgets module in the PyQt5 distribution folder. But the build process for it does not end up copying it to the SYSROOT where all the build stuff is supposed to end up.

It's a convoluted sequence. First pyqtdeploycli runs and leaves behind a .cfg file. Then the configure.py file runs and, based on the .cfg file, creates Makefiles for all the components to be built. Then you run Qmake which runs instantly. Then you run make, and it does about 10 minutes worth of compiles. Then you run make install, and all the compiled modules get copied to SYSROOT.

After an hour of futzing around I figured out that it was simple: the word QtWebEngineWidgets had been omitted from a list of modules in the .cfg file. Presumably due to a bug in pyqtdeploycli. I added that word manually and reran the remaining steps, and hurrah, the library appeared where it is supposed to be.

Then I held my breath and reran the pyqtdeploy build/qmake/make sequence for Cobro. And it ran! It produced in the build folder a Cobro.app.

Which doesn't run.

Because it is clearly unfinished. There is an 18MB executable, Contents/MacOS/Cobro, but on launch it immediately dies because it can't find a library for QtCore. Of course. Some further step is needed to gather up all the Qt Framework modules and add them to the .app structure.

I have no idea what that step is. There is a build/Cobro.pro, a QMake file, and also a build/Makefile. But qmake doesn't do anything and make just says "nothing to be done for `first'."

Still, it's big progress. An actual app may not be too far off.

Friday, February 13, 2015

Annnnd ... no.

I have set up my pyqtdeploy directory structure and in it have built the static versions of Python, Sip and PyQt. I have used pyqtdeploy to establish where all this stuff is (the Locations tab). When I tell it to build, it freezes all the python modules and runs QMake fine.

But the Make step bombs with something incomprensible. Specifically it stops because

make: *** No rule to make target `../Downloads/Python-3.4.2/Modules/Modules/audioop.c', needed by `audioop.o'.  Stop.

It happens that audioop is the first of the C modules in the make sources list. When make reaches it, it has already successfully compiled three .cpp modules.

What's incomprehensible is that to my ignorant eye, there clearly IS a rule, in fact two. Partway down the Makefile there is this:

audioop.o: ../Downloads/Python-3.4.2/Modules/Modules/audioop.c 
 $(CC) -c $(CFLAGS) $(INCPATH) -o audioop.o ../Downloads/Python-3.4.2/Modules/Modules/audioop.c

Which looks very much like a "rule" to me. And at the top there is,

####### Implicit rules
...
.c.o:
 $(CC) -c $(CFLAGS) $(INCPATH) -o "$@" "$<"

Which also looks like a rule. So what does it want? I wrote to my helpful mentor Lloyd. Hope he has a clue to give me.

Meanwhile, back to PPQT help and stuff.

Thursday, February 12, 2015

Scattered progress

Wednesday was my day to catalog items at the Computer History Museum off-site storage warehouse in Milpitas. This time we finally began work on the "Moffatt stuff", material that came from the Boston Computer Museum in the 1980s and sat in a dusty warehouse at Moffatt field in Mountain View until in 2008 it was moved to the then-new, climate-controlled warehouse in Milpitas. Only now is the museum finally getting around to a proper job of cataloging it.

We were working on pieces of Whirlwind. When that pioneering one-off machine was decommissioned, it was taken apart and hundreds of pieces were donated to the BCM. Here is a typical piece like what we were cataloging and photographing:

I have to say that this picture was taken in the prior, 2008-era cataloging effort. We are doing a better job now. The object would be squared-up to the camera, the object id tag would not be visible (lower left), and the extraneous junk like the little yellow pill boxes of numbers would not be shown. And the light would be better.

Just the same, that is a typical hunk of Whirlwind. Panels like this, with a couple rows of octal tubes, were the main logic elements. This might be a pair of flip-flops or a pulse generator.

So no work on my own projects that day, and Thursday there were other errands, and three hours watching the AT&T guy Rick install our new U-verse internet. From DSL that could barely make 4Mbs to a service that measures out to exactly the advertised 12Mbs.

But in my spare moments both days I was working on the PPQT help file, and documenting all the things it does caused me to discover some things it doesn't do right. One in particular. The Find panel, when the Regex switch is on, expects the string in the Find text field to be a regex. When it changes, keystroke by keystroke, it is checked for validity, and if it isn't a valid regex, the background of the field turns pink, and the search buttons First/Next/Prior/Last are disabled.

But it was not checking the syntax of the Replace fields. I was trying out some different PCRE-type replace strings and discovered that it is possible to have a syntax error in the replace string, and when you do, the regex module throws an exception. Naturally the Replace action wasn't in a try/except block.

What this means is I need to guard the Replace fields (there are three of them) with the same error checking logic and pink-background decoration as the Find field. Fortunately most of the code for this exists.

But the bigger news would be some success with pyqtdeploy. I'd been stymied by a crazy error from qmake, as described in the last post. Wellllll.... come to find out, the cause of the problem was the the qmake I was invoking was from /usr/local/bin, which was (apparently) the qmake installed by Qt4.6. When I invoked the current qmake from 5.4, the creation of static versions of Python, Sip and PyQt went smoothly. I have high hopes that tomorrow I might get a complete build of Cobro.

Monday, February 9, 2015

Another pyqtdeploy hitch, edit feature, help text

Today I began work on setting up pyqtdeploy from scratch with static Python, PyQt and Sip, as it should be. I'm following in part the work of Lloyd Konneker in this blog entry, and in part these pyqtdeploy docs.

Things started well; following Lloyd's suggestion I created a new folder /Developer/Static and exported SYSROOT as that path. In it I build a Downloads folder containing three folders, one containing the Python 3.4.2 source, one with the PyQt5.4.1 source, and one with the latest Sip source.

Moved into the Python folder and ran pyqtdeploycli --package python configure. Looked good. Now to run QMake. And immediately it failed with an absolutely ridiculous error. Slightly edited the sequence goes:

$ echo $SYSROOT
/Developer/Static
$ qmake SYSROOT=$SYSROOT
ERROR: SYSROOT must be defined on the qmake command line

Can you believe it? Can an error be less credible?

Anyway I wrote to Lloyd who has been very helpful and encouraging. We'll see.

Pending a reply from him, I set to work to fix a functional hole. Writing the help text yesterday I was copying stuff from the V1 help file and noticed, oops, in V1 I documented that with the focus in the editor, you could key ^u to uppercase the selection, ^l to lowercase, and ^i (initial) to title-case it. (^t is preempted to mean, replace-and-find-again.)

I had certainly meant to implement that in V2, but somehow had forgotten. So now I set to work to do it.

The V1 code was rather bulky. It got the current selection as a QString; it looped over the string using a QRegExp to isolate the next word; and it used the QString methods to change the case and replace the word with the modified text. Per the comments I was particularly staying in the QString domain because I was afraid Python would not handle Unicode casing properly.

Now in PyQt5 there are no QStrings. If I get the selected text it is as a Python string. The Python 3 string methods .upper(), .lower() and .title() are Unicode-aware, so that's no longer an issue. So I looked at how to do this in a better way. The doc for the string .title() method tipped me off to the feature of the re.sub() method that lets you pass a function that will be applied to every matching substring. It's kind of magic. Based on that, here is the main part of the code. The editor has caught one of the three keystrokes, or the user has selected one of the three corresponding Edit menu actions.

    def case_mod(self, kkey) :
        tc = self.textCursor()
        if not tc.hasSelection() :
            return # no selection, nothing to do
        text = tc.selectedText() # full selection as string
        if kkey == C.CTL_SHFT_L :
            func = lambda m : m.group(0).lower()
        elif kkey == C.CTL_SHFT_U :
            func = lambda m : m.group(0).upper()
        else:
            func = lambda m : m.group(0)[0].upper() + m.group(0)[1:].lower()
        new_text = re_word.sub( func, text ) # do it!
        tc.insertText( new_text )

That's it! There are a few more lines devoted to re-establishing the selection after the text insertion. The whole thing is about 1/8th the LOC of V1.

The magic is in re_word.sub( func, text ). re_word is a global compiled regular expression that matches to whole words, including words that contain DP special character notes like [:u] for ΓΌ, or hyphens or apostrophes or curly apostrophes. The syntax highlighter used it in the process of marking spelling errors or scannos, but it repurposes here nicely.

The .sub() method, if given a function, applies that function to every match in the text. The function receives a regex match object, and is expected to return a string to replace the matched substring. So to change the case of the matched text, just return that text, which is match group 0, in lowercase or uppercase. Or for title case, return the initial uppercase and the rest lowercase. It's just deliciously slick.

After that enjoyable session I spent some time cleaning up the extras folder, making sure all the things that are supposed to be in it are, and putting everything under version control so they'll get onto github.

That leads to a complication with the Sphinx-based help page. This page relies upon just six helper files, three .css ones and three .js ones, in a folder named _static adjacent to the .html file. So I created this structure:

extras
    sphinx
        index.html <-- the actual help file as compiled by sphinx
        _static
            blah.css
            blah.js
            etc helper files
    ppqt2help.html as a symbolic link to sphinx/index.html

If I open extras/sphinx/index.html in a browser, it looks great.

Ditto, if I open extras/ppqt2help.html in a browser, it looks great. The browser, well Firefox at least, resolves the symlink and finds src=_static.blah.css.

Not so the PPQT code. It opens extras/ppqt2help.html, reads its contents, and stuffs it into a QWebView for display. And the QWebView can't find the helper files.

And as I write this, I see the problem. I think there is some way to tell QWebView the base path for its content. I shall investigate that tomorrow.

Sunday, February 8, 2015

Working on help text

With a help viewer in place, it's time to add some help text in the form of an HTML page. Version 1, the help page was all hand-made HTML. Readable and well-organized (because I be an ace tech writer, ya know) but frankly, ugly to look at.

So for V2 I thought I'd use some doc-making package to build a nice themed page. Friday afternoon I took quick looks at several and somewhat reluctantly settled on Sphinx.

The reluctance to use what appears to be a mature, well-supported system comes, mostly, from the fact that Sphinx requires the use of reStructuredText as its input markup. I dislike several things about rST. Its syntax is arbitrary to the point of parody. I got familiar with rST while I was rewriting the doc for PyInstaller. Familiarity did not exactly breed contempt, but it didn't make me like rST either. It's just another text markup language, more complicated but maybe more complete than Markdown. Anyway.

So I started writing the Help doc for version 2. Copying lots of text from the V1 doc of course, but also rewriting for greater simplicity and clarity. I'm using lots of simple declarative sentences. One goal for V2 is to be usable in other languages than English. All the user-visible character strings in the code--buttons, titles, tooltips, errors, etc.--are enclosed in calls to tr(), the Qt translator. If there is a translation file for the current locale, they'll display in the local language.

There's no automatic translation system for a big fat HTML page (or a big fat .rst source file). But at least it will be easy to translate, if anyone ever wants to. And if not translated, easy to read for someone with English as a second or third language.

The HTML generated by Sphinx poses another problem. The V1 help was of course a single self-contained .html file, and that's what I expected to have for V2. But Sphinx does not make self-contained HTML. The HTML page it makes has half a dozen references to files in a folder named _static assumed to be next to the page.

The Sphinx build system has an option singlehtml, but that just guarantees there will be only one HTML file in the output. Normally it expects you to have a set of multiple .rst files, say one per chapter, and it makes an index.html file plus other linked files. The singlehtml build option puts all the HTML files in one, but that one still has references to CSS and javascripts in _static. (And, it doesn't have a table of contents.)

So where in V1 I had a single file, V2 will have a file plus a folder of support files. This annoys me because it's more stuff to be checked-in and put under git control, and more stuff to be distributed in the "extras" folder.

I'm not clear yet on how to organize this. The viewer code looks in the "extras" path for a file ppqt2help.html. Probably there will be a folder named "html" containing index.html and the _static folder it depends on. This is the structure Sphinx builds. Beside it will be a link, ppqt2help.html, which points to html/index.html. So the viewer can open that name and get the right page.

I have been enjoying writing bits of the help text at odd moments of the day, pretty much whenever I have a free ten minutes. Coding is very satisfying, but writing clear, terse prose is even nicer.

Monday, I hope to get back to pyqtdeploy and make some real progress with it.

Friday, February 6, 2015

help viewer, QWebEngine lacks, fuzzy matching

After yesterday's entry I had some more time and put it into starting the coding of helpview.py. This is a simple parentless widget (which makes it a modeless window) containing only one object, a web page. The main window creates one instance of it the first time the user selects File>Help.

The help widget initializes by getting the path to the "extras" folder and looks there for ppqt2help.html, and loads that file's contents into its web page. (If it doesn't find it, it loads some default text that asks the user to use Preferences to set the proper Extras folder path, and hooks the signal from the paths module that says path preferences have changed, so it can retry loading help whenever the user does so.)

The only other features of this widget are how it handles the close event, and a simple Find function. The closeEvent() method ignores the close, so the widget is never destroyed once created (until the app ends of course). It just hides itself. In the main window, when the user chooses File>Help a second time, it just calls the show() and raise() methods of the existing help widget, and it pops up nicely just where the user left it.

When I went to code the Find/Next feature I ran into yet another WebKit feature that is missing in QWebEngine. A QWebEnginePage object has a findText() method, just like QWebPage does. And the argument lists are the same: a string to find, and a selection from the FindFlags enum. However, that enum for QWebEnginePage has only two entries, FindBackward and FindCaseSensitively. Missing are 5 other find options that WebKit supports, including most importantly, FindWrapsAroundDocument.

Without the automatic wrap-around, the find is crippled; it can only find what follows the current position in the page. Or what precedes it. But repeated find-next without wraparound is unintuitive and awkward, to the user hunting around in a lengthy user manual.

Well, you know: my user manual HTML is very standard, very normal. No graphics, no HTML5, no plug-ins. So there's no benefit to using the WebEngine. So I just used the QWebView. Then I implemented a keyPressEvent handler that picked off ^f, ^g, and ^G and directed them to do_find, do_find_next, and do_find_prior. Those methods were only a few lines long. Find pops up a dialog to get some text to find, pre-loaded with any text that might be selected in the webview already. Next and Prior use the last-given find string to search forward or backward.

I got about 2/3 of that written yesterday evening and was able to finish it this morning, including modifying the main window code.

I also settled a niggling problem in wordview. A useful feature of the Word panel (lifted from good old Guiguts) is the option to see the "first harmonic" or "second harmonic" set for a given word. You right-click on a word and choose "first harmonic" from the context menu. The table then reduces to show only that word plus any words that are exactly one edit (insert, delete, or substitution) away from it. Second harmonic shows words that are one or two edits away. It's a very useful way to find important typos, like misspelled "Footnote" or "Illustration" keywords.

In version 1 I used a module that implemented the Levenstein algorithm. But for V2 I knew I wanted to use the regex package, in part because it offers "fuzzy matching" which is basically, Levenshtein implemented in C, generalized, and incorporated as a regex option.

In order to find the first harmonic to a word whose text is stored in word, I do this:

        rex = regex.compile('^(' + word + '){0<e<2}$')
        hits = set()
        for j in range(self.words.word_count()) :
            wx = self.words.word_at(j)
            if rex.match(wx) :
                hits.add(wx)

Say the word is "page". The compiled expression is ^(page){0<e<2)$ which says, match a complete string (the caret and dollar at the ends) that matches "page" with exactly one error ("e" is greater than 0 and less than 2). (For some reason you are not allowed to write {e=1}.) Run through all words in the database and put the matching ones in a set: "rage", "pale", "pages" etc. At the end, if the set isn't empty, put it into the sortFilterProxy where filterAcceptsRow() will only accept the rows containing those words.

It works and it's very quick, but it took a little while to get the regex tweaked to get the correct matches. At first I didn't have the caret/dollar delimiters. Then it would match any first harmonic that was contained within a word. For example, it accepted "pamela" because it matched "page" to "pame" with one error.

With all that done I did a "push origin master". The PPQT2 code is just about ready for an alpha test. I need to write a skeleton version of the help file. And I need to solve the packaging or bundling issue. I've made some progress on the pyqtdeploy front that I'll write about tomorrow, I hope.

Thursday, February 5, 2015

Slight progress

RL has been intruding on nerdly fun a lot. But today I had a little time and think I have pyqtdeploy pretty close to success. Maybe.

In the last post I whinged (Great old British word; "whinging" is like whining but more pathetic.) that "now I get to figure out how to clone a mercurial repository, sigh," and that was stupid. I just had to go to the repository and actually look at it, and there was a "zip" link and down came the distribution with its setup.py and all. Easy. So I have the latest pyqtdeploy and apply it to what I think is a correct Cobro.pdy file and tell it to "Build" and "QMake" and "Make".

The first bit, where it "freezes" each Python module needed by the app, goes quickly. The QMake step seems to work. Then it starts on the Make. The first four commands, compiling four pyqtdeploy modules (pyqtdeploy_main, pyqtdeploy_start, pdytools_module, qrc_pyqtdeploy) run fine.

Next up is the command that I think will put it all together and assemble the MacOS app bundle. It fails. In trying to see why, I copied it into BBEdit and changed all space-dash into newline-tab-dash. Here it is.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
 -headerpad_max_install_names
 -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk
 -stdlib=libc++
 -mmacosx-version-min=10.7
 -Wl,-rpath,/Developer/5.4/clang_64/lib
 -o Cobro.app/Contents/MacOS/Cobro pyqtdeploy_main.o pyqtdeploy_start.o pdytools_module.o qrc_pyqtdeploy.o  
 -F/Developer/5.4/clang_64/lib
 -lsqlite3
 -lbz2
 -L/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/PyQt5
 -lQt
 -lQtCore
 -lQtGui
 -lQtNetwork
 -lQtWidgets
 -lQtPrintSupport
 -lQtWebEngineWidgets
 -lQtSvg
 -lQtTest
 -lreadline
 -llibz
 -llzma
 -lssl
 -ltermcap
 -L/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages
 -lsip
 -L/Library/Frameworks/Python.framework/Versions/3.4
 -lPython
 -lcrypto
 -framework SystemConfiguration
 -framework CoreFoundation
 -framework QtWebEngineWidgets
 -framework QtWebEngine
 -framework QtQuick
 -framework QtGui
 -framework QtCore
 -framework DiskArbitration
 -framework IOKit
 -framework QtQml
 -framework QtNetwork
 -framework QtWidgets
 -framework QtPrintSupport
 -framework QtSvg
 -framework QtTest
 -framework Security
 -framework ApplicationServices
 -framework OpenGL
 -framework AGL
/Users/dcortesi/Desktop/scratch/deploy is now the current directory
make failed.
ld: library not found for
 -lQt

I have no idea what it wants. The QMake file (Cobro.pro) contains the line

LIBS += -L/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/PyQt5 -lQt -lQtCore -lQtGui -lQtNetwork -lQtWidgets -lQtPrintSupport -lQtWebEngineWidgets -lQtSvg -lQtTest

I don't speak Make, let alone QMake, but that sure looks as if it's trying to say, "for Qt, look in this path" which is correct, that's where there is a Qt.so file. So I don't know. But if this step can be made to succeed, I think I'll have an app. That's exciting.

Tuesday, February 3, 2015

Flogging away at pyqtdeploy; Design issue with help

I found another blogger who is working in PyQt5 and experimenting with pyqtdeploy, and at a much more sophisticated level than I'm at. Contacted him and started a conversation about pyqtdeploy. In the meantime I have apparently exposed a couple of bugs, but as usual Phil at Riverbank is quick to acknowledge and fix them.

Unfortunately he puts the fixes in a mercurial repository. That's unfortunate because I don't know mercurial, so I'm not quite sure how to download (in git lingo, "clone") the current dev snapshot.

Soldiering on with the distributed version, which is installed with pip. Unfortunately it fails with a traceback during its "build" phase (when it byte-compiles? anyway "freezes" all the python modules). This was the second of the traceback bugs I reported. It is triggered by some slight change in the many options of the UI. It popped up yesterday. Then I used Time Machine to get back to an early version of my .pdy (project) file and it went away. Then I did some editing on the list of included standard modules and it has come back.

So now I get to figure out how to clone a mercurial repository, sigh.

Turning from that, I thought I would spend a more pleasant hour implementing the last PPQT module needed before it is alpha-test ready, namely the Help module. And here I hit a snag.

I was rather proud of the Help in version 1. I used a WebKit browser instance to display a quite detailed and well-organized (if I do say so) manual for the whole program, in hand-made HTML. For V2, I envision writing the Help material using something like the MkDocs package, a little less trouble to write and edit than raw HTML, but generally the same approach: a complete manual that can be searched or (if opened in a real browser) printed.

The Help panel was the rightmost tab in the panel tabset. This was a somewhat awkward UI because if you wanted help on, for example, the Word panel, you had to click away to the Help panel to read, which covered up the Word panel. This was one motivator for implementing drag-out panels for V2.

Well, I opened up the stub for helpview.py and read the docstring I put there months ago when I created it.

Create and manage the display of the Help file.
Instantiated from mainwindow.py and given the path to help.html at that time.

And realized, I'd never really thought this through. In V1, Help was just another tab in the tabset. But in V2, we allow multiple open books which, as I've noted more than once, changes everything.

All the tabs in the tabset are book-unique. Created by the Book, individual to the Book, and hidden or displayed along with the Edit panel for that book. Bring a different book to the front and the whole right-side tabset is cleared and repopulated with that book's functional tabs—in the sequence the user left them, with the tab open that was open when that book was hidden.

Ok, then, but Help is not a book-unique function. So a Help panel doesn't fit well into that scheme. That was my first thought. As something apart from the current book, a hypothetical Help tab would have to be a special case. When the mainwindow make a book's tabs visible, it clears the tabset and repopulates it. It would have to remember to save the Help panel widget and put it back into the tab bar afterward.

Or, my second thought, do not put Help into the tabset. Make it an independent window. File>Show Help would be a menu item with a toggle value. When on, there would be a separate Help window the user can drag about, push behind, minimize or whatever. If the user closes it, or toggles File>Show Help, it goes away.

Or, my third thought, Help could be made Book-unique. Each Book could instantiate a Help panel just as it instantiates a Word panel or Chars panel etc. The Help panel would go into the tabset. On the plus side, each book's Help could be at (a) a different position in the tab bar, if the user dragged it around; (b) open to a different page of material. On the downside, this would bring back the problem of Help hiding the panel about which you want help. So, back to thought #2. If I do it that way I can put off implementing drag-out tab panels for quite a while, maybe forever.