To eat one's own dogfood...

While waiting for a response from the chap I hope will join me in making PyInstaller 3, I decided it was time to do a bit of a test of PPQT2 usability. That means, post-processing a book myself. So I downloaded an example of the kind of book I have enjoyed doing in the past, in this case, one volume of Hawkin's Guide to Electricity.

I also opened the "Suggested Workflow" document that gets distributed as an "extra". It outlines the steps of post-processing a book with specific guidance on using PPQT features at each step. I knew it needed updating for V2, and I'm keeping it open alongside the PPQT window (27-inch iMac heh heh heh) and referring to it as a very handy task list. And correcting it as I go.

One of the first things I ran into is the small usability problem posed by V2's policy on file encoding. In V1 I really tried to support every common file encoding. There was a File:Open With Encoding sub-menu that allowed opening Latin-1, Mac Roman, Windows CP 1252, and UTF-8. There was sneaky undocumented support for other encodings.

For V2 I decided, basta, the world has moved on, UTF-8 is it. So PPQT2 assumes any input file, the book text and the good_words and bad_words files, are going to be UTF-8. It will also accept ISO-8859-1 aka Latin-1 but there's no menu command for that; you have to tell it by renaming the files to, e.g. bookname-ltn.txt or good_words.ltn. Similarly for saving: if you save the file to a name ending in -ltn or .ltn, it will write ISO-8859-1. Otherwise, it writes UTF-8. (Obviously if you know the file should be pure ASCII, it doesn't matter on input or output.)

So Bibimbop, my one tester, hit that right away, complaining that the good-words list was full of � replacement characters. Right, that's what happpens when Latin-1 is read through the UTF-8 codec. But then I did the same thing to myself! The book I downloaded was, as almost every book from Distributed Proofreaders U.S. will be, Latin-1 with a filename of projectidxxxxxxxx.txt. I opened it without renaming it, and it had replacement chars.

This was a good thing! Because I immediately realized how the user should handle this, and I put the following text into the Suggested Workflow as "Step 2: First Open".

Start PPQT2. Use File:Open to open the book. Its text appears in the Edit panel.

Select the Chars panel and click Refresh. Click the Symbol column heading to make the table sort from high numbers to low (if it is not that way already). Note if the highest-numbered symbol is � (0xfffd, the replacement character).

If the book has any of these, it was opened with the wrong encoding. Close the book immediately without saving! Perhaps it is a Latin-1 file opened as UTF-8, or it has some other encoding such as Windows CP-1252. Find out how it is encoded. If it is Latin-1, rename the file appropriately. Otherwise, use some other utility to convert it to UTF-8. Then start this step over.

On the whole, this seems a bit squiffy to me. On one hand, I definitely do not want to support other encodings. People should use UTF-8, period. Also on that hand, this is a one-time thing; it only affects the user the very first time they open a book. Once they have resolved the encoding issue for this, it is not an issue again. (Well, unless they insist on Latin-1 and enter characters that doesn't support, but that's not my problem, and the Chars tab has a tool for finding those.)

On the other hand, it lays a bit of a trap for the user, a wee pitfall that I even fell into myself. And there's potential data loss. If you open a Latin-1 book as UTF-8, which PPQT2 will happily do, and then save it, you have just trashed your file. You've written a UTF-8 file full of replacement chars where you used to have nice accented characters.

So what to do? The app knows very well when a book is being opened for the first time (there's no metadata file). Should I put up some kind of warning dialog? Should I quickly run through the text looking for \ufffd chars and then put up a warning? Maybe that.

This is what dogfood snacks are all about.

Backgrading? Nunh-unh!

Over the weekend I thought frequently about the idea of "back-grading" PPQT2 to run under Python 2.7. Finally, I decided not to do it.

First of all, there would be a tedious amount of clerical work. I'd have to insert a from __future__ import... statement in almost every module. Then I'd have to edit every single super() call to insert the classname and self parameters. (I pondered quite a while in the middle of the night last night, whether there was some way to create a global function super23() that, when called from inside some class method, could check whether it was in Python 2 or 3, and if 2, introspect its own parent classname, and dynamically make the Python 2-style call. Decided that if the Six compatibility module didn't offer something like that, it couldn't be easy to do.)

All that would be at least a couple day's work and for what? I would not want to keep the code in Python 2. I would not want to add fixes or enhancements to it, unless I simultaneously added them to the master branch. It would be work done for no benefit and for only a temporary use.

Then I thought of two more things with serious implications. The first was the SortedDict class, upon which several modules are heavily dependent. I make great use of the valuesView and keysView it supplies, and assume that these produce lightweight access paths to existing values, not massive duplications of the dictionary values and keys. It is not clear from the SortedContainers doc whether in Python 2, the "views" are even available, or whether they work as efficiently.

But the major issue is Unicode strings. All through PPQT2 I am freely interchanging strings between Qt and Python and assuming that, one, this is done with minimal overhead, just straight memory copy because the character encoding is almost the same in both universes, and two, doesn't need explicit encoding and decoding by me, again because they both just use Unicode.

That is not the case in Python 2, where even with "from __future__ import unicode_literals", I can't be positive that when I just access the entire content of a QPlainTextEdit using the toPlainText() method, that I'll get the exact contents, special characters and all. If it works at all, it has to entail an expensive conversion from UTF-16 to UTF-8. And when I take the user-entered Find text string and stuff it back into a document using a QTextCursor, how do I need to encode that, or do I?

I realized it had been a great stress relief in version 2 to just not worry about these things because Python 3 and Qt use virtually the same Unicode string contents. If I back-grade to Python 2, I could not be at all sure of identifying all the places where attention must be paid to these potential conversions. It would not be like doing a search for the next use of "super".

So, spending the time to try to make PPQT2 work under Python 2.7, only in order to be able to bundle it with PyInstaller's current Develop branch, is not a good idea.

But if not that, what? Well, I think what I need to do is to donate my efforts to getting PyInstaller fixed for Python 3. I'm putting out some email feelers to see if I can get anybody else to help in this effort. In the meantime I will continue working on enhancements to PPQT2.

The Py2.7 experiment

So I created a virtual environment with Python 2.7, and in it installed Qt and Sip and PyQt, and brought in a little test program to bundle. Pointless and ugly little thing but it runs and makes an app and a dialog box.

from __future__ import print_function
from __future__ import unicode_literals
from future_builtins import *

from PyQt5.QtWidgets import (
 QDialog,
 QMainWindow,
 QPlainTextEdit,
 QVBoxLayout,
 QHBoxLayout,
 QPushButton
 )
from PyQt5.QtWidgets import QApplication
import sys
args = []
if sys.platform == 'linux' :
 args = ['','-style','Cleanlooks']
app = QApplication(args)
mw = QMainWindow()
mw.show()
dlg = QDialog(mw)
hb = QHBoxLayout()
okb = QPushButton("OK")
okb.setDefault(True)
okb.clicked.connect(dlg.accept)
cab = QPushButton("Cancel")
cab.setDefault(False)
cab.clicked.connect(dlg.reject)
hb.addWidget(cab)
hb.addStretch()
hb.addWidget(okb)
vb = QVBoxLayout()
pte = QPlainTextEdit()
vb.addWidget(pte,1)
vb.addLayout(hb,0)
dlg.setLayout(vb)
print(dlg.exec_())
print(pte.document().toPlainText())

So this runs under Python 2.7. So I installed PyInstaller and bundled it. This failed because of an issue with the PyInstaller "hook" for PyQt5.Qt, which stupidly insists on including every module of Qt including Qml, which I did not install. But I had already fixed that in my copy of PyInstaller for Python3, and I knew just which file to edit, and did, and the resulting bundle now ran.

Then I ran the same job with the option to make a mac .app bundle and that worked too, even giving it the right icon. So, looking good.

I also installed py2app and tried it but first I hit an obvious programming error. It was an easily-diagnosed misspelling of the name of a class method that I had not run into before because it is in code specific to running in a virtual environment, and this was the first time I'd done that with py2app. After I fixed that (and opened a second issue on Oussoren's bitbucket page), it ran and made a bundle, but said bundle died immediately with an abort trap 6. No idea why, although I did notice that the qt.conf file py2app had left in the app Resources folder contained a path that could not possibly be correct. So I opened a third issue on the py2app site, but after I did that I mentally washed my hands of py2app. As Kevin O'Leary says on Shark Tank, "you are dead to me."

So how hard would it be to convert a Python3 app to Python2? I made a copy of my Cobro program and did the job. There were three issues. First, it used the Python3 urllib. Which is the Python2 urllib2 package. So I had to change two import statements and several statements that used urllib. The changes were only syntactic; just different names for the same operations.

And I had to change every call to super() to include the parent class name; again, only syntactic. Finally, running it from the command line it crashed trying to write the QSettings file. There is a bytes value which I passed to QByteArray, and for some reason Python seemed to think it needed to convert that to a string first, which didn't work, no surprise. All I had to do was wrap the expression in bytes().

Unfortunately the program crashed after it had cleared the settings, which meant that when it next came up, my long and carefully-curated list of webcomics that I read every day—was gone! No prob, I thought, I'll just restore from Time Machine. Nope! Because apparently years ago when first setting up Time Machine I had excluded ~/Library from the list of folders to back up. So later I will have to reconstruct my list of web comics.

The good news is that PyInstaller bundled cobro, although the resulting .app does not run; it dies looking for QWebEngineProcess, the undocumented piece of QWebEngine that has to exist but they don't tell you that, or where to put it. So not an error in PyInstaller, it's just that nobody has contributed a "hook" for QWebEngine. If I can figure out where it's supposed to be...

Never mind. What next? Do I really want to fork PPQT2 and make a Python2 version of it? It looks as if I'd get something that I could bundle. But it would be a chancy thing. I'm worried about having bugs like that string-handling thing in Cobro. To be really sure I ought to perform the same experiment in Ubuntu but you know, why? I'm quite confident it would work; PyInstaller was working with Python2 for PPQT version 1, in Ubuntu and Windows. I will ponder this over the weekend.

In which we learn I am full of it

A few days back I complained about QWebHistory, and how it didn't record clicks on local anchor links. I cannot figure out now why I said that. Because now I test it some more, I find that in fact, it does. I thought I had a solid case showing that it did not, but now... I don't know. I may be losing it, folks.

Anyway I coded back and forward key functions for the help viewer and they work just fine by calling the history's back() and forward() methods without any additional help from my code.

Further proving that I'm incompetent, I once again studied about rebase and merging in Scott Chacon's Pro Git, and also downloaded Github for Mac, because why be confused at the command line when you can be confused in a GUI? Anyway, then I set out to see if I could possibly merge the Python3 branch of Pyinstaller into the Develop branch.

Answer: probably not. Because if I use Github for Mac and start that merge, it shows me a list of 350 changes, the majority of which are shown with yellow status (conflicts to resolve) and many with red status (can't merge). And these are all over the many files of the project. So, no. So PyInstaller is definitely out for now -- and appears likely to never have Python3 support, at least for a long time, which makes me sad.

Meanwhile my request for advice to the cxfreeze mailing list has produced no responses of any kind. And R. Oussoren continues to not respond in any way to my very detailed debugging info on py2app.

I would like to just rant here about the absolutely abysmal state of Python3 distribution tools, but why bother? It is simply the case that it is not possible with Python3 and PyQt, to create a stand-alone, self-contained application, on any platform, period. If I had not elected to start PPQT2 in Python3, but just stayed with 2.7, I would be able to use PyInstaller at this point. But good grief, who would have thought 15 months ago, that there would be no Python3 bundlers working in 2015?

I am debating what to do next. One option is to just keep working, finish the program (writing the Translator interface that I've got a detailed spec for, and the two Translators I mean to supply) and just hope that during the 2 months that might take, somebody will get a bundler working.

Or, I could go on eLance or the like and try to hire somebody who could make pyqtdeploy work on all three platforms. But that could run into serious money. Based on my experiments, pyqtdeploy is a badly documented, flaky, and incomplete tool. Certainly making it work would require in-depth knowledge of not only Python and PyQt but QMake, Make, and the C++ interface between Python and Qt. It would be hard finding an eLancer with those skills in the first place.

Or, I've seriously thought about chucking the whole project. Who needs it?

Edit: So I took a walk and thought about what I'd written above, and one thing that stood out is the possibility that if I had written V2 in Python 2.7, stuff might be working, in particular PyInstaller and Py2app. So, how hard would it be to retrograde the code to Python 2 syntax? Probably not awfully hard, all the super() calls and probably a few subtle things I used without realizing they were 3-level syntax. But would it really help? So I decide what I will do tomorrow is to set up a virtual environment based on 2.7, with PyQt5 and all the other packages. In that VENV I will try bundling a couple of small test cases I have, not the whole app but just something that makes a main window with a widget in it. I'll have to do that at least on Mac OS and on Ubuntu, so several hours of work potentially, but at the end I'll know if there really exists a smooth path to bundling in the older language. And if so, then back-converting the whole app shouldn't take but a few days.

So at least that's something positive to try before I totally crumble.

Spinning the old wheels

First off today, I spent some time nailing down what is happening with the cx_freeze bundle in Linux. Then I wrote what I hope is a detailed and constructive note to the cxfreeze list,

I am trying to freeze an app on Ubuntu 14.10. The app uses PyQt5 and runs under Python3.4

Cxfreeze creates a bundle into which it copies all the dependent libraries,
for example libQt5Core.so.5, libQt5Widgets.so.5, and so forth. It also
copies the PyQt libs such as PyQt5.QtCore.so, PyQt5.QtWidgets.so, etc.

When I move this bundle to another Ubuntu system where this version of Python
and Qt are not installed, I get the dreaded message,

      Cannot mix incompatible Qt library (version 0x50300) with this library (version 0x50401)
      Aborted (core dumped)


I think it is trying to load from the local Qt (Ubuntu has some version
of Qt installed by default). On that system, after copying the bundle
from the dev system, I see this:

    $ cd Desktop/PPQT2 # the bundle made by cxfreeze

    $ ldd libQt5Widgets.so.5
        linux-gate.so.1 =>  (0xb779c000)
        libQt5Gui.so.5 => /home/dcortesi/Desktop/PPQT2/./libQt5Gui.so.5 (0xb6b49000)
        libQt5Core.so.5 => /home/dcortesi/Desktop/PPQT2/./libQt5Core.so.5 (0xb6605000) 

       ...


That is good! libQt5Widgets is linking to the bundled copy of libQt5Gui and so on.

However:

    $ ldd PyQt5.QtWidgets.so
        linux-gate.so.1 =>  (0xb7776000)
        libQt5Widgets.so.5 => /usr/lib/i386-linux-gnu/libQt5Widgets.so.5 (0xb6afb000)


That is bad, PyQt5.QtWidgets is linking to the local native
libQt5Widgets -- NOT the one bundled right there alongside it.

Why is this? Am I doing something wrong? Or is something needed in the hooks for PyQt5?

Thanks for any help,

Until I get some help with that, I have no way to create a bundle on Linux.

Nor on MacOS either! I took the cx_freeze MacOS bundle to my wife's iMac where there's no Qt or Python and of course it didn't run; it dies looking for a Python in /usr/lib, I believe — I forget and I've deleted the evidence now. Anyway, something that should be in the bundle, probably is in the bundle, but that isn't where the app goes looking for it. Basically the same issue as on Linux.

I would really, really rather use py2app for MacOS anyway, as it almost makes a nice looking MacOS app object. But not quite. So I started trying to debug that code and spent a couple of hours on that. You want to see a nice detailed "issue"? Check this out. (edit: that's weird, there should be 5 comments on that, not 2, and the last 3 are very detailed debugging info.) No response from R. Oussoren yet.

It's an infuriating bug because it does not fail when run under the debugger. It only fails when running at speed from the command line.

Well, so cx_freeze is failing for MacOS and for Linux. And py2app is failing for MacOS. I haven't gotten around to trying it yet, but I don't have a lot of hope for py2exe on Windows. It claims now to support Python3, which is good, but its writeup doesn't say anything about Qt, which is ominous.

Back in V1 days, I didn't worry about all these other bundlers; I just used PyInstaller. I liked it so much I rewrote the manual for it, and I put in several days work doing the initial coding to move it to Python3 and start using Oussoren's ModuleGraph. That was more than a year ago.

Recently I got it kind of working on MacOS. But when I tried it on Linux I ran into problems that, it turns out, had already been fixed on the Python2, or main, branch. There's been quite a bit of maintenance applied in the year-plus since Python3 split off. So I wrote to one of the maintainers, and this was the reply,

> It appears my problem is trying to use the Python3 branch, and this
is missing maintenance since I don't know when, a year?

Yes, we are lagging behind a lot. development-branch advanced and 
modulegraph was merged with python3 somehow. This needs to be cleaned
up and merged/rebased - but this is a lot of work nobody has started yet.

So basically, I'm screwed. PyInstaller for Python3 is just unusable until someone does the job of merging all commits since early 2014, into the Python3 branch. I have no idea what-so-ever how to do that. None. And the people who presumably do have a clue, are too busy.

Things are not looking good on the bundling front, frankly. And without a reliable way of making standalone, independent bundles on all three platforms, I don't have a product.

Thursday I will hack away at the Windows installation, installing the needed visual C, and then py2exe, and maybe that'll work. Maybe the cx_freeze crew will have a suggestion that will fix it on both MacOS and Linux. (and maybe... never mind)

What I can and can't control

Started today with things I can control, namely, issues that affect my code. I fixed a minor issue where the numbers in the Footnotes table weren't vertically aligned same as the text items. I fixed the issue where the Bookloupe table produced a bunch of console messages when first refreshed. I closed two issues that had been rendered moot by an enhancement to the natsort package. I fixed the issue about the Help viewer not having any "back" facility.

Implementing "Back" on the Help

That was interesting. It brought me back to familiarity with the limitations of QWebPage and QWebHistory. When you first load a web page from a file, its browser history is empty. And it stays empty during any number of link-following clicks, if those clicks are only to anchors within the current page. It just doesn't record local links. Only if you click away and make it load an external URL does it make a history entry.

Well, what does a user do with a Help page, which is quite a complete little manual with a TOC on one side? Clicks around to local links, of course. Rarely exiting the page to another page. So for most of what the user does, there is no history, and self.view.history().canGoBack() returns False.

So I had to implement my own history stack. This turned out to not be difficult. There is a facility for "delegating" all links. When this is done, the user's clicking on a link only creates a signal. The slot for the signal gets a copy of the QUrl which is the target. So I just got the string value, and if it began with "file:" then I assumed it was local and appended it to a list of strings. Then handed it on to the QWebView.load().

When the user hits one of the keys ctl-[, ctl-b, or ctl-left, my keyEvent code asks first if self.view.history().canGoBack(). If so, call self.view.back(); done. Else, is the list of link strings nonempty? If so, pop off the last one, make it a QUrl, and call self.view.load(). If the list is empty, call the subroutine that reloads the help file; that puts the view back to the top of the page.

Bundling for Linux

Then, feeling rather full of myself, I ran cxfreeze to bundle the newly updated app, and zipped that up and put it on my Public Dropbox.

Then I fired up my "test" system which has none of Python or Qt installed. Well, turns out that a stock Ubuntu 14 does have Qt installed. I kind of knew that, because I kind of knew that some of the Ubuntu programs (Software Updater, I think) are Qt-based. Anyway the test is, does this stock system run the bundled app.

Would you like to guess?

No, of course it bloody doesn't. "Cannot mix incompatible Qt library (version 0x50300) with this library (version 0x50401) Aborted (core dumped)"

Sigh.

If you search on that string you will find a whole lot of people running a wide variety of software over the past 3 years who hit that exact message. What it means is, the PyQt5.QtCore library is trying to load libQt5Core.so, and it is getting one, but from the local system instead of from the bundle-folder.

But Dave, you say, I thought that you were all happy with cxfreeze because it was bundling everything, all the dependencies.

Well, it appears I was deceived. It is not bundling the Qt5 dependencies. It is treating them like libc.so, to be obtained locally.

I quickly trained myself on ltrace and ldd, the Linux equivalents of tools I was using on MacOS to figure out what nuitka wasn't bundling. Just as soon as the program calls into PyQt5.QtWidgets to create a QApplication, boom. If I do ldd on PyQt5.QtWidgets, I can see the links to /usr/lib for the libQt* libraries.

Edit: I was mistaken. The desired libraries, e.g. libQt5Core.so.5, are in fact there in the bundle. They are being copied correctly. However, the links revealed by, for example, "ldd PyQt5.QtCore.so" are not pointing at the files in the bundle. They are pointing at /usr/lib. The problem is not what is being copied, it is the link info in the PyQt5.Q* modules.

The link info in the copied libraries, e.g. the links from "ldd libQt5Core.so.5", have been altered to point into the bundle. So it is a matter of pointing the links in the PyQt modules as well. I now have to go and find out how cxfreeze does that, when it does it, and why it doesn't do it in all cases.

Opening the Window (7)

Earlier this week on my Windows 7 VM I downloaded and installed the official Python 3.4.3 distribution (as well as Qt5.4, Sip, and PyQt5.4) and thought I had pretty well got my Python environment set up. I could run python from the DOS command line, and WingIDE found it and all.

However, the first time I tried to run PPQT under Wing, it immediately reminded me that it couldn't find hunspell. Doh! of course, I need to install the four packages PPQT depends on: regex, sortedcontainers, natsort and hunspell. Well, that shouldn't be hard, just pip-pip-pip.

Well, of course not. First off, on my Unix-type boxes I'm used to pip being a command, a script in /usr/bin. Not Windows. There's no pip command to be found anywhere on the path. So I search the internet. Ah, in Windows one uses the command Python -m pip. All right, I've gotten used to ending a Python session with ctl-z, return; I can get used to this.

I do that—and there is no module named 'pip'.

Search around some more. According to what I see, pip is definitely supposed to be automatically included with Python3. But not mine! Oh, alright then, I go to pypi and download the pip package, which is of course a tar.gz file. Which Windows does not know how to open! It only knows how to open ordinary zips.

Back to my search window. How to open a tar.gz file on Windows? Top recommendation, install 7Zip. Which I do. And now I can unpack the pip distribution file.

So, cd into the folder and there is the usual setup.py. OK, I know what to do: "python setup.py install". No, of course not. Because that tries to import setup tools, and that doesn't exist in my nice new Python 3.4 installation.

Back to pypi and download ez_setup. Run that, and I now have setup tools. Now I can run pip's setup.py. And I've got the pip.

So install natsort and test it. Good. Install regex, test. Good. Install sortedcontainers, test. Good.

Take deep breath. python -m pip install hunspell. Nope. "error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat)". Well, kind of expected. So I need to get that. I'm sure it's available. But it's nearly 6pm Sunday and it will just have to wait for... probably Thursday, at a guess.

Bundling on Ubuntu

Poor old PyInstaller

OK so this morning I went back to the computer and into my Ubuntu VM and tried to get PyInstaller working. Several problems. Most dire, it aborts with the message "Security-Alert: try to store file outside of dist-directory". Quick duckduckgo search shows this has been encountered and probably fixed a few months ago, with code added to hook-distutils.py.

I realize yet again, that the Python3 branch of PyInstaller forked from the main line nearly a year ago, and it is missing a fair amount of maintenance. But I have no choice but to use it. It worked, more or less, on Mac OS, but now I'm hitting Ubuntu-related problems that have been fixed since the fork.

For example, the bug that stopped me last week on Ubuntu and entailed a lot of diagnosis and brain-sweat, the issue where the bundled app died looking "orig-prefix.txt"? I figured out how to fix it, and at one of the maintainers' request I opened an issue. Now I discover that it had been fixed some time ago on the main branch. So I closed the issue. What a waste of time!

I look at the current hook-distutils.py and think, heck, why not? So I copy its code and paste it in to replace the contents of the version I've got. Nope. When it runs, it wants things from compat.py that my version doesn't provide. Ohhhhh-kay, let's get the mainline version of compat.py. Nope. Very bad idea. It has several things that fail under Python3, it is Python-2 dependent. There's just too much divergence, I am not going to try to do a manual merge of the two branches.

cx_Freeze

Plan B, is to try cx_Freeze. So I pip-install it, or try to, and the install ends with a flock of error messages. I'm about to email their support list but I try one quick search and of course somebody has had the problem and offers a one-line edit of its setup.py to prevent it happening. Now I can't use pip to install it; I need to download it so I can edit setup.py. Which I do, and I do, and when I run the setup.py install, it runs.

So I run cxfreeze and it makes an app bundle, but just as with Mac OSX, it forgets to include QtWebKit (which is explicitly imported) and QtPrintSupport (which is not). Redo the bundle telling it to include those modules and ta-daaa, a working Ubundle!

Well, almost. It puts out an error message right away that tells me it has the old level of the natsort package, before Seth added the UNGROUPLETTERS option for me. So I upgrade that package, and the app still fails. Eh? Oh! I have to rebuild the bundle to pick up the new package! Doh! That done, it works.

Except for bookoupe, because I haven't installed that. But when I do, that works also, except for a stream of errors when the table is first built -- as reported earlier by bibimbop. Well, that'll be easy to fix. I've now noted several minor fixes that are needed. But I can zip through them Monday and have a working Ubuntu alpha bundle. Yay.

Before I do that I want to have just one little trial of Nuitka on Ubuntu. I think it may work on this platform, and if it does I want to know the difference in the bundle sizes.

Nuitka

So I installed Nuitka, which is a quick and simple process, except when you foolishly do the install from a different terminal window, one in which you have not "activated" your virtual environment. So Nuitka installs under Python 2.7 and into /usr/local/bin. So when it tries to compile a PyQt-using program, it can't find the PyQt modules.

So I spent ten minutes un-installing it (and, by the way, why does the standard setup.py not have an "uninstall" option??) and re-installing it in the virtual environment. Then I ran the compile and it died after several minutes with "virtual memory exhausted"!

So I had to shut down the VM and reconfigure it for more hard disk space. Now the compile is proceeding again. Nuitka is not fast, especially with a big program like PPQT.

OK, so it dies on the same damn assertion error that I just helped Kay fix in MacOS. It is screwing up getting the list of Qt plugins, except that I fixed the problem when it was generated by code specific to MacOS, code that ran the "otool" command and parsed its output. This would be going through a different subroutine, one for Linux, and I am not disposed to try to debug it. Fuck it. Nuitka is out as a Linux bundler. Cx_freeze rules here.

Sorting out the MacOS Bundle options

I spent about 3 hours yesterday and 2 hours today working on the Nuitka failure. It compiles all the code; builds the bundle; the bundle contents look very similar to the bundle built by PyInstaller; but it won't start. There's something wrong with the way that QApplication loads plugins that causes the app to either die or hang up and need force-killing. I finally wrote a note to the list detailing the failure and basically kissing it off. "I may try Nuitka on Linux some time."

With that out of the way, how shall I bundle PPQT2 for MacOS? For V1, I was pleased to be able to use PyInstaller on all three platforms, and I had a near-religious assumption that it was good to use the same tool on all platforms. But really, that isn't necessary. If I end up using cxFreeze on one, py2exe on another, who cares? The key is, do they build a bundle that runs; and if so, which builds the smallest bundle?

PyInstaller

I've got it working on MacOS with two small issues. One, when you launch the app from inside the folder, there's a stream of console messages apparently listing the different dylibs as they load. No idea why. And it has the problem that the app comes up and opens its window, but its menu bar does not appear until you click in another app, then click back to your app. Then you have a menu bar. There's a pull request with a purported fix; I need to get that onto the Python3 branch and try it.

Also as I noted a few days ago, it builds a bundle that zips to 69MB.

cxFreeze

I succeeded in getting it to bundle PPQT2 with about an hour's work of reading and experimenting. It does not do as good a job as the others do, of tracing out the implicit PyQt imports. I had to tell it specifically to include QtWidgets and QtPrintSupport; otherwise the app died looking for those. But once that was out of the way, it worked. Well, almost! It has the identical problem that PyInstaller apps have with the menu bar not appearing right away.

Bundle size as a zip file? 88MB. Ooops. So no reason to prefer it over PyInstaller.

py2app

The Python3 version of PyInstaller actually uses a big chunk of code written by Ronald Oussoren for his py2app package. So why not try it? I have to say that it does a good job, up to a point. It is the only one of the four (counting Nuitka as a bundler) that actually builds a MacOS app bundle with a proper icon. The others make an ordinary folder. The user has to go into the folder and double-click the PPQT2 file icon, which is an ordinary "unix executable" icon. While the app is running, that is also its icon in the dock.

py2app offers two modes. It recommends -Alias build mode for development; in this mode it makes an app bundle but populates it mostly with symlinks to the dependency files. This goes fast and takes little space, but of course isn't portable. In this mode it works well and the resulting app bundle bears my chosen icon, and while running, shows my icon in the dock.

Way back when I made the icon I want, but was never able to use it with V1. I feel a bit of trepidation about using it, but it's so cool... and with py2app, I can have it.

So what's the problem? Well, when I run py2app not in -Alias mode, it starts to build a complete self-contained app bundle and of course crashes with some stupid error. So I sigh opened an issue on Oussoren's bitbucket page. If he responds in reasonably short order, I'd prefer to use py2app. Oh, the size of its zipped-up bundle? 62MB, and it hadn't quite finished building when it quit. So, about the same as PyInstaller.

That does it for Mac. Tomorrow I am determined to make something, something work on Linux.

More Tuesday

Windows #3

So starting the Qt installation on Win7 and the Wizard wants to know what components to install. There is quite a lengthy list. It is easy to uncheck the ones related to Android and to Windows Phone. But there are at least 3 versions of OpenGL and at least 3 of Qt components. (I can't refer to the list right now because the wizard is busy downloading). All of these components note their compilers, e.g. MSC 2013 32-bit or such.

Now, I know this is a 64-bit Windows system. So I don't want any of the 32-bit things -- do I? But also, I know that it is supposed to be important for things that link to Python, that they use compatible run-times. I fire up Python and it says it is MSC 1600. Wut? Some DuckDuckGo lets me learn that this is equivalent to "MSVC 2010 (10.0 with msvcr100.dll runtime)". OK here's the thing, none of the components in the Qt install wizard list mention MSVC 2010, except one, and it's a 32-bit one.

Now what?

I install a random assortment based on guesswork. It finishes and automatically starts up Qt Creator, which doesn't come up but puts up an error dialog that is I swear to God, 500 words long, about how it can't find the OpenGL components it needs, do I have the right graphics drivers? and please make sure this, that, and the other thing are available on the PATH.

So obviously I chose wrongly.

So I run the wizard and tell it to remove all components. Will try this again.

Go to the qt.io download page, bypass the easy download-for-windows button, go to more downloads. OK, here is the list of available download packages for Windows:

    Qt 5.4.1 for Windows 64-bit (VS 2013, 722 MB)
    Qt 5.4.1 for Windows 64-bit (VS 2013, OpenGL, 711 MB)
    Qt 5.4.1 for Windows 32-bit (VS 2013, 705 MB)
    Qt 5.4.1 for Windows 32-bit (VS 2013, OpenGL, 695 MB)
    Qt 5.4.1 for Windows 32-bit (VS 2012, OpenGL, 644 MB)
    Qt 5.4.1 for Windows 32-bit (VS 2010, OpenGL, 627 MB)
    Qt 5.4.1 for Windows 32-bit (MinGW 4.9.1, 856 MB)
    Qt 5.4.1 for Android (Windows 32-bit, 939 MB)
    Qt 5.4.1 for Windows RT 32-bit (766 MB)

OK, one of the first two. But do I want the one with OpenGL or not? Given this is a Parallels VM, maybe it doesn't support OpenGL. OK. First one. While it downloads,

PyInstaller #2

On Ubuntu, the PyInstaller bundle "can't find platform plugin xcb". Well, libqxcb.so was in fact bundled, but not where the PyInstaller bootloader wants it. It (I think I blogged about this a few weeks ago) forces Qt5 plugins to the nonstandard folder qt5_plugins/platforms. No such directory in the current bundle, so clearly either the needed build-time hook didn't run, or isn't up to date. For the moment I finagle it manually creating qt5_plugins/platforms and move libqxcb into it. And hey, the symptom changes.

This issue will need to be revisited, to fix that hook. But now we have,

PyInstaller #3

Bundle fails because "ImportError: No module named 'mainwindow'". This seems unlikely. That's one of my modules which is imported by the top level program, PPQT2.py. OK, quickly resolved: there's a warning in the warning log, "no module named mainwindow". If that isn't imported, nothing of the rest of the program is imported. Because that imports book.py, which imports just about everything else. So that's a problem.

A problem two ways; it also means I have to re-run the PyInstaller build and that means I'll lose my hand-fixed libqxcb thing. So I need to fix that issue too.

Windows #3

Oh, hey, the Qt 700MB package is done downloading.

Much better; this wizard has a very short list of options, no multiple choices of compiler or whatever. So run it.

And oh, dear, same problem. When Qt Creator tries to come up, it creates the frame of its main window but only a dialog box "Welcome Mode Load Error - Failed to create OpenGL context for format QSurfaceFormat blah blah blah This is most likely caused by not having the necessary graphics drivers installed. Install a driver providing OpenGL 2.0 or higher, or blah blah blah."

I have no idea how to proceed from here.

Tomorrow's my day at the museum. Maybe I'll think of something by Thursday.

Edit:OK, so I did a quick duckduckgo on "install opengl in windows" and immediately found a forum post at the opengl site saying, no need, you already have it, just make sure your display driver is up to date."

That user then asked, "how I do that, eh?" and they said, "dxdiag in a console window." OK, so I did that and this nice program told me my graphics driver is Parallels. Of course it is, it's a Parallels VM. So then I searched on "parallels display adaptor opengl" and quickly found instructions from the Parallels site on how to change the configuration of a VM to enable OpenGL. One has to set on the configuration option "Enable 3D Acceleration". OK, that is not obviously OpenGL but whatever. I do that, reboot the VM, and hey hey! Qt Creator comes right up all happy! So that would be issue Windows #3 solved.

Tuesday doin's

Ubuntu #4

Referring to the last couple of posts, this was the issue where the locale-aware sort no longer sorted uppercase-initial words apart from lowercase-initial ones. I'd accepted that this was how it would be. But at the end of the day yesterday the natsort maintainer popped up with a new feature to fix it! This morning I tried it, and by golly, it works! So that's fixed. Thanks, Seth!

Windows #2

This was that Wing IDE didn't find the Python 3 interpreter. Well, after a reboot, it did. So lesson learned: when anything strange happens in Windows, reboot it! Seems to be the case in Win 7 just like in the old days. So that's fixed.

Another oddity was that Wing claimed its settings (including an error log) were to be found in C:\Users\Administrator\AppData\something (yes, I am running as the Admin even though I know I shouldn't). But when I opened a Windows Explorer (file browser) and tried to navigate to that, it didn't seem to be there. But if I clicked in the address bar and typed in the \Appdata part, there it was. So who knows. Anyway that mystery is kind of solved; I can now get to the Wing error log should I ever want to again.

Pyinstaller #1

This is the problem where a PyInstaller bundle dies looking for "orig-prefix.txt" which should only exist in a venv and is irrelevant to a bundled app. After my longer note to the list, one of the maintainers pointed out there is a hook, hook-site, which is supposed to fix exactly this. Oh! A hook! There are issues with hooks in the Python3 branch of PyInstaller (much of which I wrote last year) so I dived into it expecting to find either some problem with my code, or else a hook that needed updating to use the Python3 facility.

Good news: not a problem with my code, which is just fine and working perfectly for another hook that does a similar task. Bad news: the code the hook-site.py code uses ought to work, also, but doesn't.

The issue is detecting when a virtual environment is in effect. The test it is using is sys.prefix!=sys.base_prefix, and that is exactly what the Python3 doc says to do. But for some reason, in my installation, first, sys.prefix==sys.base_prefix and both point to the virtualized Python, and second, there is a sys.real_prefix which has the correct not-virtual prefix. However, that sys variable is not supposed to exist in Python3 at all! It was a Python2.x thing. The hook code tests to see, am I in Python2 or 3? If 2, it tests sys.real_prefix, if 3, sys.prefix. So I don't know what is supposed to be happening here. I kludged the hook code so it worked, but not in a way I would want to expose to a pull request against PyInstaller.

But with that, the bundled PPQT2 started up...

Pyinstaller #2

...and immediately died with "failed to start because it could not find or load the Qt platform plugin 'xcb'." So there is something new to diagnose. Qt fucking platform modules are the bane of my existence.

To Review...

So I started this silly thing of numbering my problems. Let's review them and see where we stand.

• Nuitka #1, couldn't import sortedcontainers. Fixed by Kay.
• Nuitka #2, can't copy qliboffscreen.dll; Kay acknowledged my debugging trace and said he'd look at it tomorrow. Pending.
• Ubuntu #1, PyQt5 couldn't install; fixed by repairing broken symlinks to libGL.
• Ubuntu #2, horrible graphic mess on launch, fixed by forcing QApplication to use something other than the default GTK. Kind of a hack but I'm not going to worry about it.
• Ubuntu #3, PPQT error initializing fonts, a one-line coding error. Fixed.
• Ubuntu #4, word sorting of initial caps, fixed by natsort maintainer.
• Ubuntu #5, spellcheck not working, went away by itself.
• Mac OS #1, 69MB zipped bundle. It is what it is, but possibly a Nuitka bundle would be smaller.
• Pyinstaller #2, Qt plugin not found. Open.

So there's just two open problems, the Qt plugin on Ubuntu and getting Nuitka to bundled on Mac OS. And to complete the Win7 installation process and begin testing there.

..and some more...

Windows #1

Not having enough pending and open problems, I fired up my Win7 box and prepared to install Python on it, the first of many steps needed (Python, then Qt, then Sip, then PyQt5, and hunspell and natsort and sortedcontainers. And Wing IDE at some point.) And of course step 1 didn't complete.

"There is a problem with this Windows Installer package" and "Python 3.4.3(64-bit) setup ended prematurely". Google. Well there should be something called msiexec that leaves a log file. Somebody else found the problem was only with pip, so don't include that from the checklist. I'll try that, but first, since it has been months since this VM was up, there are 28 important Windows Updates to install. So let that run first.

Yup, removing Pip and several other optional installs from the list enabled the installation to run properly, creating C:\Python34\python.exe. Unfortunately double-clicking that produced a console window with an error about unable to load encodings, but I could see in the message some text like "2.7". A quick google turned up a stack overflow note; yes of course I need to set PYTHONPATH and PYTHONHOME.

Oh crap how do I set environment variables in Windows? It's been years... Windows help was not helpful at all, searching it on "environment variable" and "system variable" produced absolutely nothing relevant. Quick google -- aha, yes, Control Panel, System Properties, Advanced, way at the bottom obscure "Environment Variable" button. Teeny tiny dialog box that cannot be made larger, so editing long strings in a tiny slit of a window about 12 characters wide. Find PYTHONPATH, change all the "Python27" entries to "Python34". Also add a PYTHONHOME var.

Now double-clicking python.exe produces a console window with a prompt. Yay me.

Windows #2

Turns out I already had Wing IDE installed, but an old version. Download and install the latest one. However, it is opening Python 2.7. How to change it. Its help is actually of help! Go to project properties for the default project, click Browse, browse to Python34.exe, click OK... and it is not OK. It claims "Could not launch or inspect the Python executable 'c:\Python34\python.exe'. It should be the name of a Python interpreter that is on your PATH..." Wait, what? On the PATH? But I just took you by the hand and led you to it, there it fucking is, open it!

Double-check, yes, C:\Python34 is the first item in the PATH variable. Actually it was named "Path" but I made a new variable "PATH" with the same contents, which seemed to replace "Path" so probably case didn't matter. But no, Wing won't open it.

Time to quit for the day.

...after another, after another...

Ubuntu #3

So the problem trying to initialize the fonts was that when fonts.py read the monospaced font's size from QSettings, it got a string '13' instead of an int. Which QFont() didn't like, understandably. All I had to do was wrap an int() around it. Inexplicably, I had already done exactly that two lines earlier, for the general font's size value. Why didn't I do the same for the near-identical monospaced code? Anyway, that was easy.

Ubuntu #4

Next, playing around with PPQT run from source, I noted that there was no difference in the Word table sorting between the Respect Case switch on or off. That should be being handled by the natsort key generator. It works on Mac OS. I suspect that I will have to install the pyICU package, as I did on Mac OS.

Ubuntu #5

Also playing around I noted that there were only two misspelled words, and they were correctly spelled to my eye (ergo, the are probably in a bad-words list but I didn't check that), and when I added deliberate misspellings and nonsense words, and Refreshed the Word table, they didn't show up as misspelled.

What all that says is, the spell checking isn't working. When PPQT can't create a Hunspell object, it just silently passes all words as correct. Except those in the bad-words list. So I need to debug that.

Nuitka #2

Monday morning now. I sat down and tried to use Wing IDE to walk through the Nuitka code and set breakpoints where Kay suggested. Unfortunately, for some reason, the compiler starts out by re-invoking another copy of itself. And that copy of course doesn't see the breakpoints I set. So I had to insert print() statements instead. Which I did, enough to show, I hope, where the code goes wrong. It is trying to copy "qliboffscreen.dll" but has misplaced the full path to the source file. I put together a long note to Kay showing what I'd learned.

Ubuntu #1

Nobody had replied to my query to the Pyinstaller list about the bundled app dying trying to find something called "orig-prefix.txt". So I investigated further. Using Wing IDE's search in files method (could of course have use BBEdit's similar feature) I quickly found that "orig-prefix.txt" is a file that is only expected when running in a virtual environment. It's just a one-liner that the virtualenv command leaves in the local lib/ directory to point to the location of the original Python interpreter.

Ah, so that would be a difference from Mac OS; it is only in Ubuntu that I'm using virtualenv. So yes, fine, at the time that PyInstaller is bundling the app, it might need to look at this file. But there should be no need for it when actually running out of the bundle. It's completely irrelevant then. But something is looking for it.

Further investigation showed that what is happening is that the PyInstaller bootloader is in the process of loading all the built-in modules, and it hits one called site.py, apparently a standard, whose function is to initialize sys.path. That's suspicious right away, because in a bundled app, sys.path needs to be set up very particular ways by the PyInstaller bootloader. No generic module is going to get it right.

Anyway, this site.py has an attempt to open orig-prefix.txt about 5 lines in, not conditional, not in a try/except, just open that file by that name. That's very strange, because I don't think that file is normal to the standard Python implementation, only to virtualenv. I'm speculating that this is a version of site.py that is peculiar to virtualenv. In which case PyInstaller ought to know not to use it. PyInstaller has specific mention of virtualenv in its documentation, they recommend using it, so if it has this peculiarity, the developers ought to have known about it.

Anyway, I sent off a longer and more informative query to the PyInstaller list. Hopefully somebody will give me some guidance on this.

Ubuntu #4

Returned to the issue of sorting. It turns out since I went to the natsort package for sorting the Word table, the Respect Case sort is not behaving like it used to do. Back in the unnatural, or locale-not-aware days, these words would sort as follows:
['All','Ball','all','ball']
Now they sort as:
['All','all','Ball','ball']
and the only difference that turning off Respect Case (which adds the IGNORECASE option to the natsort key function) is to sort them like:
['all','All','ball','Ball']
and this (it turns out when I actually look) is true on MacOS as well as Ubuntu. So it isn't a bug, it's a feature.

It's a feature I could well do without, thankyouverymuch! It's very useful to be able to group all the uppercase initial words at one end of the list. That puts all the proper nouns together. Well it also mixes in among them, all the sentence-opening ordinary words: not only Arnold and Albuquerque, but also All and Any if they are capitalized. Anyway, I have written to Seth Morton, the natsort maintainer, to see if there is some way to restore the pure-ordinal behavior, while retaining the proper ordering of accented characters.

If there's no way to get that behavior back, I will add another filter, initial-cap, to the popup menu of filters.

Ubuntu #5

The spell checker is working perfectly now. I don't know what I saw Saturday, but it's gone now.

Mac OS #1

PyInstaller is working on Mac OS anyway, so I prepared a Mac bundle complete with the extras folder and a README and zipped it up and it came out to... 69 megabytes. Ye gods and little fishes! That is almost double the 39MB of the V1 bundle for Mac OS.

Just one goddam thing after another...

I'm now working on multiple fronts to try to make a nice distribution package of what I'm calling the alpha version of PPQT2. And the world is fighting back at me hard. Here's some of the scrimmages over the last 24 hours.

Nuitka #1

In recent days the hard-working Kay Hayens has added more support for making standalone PyQt apps with his Nuitka compiler. It not only compiles one's Python code to C++ and compiles it, it also tries to pull in all related modules and DLLs into a single folder. There were (and continue to be) issues with Nuitka compiling Cobro, because Cobro uses QWebEngine. But PPQT2 does not, so I thought I'd give it a try.

Immediately the compiler died trying to handle the import of the sortedcontainers package. I reported this with a one-line test case on Friday afternoon. A few hours later, Kay had fixed the problem. Nuitka saw two paths to the same module, via /Library/Frameworks/Python.framework/Versions/3.4 and via the standard symlink, /Library/Frameworks/Python.framework/Versions/Current. Mishandling of imports. So that was fixed by Saturday morning.

Nuitka #2

So now the Nuitka compile goes all the way through the gathering and compiling and C++ compiles to the point where it is pulling in DLLs and then it fails because it can't find "qliboffscreen.dll" which does exist in the Qt plugins directory. Setting DYLIB_LIBRARY_PATH doesn't help. So I fire off another email this morning at 10:24am and get an answer at 10:32! With a suggestion on where in the Nuitka code I can start debugging the problem. ohhhkay then.

Ubuntu #1

After sending the note on Nuitka yesterday afternoon I fired up my Ubuntu 32-bit dev system and installed virtualenv, and made a VENV, and inside that I installed Qt5.4 and SIP and tried to install PyQt5.4.1. Unfortunately that didn't work.

PyQt has a very (very) elaborate config.py which tests lots of things and decides which modules it can build and sets up dozens of little make files. But it was deciding it could build only about 1/4 of the available modules. And of course it didn't say why.

So I started reading the code of its config.py. Turns out, it has a long list of modules and it tests whether it ought to build each one by creating a tiny C++ source file that just does a new to create an object of the corresponding Qt class. Then it uses popen to run qmake to run the compiler to compile that source, and if the compile works, it sets up to build the matching PyQt module. And if not, it silently omits that module from the build list.

So 3/4 of the modules in the list were not compiling. Why? Well, there is a --verbose switch. Turning it on displays the qmakes. And they were all failing because "cannot find -lGL".

WUT?

Google google google.

Ok, thats libgl.so, and it should exist, but just to make sure I bring up the Synaptic package manager and re-install libgl-dev. And I verify that there does exists a file:

/usr/lib/i386-linux-gnu/libGL.so

Which is in turn a link to

/usr/lib/i386-linux-gnu/mesa/libGL.so

Which is in turn a link to

/usr/lib/i386-linux-gnu/mesa/libGL.so.1.2

And LD_LIBRARY_PATH is set correctly. So, all good, right? Wrong. The makes still report "cannot find -lGL". So I fire off a terse note to the PyQt mailing list, hoping somebody will point out an obvious mistake, and call it a day.

Saturday morning I return to the Nuitka front, download the latest dev version with the results noted above. Then because there was no reply to my PyQt query this morning, I decided I would tackle that by making a copy of libGL.so into every place I could think of, starting with /usr/lib. So I sit down and do,

sudo cp /usr/lib/i386-linux-gnu/libGL.so /usr/lib

And guess what? cp "cannot stat" that file. WUT?!? Turns out, that link is dangling, there is no target file at the end of it. So also is the link /usr/lib/i386-linux-gnu/mesa/libGL.so. At the end of the chain there is no file libGL.so.1.2. Just dangling links all the way down.

There was in fact a mesa/libGL.so.10.1.2.28859, and when I copied that to /usr/lib, the PyQt5 configure and make proceeded normally. So somehow for reasons I cannot fathom, the installation of libgl-dev left broken links.

PyInstaller #1

So now I have a working PyQt5 on Ubuntu so I downloaded the Python3 branch of PyInstaller to Ubuntu and used it to bundle PPQT2. It seemed to go fine, but when I tried to execute the bundled app, it died in the PyInstaller boot-loader module looking for something called orig-prefix.txt. I looked to see if the MacOS version of PyInstaller had made anything like that, but no. So... send off an email to the PyInstaller list.

It's now nearly noon Saturday. I'm going for a pleasant walk with my wife, have a coffee. Maybe this afternoon, if I feel real energetic, I will return to the Nuitka #2 problem.

Ubuntu #2

OK so she wasn't ready for a walk and I returned to the keyboard. Thought I would at least try out PPQT2 from source in Ubuntu. Glad I did, gaaahhhh!

When I ran it, as soon as it opened the window, there was just all kinds of bad stuff happening on the screen. There was a huge title bar the full width of the screen. When I dragged the window around it left comet-trails of pixels. And the terminal window was full of looping messages about QPainter issues.

I killed it (which wasn't easy, the only way to kill it was to click the (x) in that full-width title bar) and pondered. Then I tried executing a tiny little test file from my trials folder. This creates a mainwindow with just three widgets, it's about 20 lines of code.

Same deal, constant messages from QPainter. But by killing it quickly before they could fill up the scroll-back area, I found out that on startup there were several messages of the form

(python:16446): Gtk-CRITICAL **: IA__gtk_widget_style_get: assertion 'GTK_IS_WIDGET (widget)' failed

Googled that. Very common problem. Found people who had this and got around it by setting the QApplication style to something other than the default, thus avoiding use of GTK. I quickly modified my little test to do this, it took about 5 trials to work out the proper syntax for the argv strings, but when I added this code to the top of the test file,

args = []
if sys.platform == 'linux' :
    args = ['','-style','Cleanlooks']
app = QApplication(args)

Then it came up nice and clean, no problems. Interesting, I also tried the 'Plastique' style which is supposed to be different, but it wasn't. In fact both looked exactly the same and exactly like the default Ubuntu look. But specifying any of them eliminated this horrible graphic loop catastrophe.

Ubuntu #3

So I put that code at the top of PPQT2.py and ran it, and now it won't start up because it is not initializing fonts. Some problem with QFont() arguments. It did not do this the first time it started up, which tells me the problem now must be because now it has a QSettings to read and what it is getting from there is not right.

I am now going for that walk & coffee and to hell with this.

A darn good day's work

Monday and Tuesday I implemented the production of sort vectors in worddata, basically recreating what I had done in my table sorting testbed. Today I made the many detail changes in wordview to remove its QSortFilterProxyModel and perform all the sorting and filtering with my own code. Many many little changes. But now it is working nicely. When I opened a 1.5MB text file that uses over 13,000 unique words, it took just over 2 seconds to do the "Refresh" operation. That includes scanning the whole file and counting and categorizing the words, plus doing the initial ascending sort on column 0. Subsequent sort actions that need a new vector, that is, clicking on the head of column 2 or column 3, take about a second. Sort actions that can re-use a cached vector, for example going back to ascending sort on column 0 after sorting on a different column or different order, are effectively instantaneous.

In the course of this I added support for the Home and End keys, so I could quickly pop to the top or bottom of the table.

I also found an old bug, one that probably is present in version 1 as well. Part of the Refresh is to note the properties of a word-token. The logic went something like this:

    if the word contains an apostrophe,
        note the AP property
        strip out the apostrophe(s)
    if the word contains a hyphen,
        note the HY property
        strip out the hyphen(s)
    if not word.isalpha() : # word contains some digits
        note the ND property
    ...

The problem here is that I was assuming that a return of False from the Python str.isalpha() string method meant there were digits in the word. Not so; it means there is some character in the word that does not have the Unicode Letter property. Well, I'd removed hyphens and apostrophes; the only non-Letters would be digits, no?

No! I was forgetting the DP convention of representing non-ASCII characters with bracket notation, for example [~n] for ñ or [c,] for ç. So a legitimate word could contain several non-letters and still not be numeric. I had to change the logic to look specifically for digit characters.

I also ran into an unexpected problem with the natsort package, which I posed as an issue at the package's github site, and was pleased when the maintainer got back to me in less than an hour. I need to up my support game. I've had this kind of super-responsive support before from amateur maintainers and it is so satisfying. I must try to remember that and do as well.

Anyway, next up is upgrading my laptop to Yosemite level, I've been putting that off, and to upgrade my desktop system to [Py]Qt5.4.1. And then, one more try to see if Nuitka can package PPQT. Next week will be devoted to packaging up the alpha level of PPQT2 for Mac, Linux and Windows -- using Nuitka if possible, else PyInstaller.