To eat one's own dogfood...

While waiting for a response from the chap I hope will join me in making PyInstaller 3, I decided it was time to do a bit of a test of PPQT2 usability. That means, post-processing a book myself. So I downloaded an example of the kind of book I have enjoyed doing in the past, in this case, one volume of Hawkin's Guide to Electricity.

I also opened the "Suggested Workflow" document that gets distributed as an "extra". It outlines the steps of post-processing a book with specific guidance on using PPQT features at each step. I knew it needed updating for V2, and I'm keeping it open alongside the PPQT window (27-inch iMac heh heh heh) and referring to it as a very handy task list. And correcting it as I go.

One of the first things I ran into is the small usability problem posed by V2's policy on file encoding. In V1 I really tried to support every common file encoding. There was a File:Open With Encoding sub-menu that allowed opening Latin-1, Mac Roman, Windows CP 1252, and UTF-8. There was sneaky undocumented support for other encodings.

For V2 I decided, basta, the world has moved on, UTF-8 is it. So PPQT2 assumes any input file, the book text and the good_words and bad_words files, are going to be UTF-8. It will also accept ISO-8859-1 aka Latin-1 but there's no menu command for that; you have to tell it by renaming the files to, e.g. bookname-ltn.txt or good_words.ltn. Similarly for saving: if you save the file to a name ending in -ltn or .ltn, it will write ISO-8859-1. Otherwise, it writes UTF-8. (Obviously if you know the file should be pure ASCII, it doesn't matter on input or output.)

So Bibimbop, my one tester, hit that right away, complaining that the good-words list was full of � replacement characters. Right, that's what happpens when Latin-1 is read through the UTF-8 codec. But then I did the same thing to myself! The book I downloaded was, as almost every book from Distributed Proofreaders U.S. will be, Latin-1 with a filename of projectidxxxxxxxx.txt. I opened it without renaming it, and it had replacement chars.

This was a good thing! Because I immediately realized how the user should handle this, and I put the following text into the Suggested Workflow as "Step 2: First Open".

Start PPQT2. Use File:Open to open the book. Its text appears in the Edit panel.

Select the Chars panel and click Refresh. Click the Symbol column heading to make the table sort from high numbers to low (if it is not that way already). Note if the highest-numbered symbol is � (0xfffd, the replacement character).

If the book has any of these, it was opened with the wrong encoding. Close the book immediately without saving! Perhaps it is a Latin-1 file opened as UTF-8, or it has some other encoding such as Windows CP-1252. Find out how it is encoded. If it is Latin-1, rename the file appropriately. Otherwise, use some other utility to convert it to UTF-8. Then start this step over.

On the whole, this seems a bit squiffy to me. On one hand, I definitely do not want to support other encodings. People should use UTF-8, period. Also on that hand, this is a one-time thing; it only affects the user the very first time they open a book. Once they have resolved the encoding issue for this, it is not an issue again. (Well, unless they insist on Latin-1 and enter characters that doesn't support, but that's not my problem, and the Chars tab has a tool for finding those.)

On the other hand, it lays a bit of a trap for the user, a wee pitfall that I even fell into myself. And there's potential data loss. If you open a Latin-1 book as UTF-8, which PPQT2 will happily do, and then save it, you have just trashed your file. You've written a UTF-8 file full of replacement chars where you used to have nice accented characters.

So what to do? The app knows very well when a book is being opened for the first time (there's no metadata file). Should I put up some kind of warning dialog? Should I quickly run through the text looking for \ufffd chars and then put up a warning? Maybe that.

This is what dogfood snacks are all about.

Backgrading? Nunh-unh!

Over the weekend I thought frequently about the idea of "back-grading" PPQT2 to run under Python 2.7. Finally, I decided not to do it.

First of all, there would be a tedious amount of clerical work. I'd have to insert a from __future__ import... statement in almost every module. Then I'd have to edit every single super() call to insert the classname and self parameters. (I pondered quite a while in the middle of the night last night, whether there was some way to create a global function super23() that, when called from inside some class method, could check whether it was in Python 2 or 3, and if 2, introspect its own parent classname, and dynamically make the Python 2-style call. Decided that if the Six compatibility module didn't offer something like that, it couldn't be easy to do.)

All that would be at least a couple day's work and for what? I would not want to keep the code in Python 2. I would not want to add fixes or enhancements to it, unless I simultaneously added them to the master branch. It would be work done for no benefit and for only a temporary use.

Then I thought of two more things with serious implications. The first was the SortedDict class, upon which several modules are heavily dependent. I make great use of the valuesView and keysView it supplies, and assume that these produce lightweight access paths to existing values, not massive duplications of the dictionary values and keys. It is not clear from the SortedContainers doc whether in Python 2, the "views" are even available, or whether they work as efficiently.

But the major issue is Unicode strings. All through PPQT2 I am freely interchanging strings between Qt and Python and assuming that, one, this is done with minimal overhead, just straight memory copy because the character encoding is almost the same in both universes, and two, doesn't need explicit encoding and decoding by me, again because they both just use Unicode.

That is not the case in Python 2, where even with "from __future__ import unicode_literals", I can't be positive that when I just access the entire content of a QPlainTextEdit using the toPlainText() method, that I'll get the exact contents, special characters and all. If it works at all, it has to entail an expensive conversion from UTF-16 to UTF-8. And when I take the user-entered Find text string and stuff it back into a document using a QTextCursor, how do I need to encode that, or do I?

I realized it had been a great stress relief in version 2 to just not worry about these things because Python 3 and Qt use virtually the same Unicode string contents. If I back-grade to Python 2, I could not be at all sure of identifying all the places where attention must be paid to these potential conversions. It would not be like doing a search for the next use of "super".

So, spending the time to try to make PPQT2 work under Python 2.7, only in order to be able to bundle it with PyInstaller's current Develop branch, is not a good idea.

But if not that, what? Well, I think what I need to do is to donate my efforts to getting PyInstaller fixed for Python 3. I'm putting out some email feelers to see if I can get anybody else to help in this effort. In the meantime I will continue working on enhancements to PPQT2.

The Py2.7 experiment

So I created a virtual environment with Python 2.7, and in it installed Qt and Sip and PyQt, and brought in a little test program to bundle. Pointless and ugly little thing but it runs and makes an app and a dialog box.

from __future__ import print_function
from __future__ import unicode_literals
from future_builtins import *

from PyQt5.QtWidgets import (
 QDialog,
 QMainWindow,
 QPlainTextEdit,
 QVBoxLayout,
 QHBoxLayout,
 QPushButton
 )
from PyQt5.QtWidgets import QApplication
import sys
args = []
if sys.platform == 'linux' :
 args = ['','-style','Cleanlooks']
app = QApplication(args)
mw = QMainWindow()
mw.show()
dlg = QDialog(mw)
hb = QHBoxLayout()
okb = QPushButton("OK")
okb.setDefault(True)
okb.clicked.connect(dlg.accept)
cab = QPushButton("Cancel")
cab.setDefault(False)
cab.clicked.connect(dlg.reject)
hb.addWidget(cab)
hb.addStretch()
hb.addWidget(okb)
vb = QVBoxLayout()
pte = QPlainTextEdit()
vb.addWidget(pte,1)
vb.addLayout(hb,0)
dlg.setLayout(vb)
print(dlg.exec_())
print(pte.document().toPlainText())

So this runs under Python 2.7. So I installed PyInstaller and bundled it. This failed because of an issue with the PyInstaller "hook" for PyQt5.Qt, which stupidly insists on including every module of Qt including Qml, which I did not install. But I had already fixed that in my copy of PyInstaller for Python3, and I knew just which file to edit, and did, and the resulting bundle now ran.

Then I ran the same job with the option to make a mac .app bundle and that worked too, even giving it the right icon. So, looking good.

I also installed py2app and tried it but first I hit an obvious programming error. It was an easily-diagnosed misspelling of the name of a class method that I had not run into before because it is in code specific to running in a virtual environment, and this was the first time I'd done that with py2app. After I fixed that (and opened a second issue on Oussoren's bitbucket page), it ran and made a bundle, but said bundle died immediately with an abort trap 6. No idea why, although I did notice that the qt.conf file py2app had left in the app Resources folder contained a path that could not possibly be correct. So I opened a third issue on the py2app site, but after I did that I mentally washed my hands of py2app. As Kevin O'Leary says on Shark Tank, "you are dead to me."

So how hard would it be to convert a Python3 app to Python2? I made a copy of my Cobro program and did the job. There were three issues. First, it used the Python3 urllib. Which is the Python2 urllib2 package. So I had to change two import statements and several statements that used urllib. The changes were only syntactic; just different names for the same operations.

And I had to change every call to super() to include the parent class name; again, only syntactic. Finally, running it from the command line it crashed trying to write the QSettings file. There is a bytes value which I passed to QByteArray, and for some reason Python seemed to think it needed to convert that to a string first, which didn't work, no surprise. All I had to do was wrap the expression in bytes().

Unfortunately the program crashed after it had cleared the settings, which meant that when it next came up, my long and carefully-curated list of webcomics that I read every day—was gone! No prob, I thought, I'll just restore from Time Machine. Nope! Because apparently years ago when first setting up Time Machine I had excluded ~/Library from the list of folders to back up. So later I will have to reconstruct my list of web comics.

The good news is that PyInstaller bundled cobro, although the resulting .app does not run; it dies looking for QWebEngineProcess, the undocumented piece of QWebEngine that has to exist but they don't tell you that, or where to put it. So not an error in PyInstaller, it's just that nobody has contributed a "hook" for QWebEngine. If I can figure out where it's supposed to be...

Never mind. What next? Do I really want to fork PPQT2 and make a Python2 version of it? It looks as if I'd get something that I could bundle. But it would be a chancy thing. I'm worried about having bugs like that string-handling thing in Cobro. To be really sure I ought to perform the same experiment in Ubuntu but you know, why? I'm quite confident it would work; PyInstaller was working with Python2 for PPQT version 1, in Ubuntu and Windows. I will ponder this over the weekend.

In which we learn I am full of it

A few days back I complained about QWebHistory, and how it didn't record clicks on local anchor links. I cannot figure out now why I said that. Because now I test it some more, I find that in fact, it does. I thought I had a solid case showing that it did not, but now... I don't know. I may be losing it, folks.

Anyway I coded back and forward key functions for the help viewer and they work just fine by calling the history's back() and forward() methods without any additional help from my code.

Further proving that I'm incompetent, I once again studied about rebase and merging in Scott Chacon's Pro Git, and also downloaded Github for Mac, because why be confused at the command line when you can be confused in a GUI? Anyway, then I set out to see if I could possibly merge the Python3 branch of Pyinstaller into the Develop branch.

Answer: probably not. Because if I use Github for Mac and start that merge, it shows me a list of 350 changes, the majority of which are shown with yellow status (conflicts to resolve) and many with red status (can't merge). And these are all over the many files of the project. So, no. So PyInstaller is definitely out for now -- and appears likely to never have Python3 support, at least for a long time, which makes me sad.

Meanwhile my request for advice to the cxfreeze mailing list has produced no responses of any kind. And R. Oussoren continues to not respond in any way to my very detailed debugging info on py2app.

I would like to just rant here about the absolutely abysmal state of Python3 distribution tools, but why bother? It is simply the case that it is not possible with Python3 and PyQt, to create a stand-alone, self-contained application, on any platform, period. If I had not elected to start PPQT2 in Python3, but just stayed with 2.7, I would be able to use PyInstaller at this point. But good grief, who would have thought 15 months ago, that there would be no Python3 bundlers working in 2015?

I am debating what to do next. One option is to just keep working, finish the program (writing the Translator interface that I've got a detailed spec for, and the two Translators I mean to supply) and just hope that during the 2 months that might take, somebody will get a bundler working.

Or, I could go on eLance or the like and try to hire somebody who could make pyqtdeploy work on all three platforms. But that could run into serious money. Based on my experiments, pyqtdeploy is a badly documented, flaky, and incomplete tool. Certainly making it work would require in-depth knowledge of not only Python and PyQt but QMake, Make, and the C++ interface between Python and Qt. It would be hard finding an eLancer with those skills in the first place.

Or, I've seriously thought about chucking the whole project. Who needs it?

Edit: So I took a walk and thought about what I'd written above, and one thing that stood out is the possibility that if I had written V2 in Python 2.7, stuff might be working, in particular PyInstaller and Py2app. So, how hard would it be to retrograde the code to Python 2 syntax? Probably not awfully hard, all the super() calls and probably a few subtle things I used without realizing they were 3-level syntax. But would it really help? So I decide what I will do tomorrow is to set up a virtual environment based on 2.7, with PyQt5 and all the other packages. In that VENV I will try bundling a couple of small test cases I have, not the whole app but just something that makes a main window with a widget in it. I'll have to do that at least on Mac OS and on Ubuntu, so several hours of work potentially, but at the end I'll know if there really exists a smooth path to bundling in the older language. And if so, then back-converting the whole app shouldn't take but a few days.

So at least that's something positive to try before I totally crumble.

Spinning the old wheels

First off today, I spent some time nailing down what is happening with the cx_freeze bundle in Linux. Then I wrote what I hope is a detailed and constructive note to the cxfreeze list,

I am trying to freeze an app on Ubuntu 14.10. The app uses PyQt5 and runs under Python3.4

Cxfreeze creates a bundle into which it copies all the dependent libraries,
for example libQt5Core.so.5, libQt5Widgets.so.5, and so forth. It also
copies the PyQt libs such as PyQt5.QtCore.so, PyQt5.QtWidgets.so, etc.

When I move this bundle to another Ubuntu system where this version of Python
and Qt are not installed, I get the dreaded message,

      Cannot mix incompatible Qt library (version 0x50300) with this library (version 0x50401)
      Aborted (core dumped)


I think it is trying to load from the local Qt (Ubuntu has some version
of Qt installed by default). On that system, after copying the bundle
from the dev system, I see this:

    $ cd Desktop/PPQT2 # the bundle made by cxfreeze

    $ ldd libQt5Widgets.so.5
        linux-gate.so.1 =>  (0xb779c000)
        libQt5Gui.so.5 => /home/dcortesi/Desktop/PPQT2/./libQt5Gui.so.5 (0xb6b49000)
        libQt5Core.so.5 => /home/dcortesi/Desktop/PPQT2/./libQt5Core.so.5 (0xb6605000) 

       ...


That is good! libQt5Widgets is linking to the bundled copy of libQt5Gui and so on.

However:

    $ ldd PyQt5.QtWidgets.so
        linux-gate.so.1 =>  (0xb7776000)
        libQt5Widgets.so.5 => /usr/lib/i386-linux-gnu/libQt5Widgets.so.5 (0xb6afb000)


That is bad, PyQt5.QtWidgets is linking to the local native
libQt5Widgets -- NOT the one bundled right there alongside it.

Why is this? Am I doing something wrong? Or is something needed in the hooks for PyQt5?

Thanks for any help,

Until I get some help with that, I have no way to create a bundle on Linux.

Nor on MacOS either! I took the cx_freeze MacOS bundle to my wife's iMac where there's no Qt or Python and of course it didn't run; it dies looking for a Python in /usr/lib, I believe — I forget and I've deleted the evidence now. Anyway, something that should be in the bundle, probably is in the bundle, but that isn't where the app goes looking for it. Basically the same issue as on Linux.

I would really, really rather use py2app for MacOS anyway, as it almost makes a nice looking MacOS app object. But not quite. So I started trying to debug that code and spent a couple of hours on that. You want to see a nice detailed "issue"? Check this out. (edit: that's weird, there should be 5 comments on that, not 2, and the last 3 are very detailed debugging info.) No response from R. Oussoren yet.

It's an infuriating bug because it does not fail when run under the debugger. It only fails when running at speed from the command line.

Well, so cx_freeze is failing for MacOS and for Linux. And py2app is failing for MacOS. I haven't gotten around to trying it yet, but I don't have a lot of hope for py2exe on Windows. It claims now to support Python3, which is good, but its writeup doesn't say anything about Qt, which is ominous.

Back in V1 days, I didn't worry about all these other bundlers; I just used PyInstaller. I liked it so much I rewrote the manual for it, and I put in several days work doing the initial coding to move it to Python3 and start using Oussoren's ModuleGraph. That was more than a year ago.

Recently I got it kind of working on MacOS. But when I tried it on Linux I ran into problems that, it turns out, had already been fixed on the Python2, or main, branch. There's been quite a bit of maintenance applied in the year-plus since Python3 split off. So I wrote to one of the maintainers, and this was the reply,

> It appears my problem is trying to use the Python3 branch, and this
is missing maintenance since I don't know when, a year?

Yes, we are lagging behind a lot. development-branch advanced and 
modulegraph was merged with python3 somehow. This needs to be cleaned
up and merged/rebased - but this is a lot of work nobody has started yet.

So basically, I'm screwed. PyInstaller for Python3 is just unusable until someone does the job of merging all commits since early 2014, into the Python3 branch. I have no idea what-so-ever how to do that. None. And the people who presumably do have a clue, are too busy.

Things are not looking good on the bundling front, frankly. And without a reliable way of making standalone, independent bundles on all three platforms, I don't have a product.

Thursday I will hack away at the Windows installation, installing the needed visual C, and then py2exe, and maybe that'll work. Maybe the cx_freeze crew will have a suggestion that will fix it on both MacOS and Linux. (and maybe... never mind)

What I can and can't control

Started today with things I can control, namely, issues that affect my code. I fixed a minor issue where the numbers in the Footnotes table weren't vertically aligned same as the text items. I fixed the issue where the Bookloupe table produced a bunch of console messages when first refreshed. I closed two issues that had been rendered moot by an enhancement to the natsort package. I fixed the issue about the Help viewer not having any "back" facility.

Implementing "Back" on the Help

That was interesting. It brought me back to familiarity with the limitations of QWebPage and QWebHistory. When you first load a web page from a file, its browser history is empty. And it stays empty during any number of link-following clicks, if those clicks are only to anchors within the current page. It just doesn't record local links. Only if you click away and make it load an external URL does it make a history entry.

Well, what does a user do with a Help page, which is quite a complete little manual with a TOC on one side? Clicks around to local links, of course. Rarely exiting the page to another page. So for most of what the user does, there is no history, and self.view.history().canGoBack() returns False.

So I had to implement my own history stack. This turned out to not be difficult. There is a facility for "delegating" all links. When this is done, the user's clicking on a link only creates a signal. The slot for the signal gets a copy of the QUrl which is the target. So I just got the string value, and if it began with "file:" then I assumed it was local and appended it to a list of strings. Then handed it on to the QWebView.load().

When the user hits one of the keys ctl-[, ctl-b, or ctl-left, my keyEvent code asks first if self.view.history().canGoBack(). If so, call self.view.back(); done. Else, is the list of link strings nonempty? If so, pop off the last one, make it a QUrl, and call self.view.load(). If the list is empty, call the subroutine that reloads the help file; that puts the view back to the top of the page.

Bundling for Linux

Then, feeling rather full of myself, I ran cxfreeze to bundle the newly updated app, and zipped that up and put it on my Public Dropbox.

Then I fired up my "test" system which has none of Python or Qt installed. Well, turns out that a stock Ubuntu 14 does have Qt installed. I kind of knew that, because I kind of knew that some of the Ubuntu programs (Software Updater, I think) are Qt-based. Anyway the test is, does this stock system run the bundled app.

Would you like to guess?

No, of course it bloody doesn't. "Cannot mix incompatible Qt library (version 0x50300) with this library (version 0x50401) Aborted (core dumped)"

Sigh.

If you search on that string you will find a whole lot of people running a wide variety of software over the past 3 years who hit that exact message. What it means is, the PyQt5.QtCore library is trying to load libQt5Core.so, and it is getting one, but from the local system instead of from the bundle-folder.

But Dave, you say, I thought that you were all happy with cxfreeze because it was bundling everything, all the dependencies.

Well, it appears I was deceived. It is not bundling the Qt5 dependencies. It is treating them like libc.so, to be obtained locally.

I quickly trained myself on ltrace and ldd, the Linux equivalents of tools I was using on MacOS to figure out what nuitka wasn't bundling. Just as soon as the program calls into PyQt5.QtWidgets to create a QApplication, boom. If I do ldd on PyQt5.QtWidgets, I can see the links to /usr/lib for the libQt* libraries.

Edit: I was mistaken. The desired libraries, e.g. libQt5Core.so.5, are in fact there in the bundle. They are being copied correctly. However, the links revealed by, for example, "ldd PyQt5.QtCore.so" are not pointing at the files in the bundle. They are pointing at /usr/lib. The problem is not what is being copied, it is the link info in the PyQt5.Q* modules.

The link info in the copied libraries, e.g. the links from "ldd libQt5Core.so.5", have been altered to point into the bundle. So it is a matter of pointing the links in the PyQt modules as well. I now have to go and find out how cxfreeze does that, when it does it, and why it doesn't do it in all cases.

Opening the Window (7)

Earlier this week on my Windows 7 VM I downloaded and installed the official Python 3.4.3 distribution (as well as Qt5.4, Sip, and PyQt5.4) and thought I had pretty well got my Python environment set up. I could run python from the DOS command line, and WingIDE found it and all.

However, the first time I tried to run PPQT under Wing, it immediately reminded me that it couldn't find hunspell. Doh! of course, I need to install the four packages PPQT depends on: regex, sortedcontainers, natsort and hunspell. Well, that shouldn't be hard, just pip-pip-pip.

Well, of course not. First off, on my Unix-type boxes I'm used to pip being a command, a script in /usr/bin. Not Windows. There's no pip command to be found anywhere on the path. So I search the internet. Ah, in Windows one uses the command Python -m pip. All right, I've gotten used to ending a Python session with ctl-z, return; I can get used to this.

I do that—and there is no module named 'pip'.

Search around some more. According to what I see, pip is definitely supposed to be automatically included with Python3. But not mine! Oh, alright then, I go to pypi and download the pip package, which is of course a tar.gz file. Which Windows does not know how to open! It only knows how to open ordinary zips.

Back to my search window. How to open a tar.gz file on Windows? Top recommendation, install 7Zip. Which I do. And now I can unpack the pip distribution file.

So, cd into the folder and there is the usual setup.py. OK, I know what to do: "python setup.py install". No, of course not. Because that tries to import setup tools, and that doesn't exist in my nice new Python 3.4 installation.

Back to pypi and download ez_setup. Run that, and I now have setup tools. Now I can run pip's setup.py. And I've got the pip.

So install natsort and test it. Good. Install regex, test. Good. Install sortedcontainers, test. Good.

Take deep breath. python -m pip install hunspell. Nope. "error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat)". Well, kind of expected. So I need to get that. I'm sure it's available. But it's nearly 6pm Sunday and it will just have to wait for... probably Thursday, at a guess.