Friday, January 30, 2015

Added "Book Facts", exploring PyQtDeploy

Turning away from the frustrations of bundling, I coded a feature of PPQT. Driving along a few days ago on a routine errand, the thought of book metadata came to me. Both fp and fpgen, as well as MarkDown and AsciiDoc and others, have support for "title", "author", "publication date" in some form or other. PPQT, I realized, should offer some method of entering and editing such data.

There's no standard for the format of these, but they all amount to a set of key:value pairs, in short, a Python (or JSON) dict, like {"Title":"The Telltale Hare", "Author":"Bugs Bunny", "Published":1931} and so on.

What to call them? Please, not "metadata", not where the user can see it. That term is already being overused. These are simply facts about the book.

Because the book-facts are unique to the book (duh!) they need to be maintained by the Book object. Because they need to be persistent from session to session, they need to be stored in the metadata. So it was clear the Book should have the code to keep the dict of book-facts, and the code to read it from the metadata file and save it to the metadata file. That code took about 20 minutes to write and implement.

How to present these data to the user for editing? While driving along that afternoon, I realized that it was a dict, so present it as a dict (but not called that). Just give the user a dialog box with some text lines in it, in the form "keystring :valuestring", basically a dump of the dict.

Since the user might want to add or delete lines, it would be a multiline display, and the easiest way to do that is to present a QPlainTextEdit.

When, and by what UI? Well, the Edit view already has a context menu. It currently had four choices: toggles for highlight spelling and highlight scannos, and commands to choose a scanno file and choose a spelling dictionary. These are book-level choices, and early on I decided that book-level choices (as opposed to application-level choices) would be in a context menu, not in the global menu bar.

Editing book facts is another book-level action. So I just added that as a fifth command in the context menu: Edit Book Facts... The code is in the Book, but I put the nuts and bolts of building the QDialog in utilities where all the other dialog boxes are confined. Here's the context menu in action.

Here's the dialog.

Preparing the text for display is just this simple,

        for (key, arg) in self.book_facts.items() :
            starting_text += '{} : {}\n'.format(key,arg)

And storing it after the user has clicked Ok is like this:

                    try :
                        (key, arg) = line.split(':') # exception if not exactly 1 colon
                        self.book_facts[key.strip()] = arg.strip()

So that all went in very nicely.


Then I turned to trying out pyqtdeploy. Based on its tutorial overview it does just what a bundler would want: it converts all the messy parts of a Python/PyQt5 application into a single (no doubt huge) binary executable. I started trying to set it up to bundle Cobro without actually reading the more than the first few pages of doc. And naturally ran into some issues. But it looks promising. I hope I can get it to work; I would be so pleased to be able to forget all about cxfreeze, pyinstaller, even Nuitka. (Well, I might keep Nuitka around but use it to compile sub-modules only. PyQtDeploy should be able to bundle compiled modname.so files...?)

Thursday, January 29, 2015

Gettin' fed up with bundlers...

Real Life has been intruding heavily on my coding adventures, but Tuesday and today I got back to trying to solve the issues involved in bundling Cobro, or indeed, any PyQt5/Python3 app.

Here's the status of each.

PyInstaller. I put a shit-ton of time into research on why the bundled app could not find its platform plugin "cocoa". When I finally posted two very detailed notes to the PyInstaller list, another user pointed to a run-time hook (PyInstaller supports run-time hook code that is called by the boot-loader before one's actual program is called) that disables all the five (yes, five) different ways that Qt looks for platform plugins in favor of a completely undocumented folder qt5_plugins. Supposedly PyInstaller should build this automatically and put the relevant plugin DLL into it. It doesn't. When I created it manually, the app started up. Or tried to. It immediately died with a traceback, trying to read the saved settings.

That was an old problem, one I'd struggled with under Nuitka and never solved because it just went away after some change in Nuitka. Now I tackled it again, building a beautifully minimal test case and showing that while the code worked run from the command line, it failed when executed out of a bundle created by either cxfreeze or PyInstaller.

Wrote a highly detailed and awesomely informative post for the PyQt list...

...and sent it to the PyInstaller list.

Fuck. There are few things that can make a person writhe with embarrassment like sending a message meant for one product to the list for another product.

When I realized that the next day, I sent it also to the PyQt list, and next day one user replied with The Answer. Instead of writing

settings.setValue('b',b'\xde\xad\xbe\xef')

instead, use an explicit coercion,

settings.setValue('b', QByteArray(b'\xde\xad\xbe\xef'))

The settings plist that PyQt writes in the two cases is entirely different! And is the same whether the writer is bundled or not, and is accepted by the reader whether bundled or not.

I had supposed PyQt5 would do exactly that coercion automatically, just as it automatically converts between Python strings and QStrings. Apparently not. What it does with raw byte data is (based on my experiences with Nuitka) to use the pickle module. But I don't care; that problem was solved.

So I made that one-line change to Cobro, removed its settings plist and made it write a new one. Then I bundled it with PyInstaller. It still doesn't automatically supply the platform plugin, so I put that in manually; then the "you might be loading two sets of Qt binaries" thing came up; so then I used install_name_tool to manually relativize the links in the platform plugin. (I have become way more familiar with install_name_tool and otool the past weeks than I ever wanted to be.)

Then it couldn't start because QtWebEngine "couldn't find QtWebEngineProcess". WTF? Well, it seems that QtWebEngine requires a "helper" named QtWebEngineProcess which is not only a DLL but a whole goddamn Mac OS app. Documented? I think not! After copying both the DLL and the app folder into the cobro folder, the app still wouldn't start. Now it got into QWebEngine and that died because "Couldn't mmap icudtl.dat".

What?

Well, that's some file that the Chromium browser uses, ghu alone knows what it is or why. I found a copy inside the Chromium app folder in /Applications and copied it into the cobro folder. Now it starts up (but note, if I try to start it up from a different CWD, it again "can't mmap icudtl.dat" so apparently it has to be in the CWD?!?!)

Now the PyInstaller bundled app starts but again on the console is the message "dyld: Library not loaded: @executable_path/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess; Referenced from: /Users/dcortesi/Desktop/scratch/piwork/dist/cobro/QtWebEngineProcess"—and although the app is kinda started, the whole web view is blank and it won't respond to clicks.

Giving up on PyInstaller, I turn to...

cxfreeze. Bundle Cobro afresh. Run it. Much better results: cxfreeze has apparently got the cocoa plugin in the right place and all, there's no QtWebEngineProcess malarky, and the app appears to be running. All the widgets are there; it read the list of comics from settings and is refreshing them; it displays comics. Yeah!

Except... it has no menu bar. It has a dock icon of a generic unix app. If I click that dock icon, then it gets its proper menu bar. When it starts up, no menu bar. Click in another app, then click on Cobro's window, now it has a menu bar.

Well, that's a lot better than PyInstaller but still not acceptable. Which brings us to...

Nuitka: Kay's latest version still doesn't properly relative something, compiling Cobro and running the resulting cobro.exe produces "On Mac OS X, you might be loading two sets of Qt binaries into the same process." and the app comes partway up, but the web engine panel is blank and it has no menu bar.

So, you know... not a good day.

Sunday, January 25, 2015

Progress in bits n pieces

Saturday I finished up adding a flexible Edit menu to the Words panel. It is really two panels in one, the Word table that lists the vocabulary, and beside it, a good-words list. When the focus is in the Word table, the Edit menu has the choices Copy, Find, and Find Next. When you slide right and click on the good-words list, the Edit menu has the choices Copy, Paste, and Delete. These represent new function compared to V1.

  • In the Word table you can now Edit>Copy (or control-c) the selected words (the table allows complex selections using shift-click and control-click). All selected words go to the clipboard as a space-separated list. In V1 you could do this with control-shift-click; the Edit menu choice had no effect because it was bound to the editor.
  • In the Word table, the Find and Find-Next are new features, a simple search for a word. The found word is scrolled to the middle of the window. The Word table can be quite large and this is handy.
  • In the good-words list, Edit>Copy/^c copies words selected in that list to the clipboard.
  • In the good-words list, Edit>Paste/^v adds any words on the clipboard to the list, which automatically makes them correctly-spelled in the Word table. This function was already there as drag-drop from the Word table so it was easy to implement.
  • In the good-words list, Edit>Delete removes selected words from the list, which automatically makes the Word table re-check their spelling. This function was already there for the Delete/Backspace key, so again it was easy to implement

This all went in very simply and cleanly, which I take as proof I've built a pretty decent software structure.

Saving me from some kind of shoulder injury from patting myself on the back, I noted in testing these features that the Refresh operation was taking a very long time. The test book has only 1,400 unique words (in a total of 4,500). When you open the book, the Word table is empty. You click Refresh and the program does these things:

  1. Tells the Qt table model beginResetModel().
  2. Runs through the entire text (25K in this case) using a regex to isolate word-like tokens, spell-checking them, and adding them to an ordered dict. This takes about 0.1 second.
  3. Tells the Qt table endResetModel(). This causes Qt to rebuild its table by calling rowCount() then calling the data() method for values for all the columns (3) and rows (1400). Then, one presumes, it calls the sortFilterProxy which is needed to support sorting columns.

Step 3 takes just over seven seconds. Not good. I spent some time instrumenting parts of this so I have some numbers to compare as I make changes. But it isn't clear just where I should look to make improvements. Much of this process happens outside my control. Yes, it is calling into the table model data() function at least 3*1400 times (probably many more, there are several different "roles" it can request for each cell). But here's the thing. The seven seconds only happens the first time I click Refresh. If I click it again, which goes through the exact same process, it takes much less time, less than a second!

So something very lengthy happens the first time an empty table is populated, that does not happen when the table is re-populated. Such work has to be down in Qt-land out of reach of my code. I think. So I may just leave this issue open for now.

Meanwhile, I got an updated version of Nuitka. It does all the right things about relativizing the links within DLLs. However, the compiled app then crashes with a warning that "you might be loading two sets of Qt binaries into the same process". This is a fairly common problem, as a quick duckduckgo search reveals. I spent an hour this morning trying to find the problem, sending several emails to Kay as I went. In the end it looks as if the problem can be resolved by including a qt.conf file. But that remains to be tried out tomorrow.

Friday, January 23, 2015

Continuing to hammer on pyinstaller; Word panel

I spent the morning processing and printing pictures from a family collection so my wife can make display panels for a relative's funeral. But after lunch I was able to spend a couple of hours adding the customized Edit menu to the PPQT Words panel. I can wrap that up tomorrow.

For the, I don't know, fifth? day in a row, I read webcomics using Cobro in its WebEngine version. It's pretty smooth. All comics load; no messages of any kind to the console. Quite a pleasant experience. If I can find a way to bundle this as a self-contained app, it would be worth distributing.

With that in mind, and since nothing was new from Nuitka development, I resumed attempting to make PyInstaller work. In response to yesterday's query, I had one comment that libqcocoa.dylib should be included in the bundle. So I verified that it wasn't; then I took steps to make sure it was; then I wrote the following email, which I quote in full to show how thorough I'm being.

The file libqcocoa.dylib does exist in the Qt distribution.

Definitely the stock pyinstaller -w operation does not include it or anything else with a name like "*coco*" in the output bundle.

I modified the spec file to read as follows:

    # -*- mode: python -*-
    a = Analysis(['/Users/dcortesi/Dropbox/David/PPQT/CoBro/cobro.py'],
                 pathex=['/Users/dcortesi/Desktop/scratch/pyinst'],
                 hiddenimports=[],
                 hookspath=None,
                 runtime_hooks=None)
    pyz = PYZ(a.pure)

    exe = EXE(pyz,
              a.scripts,
              exclude_binaries=True,
              name='cobro',
              debug=False,
              strip=None,
              upx=True,
              console=False , icon='cobro.icns')

    a.binaries += [ ('cocoa', '/Developer/5.4/clang_64/plugins/platforms/libqcocoa.dylib', 'BINARY' ) ]
    a.binaries += [ ('libqcocoa', '/Developer/5.4/clang_64/plugins/platforms/libqcocoa.dylib', 'BINARY' ) ]
    a.binaries += [ ('libqcocoa.dylib', '/Developer/5.4/clang_64/plugins/platforms/libqcocoa.dylib', 'BINARY' ) ]

    coll = COLLECT(exe,
                   a.binaries,
                   a.zipfiles,
                   a.datas,
                   strip=None,
                   upx=True,
                   name='cobro')
    app = BUNDLE(coll,
                 name='cobro.app',
                 icon='cobro.icns')

When this is run, the desired lib does appear, in three copies under three names, in the cobro.app bundle:

    $ find dist/cobro.app -name '*coco*'
    dist/cobro.app/Contents/MacOS/cocoa
    dist/cobro.app/Contents/MacOS/libqcocoa
    dist/cobro.app/Contents/MacOS/libqcocoa.dylib

Nevertheless, when the app is executed it dies as before with the abort,

This application failed to start because it could not find or load the Qt platform plugin "cocoa".

I go into the "hooks" folder and execute cat hook-PyQt5* | grep cocoa and get no output, so it is not mentioned in a hook. (Nor in PyQt4*)

Any ideas welcome...

Thursday, January 22, 2015

Back on the job, debugging Nuitka and PyInstaller

Back from a rush trip to Seattle and trying to get one of PyInstaller or Nuitka to do something useful.

PyInstaller: I had reported quite incorrectly that the feature of creating a Mac OS app bundle was not working. In fact it is working just fine. What had happened was a rather subtle mistake on my part.

When you run pyinstaller many-options-here myscript.py, besides writing the bundle of code and libraries into the "dist" folder, it also writes a file myscript.spec that summarizes the possibly many options you used. So the next time you can just run pyinstaller myscript.spec and skip all the options.

So stupidly, when I wanted to attempt to make a proper Mac OS app bundle I just ran pyinstaller -w myscript.spec, passing the -w argument requesting a "windowed" app bundle, but giving it the spec file, not the original code file. So it happily ignored the -w argument and built the non-windowed bundle as before—and I reported that Mac OS app bundling wasn't working.

Realized this while I was traveling. So today I properly ran the command to make an app bundle and it did so exactly as it should. It even installed my icon file, so the app has the smiley-face icon I want! But also, the app when executed, terminates with SIGABRT because, just as before, it can't find module "cocoa". That was the original issue that was keeping me from using PyInstaller to bundle Cobro, and here it still is.

So I wrote a note to the pyinstaller list, apologizing for being wrong about the Mac OS thing, but continuing with detailed and I hope helpful info on the absence of anything called *oco* anywhere in the output bundle or any of the build files and logs it writes.

Nuitka: Where were we? Oh yeah. Nuitka built a working (yay!) compiled Cobro but the output folder includes, besides the cobro.exe, a whole bunch of the DLLs it depends on, and many of these have hard links to other DLLs. The Qt ones, for example, have links to other Qt DLLs in the form of hard links to /Developer/5.4/clang_64/lib/, the location where Qt is installed on this system. The PyQt5 modules have hard links to /Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/PyQt5. As a result, the compiled program can only execute on a system with those features installed at those locations.

So Kay Hayer, aware of this, updated Nuitka to try to "relativize" these links. This is done by calling the install_name_tool utility from XCode, using it to change hard prefixes to the relative one of @executable_path. (Note that PyInstaller changes them to @loader_path which amounts to the same thing.)

So I tested the new compiler version and it terminates with an assertion error because it is passing install_name_tool an invalid path. I have already been in that part of the compiler code, so I immediately went in and added a debugging print statement. And collected the output and sent that off to the nuitka mailing list.

And here it is 4pm and that's all I've managed to do, help debug two other packages. Meanwhile my own code mildews on the disk. Tomorrow I will by golly get something productive done.

Sunday, January 18, 2015

QWebEngine, Nuitka, Travel

QWebEngine: I mentioned that in my class derived from QWebEngineView, my overriding keyPressEvent() handler was not being entered. I wrote about that to the relevant mail list and got a reply. It's a design restriction. There is a known bug describing this. Apparently in order to shoe-horn the Chrome browser into the Qt framework they have had to play some tricks with the class heirarchy. So mouse events and key events end up in "a child widget" of the QWebEngineView and stop there, never being passed along.

This is one of a number of functional and design shortcomings of the QWebEngine package. As a browser it runs smoothly and without crashes or superfluous blather on stderr, and that keeps me using it in preference to my experience of the rickety QWebKit. But it really wasn't finished when released in 5.4. A year from now, around Qt 5.6, it will probably be a very nice and complete package.

Nuitka: I spent a happy 3 hours Saturday writing a program that would go through all the DLL's in a Nuitka "dist" folder and relativize them, that is, use install_name_tool to convert absolute paths to relative ones based on @executable_path. When I applied it, some strange things happened to the modified program and I didn't have time to debug it.

This afternoon I got email from Kay—who is about the ideal model of the open-source author, just amazingly responsive and willing to communicate—saying he had modified the compiler to relativize copied DLLs.

Unfortunately I can't do anything to try his code right away. We are off for a quick trip to Seattle. When I return on Thursday I will try it out.

Friday, January 16, 2015

Nuitka, pyinstaller, QWebEngineView issues

Nuitka: Nuitka does compile Cobro to an executable in a folder full of libs, and it runs just lovely. That's a win, but not a complete victory. First thing I did was to copy that folder to a USB stick and plug it into my laptop, which does not have the same level of libs on it. And it immediately failed trying to import Qt modules from the fixed path to the Qt distribution folder on the dev system. In other words, the copied libs have not been relativized.

There is code in the Nuitka compiler to do this. I could see it, and I inserted print statements (nice to have a compiler in source code one can hack on) to verify that it was being called, but only once, to relativize the links in the compiled cobro.exe. It also needs to be applied to every copied DLL. I wrote to the Nuitka dev list with this report. No reply as of today.

When that one thing is fixed, I will proceed to try to build a Mac App folder from it. Basically that's a folder with the .app suffix and some child folders like Frameworks, Resources, and Contents. Rather than try to educate myself on exactly what should be where in an app folder, I plan to pick a simple app, copy it, and basically replace its contents with the contents of the Nuitka output ("dist") folder.

PyInstaller: PyInstaller is capable of building a Mac App, and I tried doing that with Cobro. However, it ignored the options that should cause it to build one (--onefile --windowed). So maybe that feature isn't supported yet in the Python3 version? I wrote to its list to ask.

QWebEngineView: I did the really quite simple modification of Cobro to use QWebEngineView in place of QWebView. And ran it and... no messages! The QWebEngine browser does not produce any of the yucky incomprehensible stderr messages that QWebKit does. So that settles that; I shall have to use QWebEngine regardless of its shortcomings and missing features.

When I did this the first time I noted several missing features, and started a little string on the QWebEngine mailing list about them. But today I found yet another one. Although QWebEngineView claims to be a descendant of QWidget, it apparently does not support the keyPressEvent() handler. My QWebView-based widget had a keyPressEvent handler that worked quite well. In it, I trapped ctl-plus and ctl-minus to implement zooming, and ctl-[ and other keys for the "back" function. Same widget layout, only replacing the parent class, but control never enters my key event handler. So I wrote a query to the QWebEngine list asking about that.

I also had to change some of the code that sets up the Cobro fonts. It changes the font of the names of comics in the list to show their status, with bold for the ones that the user hasn't read yet. But suddenly, it didn't do that; NEWCOMIC status was not reflected in bold type. I proved to myself that the status was being set, and that the QListView was calling its list model's data() function for the Font role, and that the data() function was returning the QFont that had been prepared for the NEWCOMIC status as appropriate. But that wasn't bold.

The code couldn't be simpler. Given that qf is a QFont derived from the desired family (Comic Sans, of course), then:

FONTLIST[NEWCOMIC] = QFont(qf)
FONTLIST[NEWCOMIC].setWeight(QFont.Bold)

That used to work. Now it didn't. Just to verify everything else was working as expected I changed it to

FONTLIST[NEWCOMIC] = QFont(qf)
FONTLIST[NEWCOMIC].setUnderline(True)

Bingo, all unread comics showed up in the list with underlines. So the font was being set and given to the list view and applied. But setting the weight didn't make it bold. Well then,

FONTLIST[NEWCOMIC] = QFont(qf)
FONTLIST[NEWCOMIC].setBold(True)

Nope. No boldness. The only thing that worked was to derive a bold variant direct from the QFontDatabase, so:

FONTLIST[NEWCOMIC] = fdb.font(family,'BOLD',16)

That got a bold QFont that displayed as bold. I have no idea why that was necessary.

Next, pending developments from Nuitka, I return to PPQT2 and finish up the Word panel Edit menu.


After writing the above, I spent another 2 hours configuring my Ubuntu 64-bit development VM with Python3.4, Qt5.4, Sip and PyQt5.4. Installed everything separately from the Ubuntu /usr tree, in my home directory and using a VirtualEnv. This because an earlier attempt to upgrade the installed PyQt caused the Ubuntu "Software Center" to quit working. It apparently has dependencies on the installed Python2 and PyQt4; change those in any way and it fails to launch with no message. So this time I've sandboxed my dev environment.

At the end, it wasn't finding PyQt5 to import it. But my brain is fried and I'm quitting for the day.

Thursday, January 15, 2015

Nuitka compiling Cobro successfully

Well, a big sigh of relief: Kay Hayer found a problem in the way Nuitka bundled binaries, and with the new 0.5.9pre1 version, not only the "reader" test case but the full Cobro app compile and run correctly. This moves Nuitka up to the top of the candidate list as a way to make self-contained, portable distributions of Python apps.

Where are the other bundlers? Well, as noted in the prior posts, cx_Freeze makes a QSettings writer that writes data that a QSettings reader can't open. Cobro and PPQT both depend strongly on saving binary objects in settings, so cx_Freeze is out. I don't feel like pursuing this problem, although it is so odd, I might post a query on the PyQt mailing list to see if anyone has an idea about the peculiar behavior of writing strings in latin-1 instead of UTF-16.

As for pyinstaller, it produces a Cobro that fails immediately not finding libcocoa. I've written about this to the pyinstaller list and done some more forensics with my newfound skill in using otool, with no results. So pyinstaller is off the table.

Which leaves pyqtdeploy, but for the time being I will put off all thoughts of trying to ascend that learning curve, and focus on installing and running Nuitka on my 2 additional platforms, Ubuntu 14.10 and Windows 7.

Also I will downgrade the source of Cobro to use QWebEngine, to see if it is free of the annoying and uncurable console messages that QWebKit likes to put out.

If those two steps pan out, I will have a Cobro app that I could distribute with some confidence to the world of webcomic readers.

With respect to PPQT2, I want to finish a change to the words panel to give it an Edit menu of its own, and review my to-do list (and do a multi-file search for the string "TODO" as quite a few of those are scattered through the source). Then try to compile all of PPQT2 with Nuitka. That will be a major stress test; it takes about 3 minutes (on a 3.5GHz iMac) to compile the 1800-line Cobro. So all of PPQT will take 30-60 minutes, probably.

Tuesday, January 13, 2015

Clarifying one issue with cxfreeze

Continuing the prior post...

The cx_Freeze-bundled writer produced settings that neither the CPython- nor the cx_Frozen reader could read. Under CPython, the writer produced settings that the CPython-reader could read. Clearly the output of the two writers was different; in what way?

cx_Frozen writer settings plist:
0000: 62 70 6C 69 73 74 30 30 D3 01 02 03 04 05 06 51  bplist00.......Q
0010: 63 51 69 51 62 5A 63 68 61 72 61 63 74 65 72 73  cQiQbZcharacters
0020: 11 0F FF 5F 10 24 40 56 61 72 69 61 6E 74 28 00  ..._.$@Variant(.
0030: 00 00 7F 00 00 00 0E 50 79 51 74 5F 50 79 4F 62  ......PyQt_PyOb
0040: 6A 65 63 74 00 00 00 00 00 29 08 0F 11 13 15 20  ject.....)..... 
0050: 23 00 00 00 00 00 00 01 01 00 00 00 00 00 00 00  #...............
0060: 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0070: 4A                                               J
CPython writer settings plist:
0000: 62 70 6C 69 73 74 30 30 D3 01 02 03 04 05 06 51  bplist00.......Q
0010: 63 51 69 51 62 5A 63 68 61 72 61 63 74 65 72 73  cQiQbZcharacters
0020: 11 0F FF 6F 10 2F 00 40 00 56 00 61 00 72 00 69  ...o./.@.V.a.r.i
0030: 00 61 00 6E 00 74 00 28 00 00 00 00 00 00 00 7F  .a.n.t.(.......
0040: 00 00 00 00 00 00 00 0E 00 50 00 79 00 51 00 74  .........P.y.Q.t
0050: 00 5F 00 50 00 79 00 4F 00 62 00 6A 00 65 00 63  ._.P.y.O.b.j.e.c
0060: 00 74 00 00 00 00 00 00 00 00 00 0B 00 80 00 03  .t..............
0070: 00 43 00 04 00 DE 00 AD 00 BE 00 EF 00 71 00 00  .C...........q..
0080: 00 2E 00 29 08 0F 11 13 15 20 23 00 00 00 00 00  ...)..... #.....
0090: 00 01 01 00 00 00 00 00 00 00 07 00 00 00 00 00  ................
00A0: 00 00 00 00 00 00 00 00 00 00 84                 ...........

The writer program, when cx_Frozen, writes strings in Latin-1. The CPython writer, writes what I assume are UTF-16 character strings. At least, how else to explain "@Variant" and "PyQt_PyObject" being spaced out with \x00 bytes?

However, remember the behaviour.

[12:44:30 pyinst] rm /Users/dcortesi/Library/Preferences/com.bogosity.BOGUS.plist
[12:45:39 pyinst] writer.dist/writer
[12:45:43 pyinst] python reader.py c
key c value characters
[12:45:48 pyinst] python reader.py i
key i value 4095
[12:45:53 pyinst] python reader.py b
Traceback (most recent call last):
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

The reader is not bothered when fetching the "c" characters or the "i" integer. But! The three keys (QcQiQb) and the values for "c" ("characters") and for "i" (int 4095, visible at 00021-2) all precede the point where the UTF-16 strings begin. So the CPython reader execution can get those without trouble. It only runs into a problem when it has to decode what I assume is the header for a QVariant.

So: character encoding of the QSettings plist changes when the code is cx_Frozen. Does the Nuitka-compiled writer also do this?

Nuitka writer settings plist:
0000: 62 70 6C 69 73 74 30 30 D3 01 02 03 04 05 06 51  bplist00.......Q
0010: 63 51 69 51 62 5A 63 68 61 72 61 63 74 65 72 73  cQiQbZcharacters
0020: 11 0F FF 6F 10 2F 00 40 00 56 00 61 00 72 00 69  ...o./.@.V.a.r.i
0030: 00 61 00 6E 00 74 00 28 00 00 00 00 00 00 00 7F  .a.n.t.(.......
0040: 00 00 00 00 00 00 00 0E 00 50 00 79 00 51 00 74  .........P.y.Q.t
0050: 00 5F 00 50 00 79 00 4F 00 62 00 6A 00 65 00 63  ._.P.y.O.b.j.e.c
0060: 00 74 00 00 00 00 00 00 00 00 00 0B 00 80 00 03  .t..............
0070: 00 43 00 04 00 DE 00 AD 00 BE 00 EF 00 71 00 00  .C...........q..
0080: 00 2E 00 29 08 0F 11 13 15 20 23 00 00 00 00 00  ...)..... #.....
0090: 00 01 01 00 00 00 00 00 00 00 07 00 00 00 00 00  ................
00A0: 00 00 00 00 00 00 00 00 00 00 84                 ...........

No. The writer compiled by Nuitka produces the exact same bytes that the CPython writer produces. The Nuitka-compiled reader segfaults reading what I assume is the "correct" way to encode a settings plist.

Could it be that the Nuitka reader doesn't expect UTF-16?

[12:45:57 pyinst] rm /Users/dcortesi/Library/Preferences/com.bogosity.BOGUS.plist
[12:54:02 pyinst] writer.dist/writer
[12:54:06 pyinst] ../nk/reader.dist/reader.exe b
Traceback (most recent call last):
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

No! The Nuitka-compiled reader produces a reasonable exception when reading the not-UTF plist format.

So I see either two or three problems.

One, the writer code, when frozen by cx_Freeze, writes QSettings plists incorrectly, not using UTF-16 (which is probably just a dump of a Qt QString literal).

Two, the cx_Frozen reader gets an exception converting a QVariant no matter how encoded.

Three, the Nuitka-compiled reader segfaults converting a QVariant properly(?) encoded.

back to mulling....

cxfreeze and nuitka issues converge

Some days back I tried bundling Cobro with cx_Freeze and ran into a problem where the bundled program died trying to link libz from /opt/local/lib. This was baffling, as that path had never existed on this machine. I wrote to the cx_Freeze mailing list and nobody responded.

Yesterday, still trying to solve the baffling segfault in the Nuitka-compiled code, Kay had me use the dtruss trace tool to list the modules opened by the compiled program. This exposed an issue where the compiled program was loading QtCore from the Qt distribution folder. It should be loading it from the local "dist" folder where Nuitka has collected all the referenced modules, and where the compiled executable is placed.

In order to fix this I had to give myself a crash course in the use of otool and install_name_tool. The former is a utility that displays info from a DLL. The latter can be used to change some of the things encoded in a DLL. Specifically, changing a hard-coded link path to a relative one.

In the course of applying "otool -L" to the various modules, I happened to note in its output, that QtCore had an external link to, guess what, "/opt/local/lib/libz.1.so". So that's where that reference came from.

Today, after reaching yet another dead end on the Nuitka problem, I thought I'd fix that and see if cx_Freeze worked. So I went into the Qt distribution folder and tediously checked the links of every DLL. Well, not so tedious; because I wrote a very complicated shell script to find it:

#!/bin/bash
otool -L $1.framework/Versions/Current/$1 | grep "/opt"

Turns out, there were four or five modules that used it. QtWebKit had three links to things in "/opt/local/lib". It must be that I overlooked some install option that would have corrected this when I installed Qt5.4. I've no idea what, but I must look carefully when installing 5.5.

Anyway, with that done, I could now cx_Freeze Cobro and the frozen app ran. For just a little bit. Then it died with a Python error that made me say "aHA!":

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "cobro.py", line 1785, in 
  File "cobro.py", line 1429, in __init__
  File "cobro.py", line 964, in load
TypeError: unable to convert a QVariant back to a Python object

That is the exact point in the program where the Nuitka compilation dies with a segfault: the moment when PyQt is trying to fetch a byte string from a QSettings value. Here is the current Nuitka minimal test case:

from PyQt5.QtCore import QCoreApplication
import sys
app = QCoreApplication(sys.argv)
# This defines the key to the settings file -- Registry key in windows,
# in Mac OS, ~/Library/Preferences/com.bogosity.BOGUS.plist
app.setOrganizationName("BOGUS_NAME")
app.setOrganizationDomain("bogosity.com")
app.setApplicationName("BOGUS")
from PyQt5.QtCore import QSettings
settings = QSettings() # open settings file
# retrieve the key from argv[1]
value = settings.value(sys.argv[1],"?")
print('key',sys.argv[1],'value',value)

A "writer" program has saved a character string under key "c", an integer under "i", and a byte string under "b". Executing reader.exe b produces a segfault, where executing it with arguments of "c" or "i" produce the expected output. I quickly ran cxfreeze on the reader and writer programs and tested them:

[11:11:48 pyinst] dist/reader b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

The exact test that segfaults in Nuitka, yields an error in the frozen module. Note that python reader.py b does not do either; it produces key b value b'\xde\xad\xbe\xef'.

I am pretty sure that the Nuitka-compiled code is segfaulting either (1) while it is attempting and failing to convert the bytestring value, or (2) while it is trying to create the exception. The stack trace makes me favor (1). This is it in part:

0   Python                         0x00000001071532ec PyModule_GetState + 12
1   _pickle.so                     0x0000000108bdab60 Pdata_pop + 32
2   _pickle.so                     0x0000000108be4740 load + 928
3   _pickle.so                     0x0000000108be5879 _pickle_loads + 361
4   Python                         0x00000001070ffed8 PyObject_Call + 104
5   Python                         0x0000000107101ec8 PyObject_CallFunctionObjArgs + 408

What I kinda suspect is that there is a segfault in both cases, but the Python interpreter is catching it and converting it to an exception.

Ah, but now the plot thickens a bunch. The problem could be in the writer! Observer this sequence:

[12:05:42 pyinst] writer.dist/writer
[12:05:49 pyinst] python reader.py b
Traceback (most recent call last):
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object
[12:05:55 pyinst] python writer.py
[12:06:02 pyinst] python reader.py b
key b value b'\xde\xad\xbe\xef'

So the CPython reader call fails, if the value was written by the frozen writer. Running the CPython writer makes the CPython reader ok again. Ok then,

[12:08:48 pyinst] reader.dist/reader b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

OK, the frozen reader cannot convert the value even when written by the CPython writer. Just the same, I believe I need to go back to the nuitka folder for another try.

[12:10:26 nk] python writer.py
[12:10:31 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'
[12:10:33 nk] reader.dist/reader.exe b
Segmentation fault: 11
[12:10:47 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'
[12:10:58 nk] writer.dist/writer.exe
[12:11:03 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'

Hmmm. Let us review.

  • CPython writer output can be read by CPython reader.
  • Nuitka-compiled writer output can be read by CPython reader.
  • Frozen writer output cannot be read by CPython reader.
  • Frozen reader cannot convert bytestring from any writer.
  • Nuitka-compiled reader cannot convert bytestring from any writer.

I must mull...

Saturday, January 10, 2015

Bundle blues

I am now deep into trying to find a way to bundle either of my apps. To "bundle" a Python app is to wrap the main script, all its Python and library dependencies, and a Python interpreter, into a single, self-contained, executable thing. That thing can be given to a user who may not have PyQt, or Qt, or a compatible Python, or any Python at all, and will still run.

Lots of people have tackled the problem of making a Python app-bundler. It is a highly non-trivial problem. Just figuring out what Python modules a script depends on is non-trivial, even setting aside the ability of a Python program to introspect the import-mechanism and direct it to different targets at run-time.

Another issue is, which platform does a bundler support? There is py2app which bundles Python on Mac OS for Mac OS. There is py2exe which bundles Python on Windows, for Windows.

I know of three possible cross-platform solutions. Both cx_Freeze and PyInstaller run on Windows, Mac, or Linux and create bundles for the platform on which they run. And Nuitka runs on all three platforms and compiles a Python app (via C++) to an executable.

In recent days I've tried all three for Cobro with failures in every case. Different, unique failures. Sigh.

With Nuitka, as noted in the prior post, a tiny PyQt5 program dies with a segfault. I'm in an email conversation with Nuitka's developer Kay. It appears the problem may be that Nuitka bundles a different "pickle" module than the built-in module. PyQt5 uses "pickle" to store and retrieve QSettings values. So that may be the conflict, although it is not clear yet. Kay claims it doesn't fail except on Mac OS. I think my next move with Nuitka would be to finish setting up my Ubuntu 14.10 dev system, put Nuitka on it, and try it there.

I tried cx_Freeze and as described yesterday, its generated bundle dies looking for zlib in a nonexistent location. It has been 36 hours since I posted that to the cx_Freeze mailing list with no replies.

I am most familiar with PyInstaller. I contributed a major rewrite of its documentation early in 2014, and also spent some weeks doing a preliminary job of coding its Python3 support. Since then the Python3 support has advanced to a beta-ish level, so I downloaded it and tried it.

I quickly found a couple of minor bugs, and posted them as issues on github. After patching them on my local copy, I was able to get a complete bundle, but this dies with a different mesage:

    This application failed to start because it could not find or load the Qt platform plugin "cocoa".
    Reinstalling the application may fix this problem.
    Abort trap: 6

Searching on this messsage text turns up many hits. Only one offers some kind of solution which is extraordinarily complex, involving use of the XCode utility install_name_tool. Give me a break...

So at the moment I am stymied on three bundling fronts. There is one more approach to try, one that I've been putting off because it looks like a steep learning curve: using pyqtdeploy to create a file that can be processed by QMake. I guess that will be my next step. Sigh. Maybe monday.

Friday, January 9, 2015

cx_Freeze and Nuitka non-progress, Cobro first crash

Started the day by trying to do detailed analysis of the Nuitka crash. For the record, this is the minimal test case.

#from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QCoreApplication
from PyQt5.QtCore import QSettings
import sys
app = QCoreApplication([])
app.setOrganizationName("BOGUS_NAME")
app.setOrganizationDomain("bogosity.com")
app.setApplicationName("BOGUS")
settings = QSettings()
byte_string = b'\xde\xad\xbe\xef' # or 'just plain characters'
settings.clear()
settings.setValue('bogus_byte_string',byte_string)
settings.sync()
return_string = settings.value('bogus_byte_string',b'\x00\x00\x00\x00')
if sys.version_info >= (3,):
    assert return_string == byte_string, (repr(return_string), "!=", byte_string)
print("OK.")

If the byte_string variable has plain characters, not a bytes value, it runs and prints "OK". With the bytes value, it dies with a segfault inside the pickle module, while executing the QSettings.value() function. Probably while PyQt is trying to coerce a QByteArray back into a Python bytes()—although why inahell it needs pickle for that is a mystery.

Anyway I tried all the variations that Kay Hayer had suggested (none helped) and sent in a detailed report.


Yesterday I downloaded and tried cx_freeze on Cobro. It generated some ominous looking compiler warnings during installation; and then when I ran it, it died looking for /opt/local/lib/zlib.dylib or such. There is no path "/opt" on this machine. There does exist a /usr/local/lib/zlib.dylib. So I wrote two short notes to the cx-freeze mailing list describing these issues. No response after 24+ hours.


Today when I ran it, Cobro had its first actual crash under Qt5.4. From the crash report the call sequence (innermost or most-recent first) was:

WebCore::ResourceLoader::ResourceLoader()
WebCore::NetscapePlugInStreamLoader::create()
WebCore::ResourceLoadScheduler::schedulePluginStreamLoad()
WebCore::PluginStream::start()
WebCore::PluginView::performRequest()
WebCore::PluginView::requestTimerFired()
WebCore::ThreadTimers::sharedTimerFiredInternal()
QObject::event()
QApplicationPrivate::notify_helper()
QApplication::notify()
sipQApplication::notify()
QCoreApplication::notifyInternal()
QTimerInfoList::activateTimers()
libqcocoa.dylib QCocoaEventDispatcherPrivate::activateTimersSourceCallback(void*)
etc, etc

In other words, a timer popped, and WebCore tried to load a plug-in (a Netscape plug-in??) and died. I would love to set plug-ins disabled, but too many web comics rely on Flash. Anyway, this is a new crash, it never occurred in the log I kept of the many Qt5.3 Webkit crashes. And it's pretty depressing that it happened at all. I just don't know what to do with Cobro. I had a dream of releasing it to the world, promoting it on reddit and elsewhere, getting universal acclaim.

Ain't gonna get no universal acclaim from something that frequently segfaults. Just be a pain in the ass fielding trouble reports about which I can do abso-bloody-lutely nothing because they just happen deep inside WebKit.

I gotta think harder about, could I use WebEngine even lacking the features it does...

Thursday, January 8, 2015

Menu Loop

Today I reworked the mainwindow module to handle the Edit menu completely differently. It now creates an Edit menu and disables it. Then it offers these global functions:

def set_up_edit_menu(key, action_list) :
    global _EDIT_MENU, _LAST_KEY
    if key != _LAST_KEY :
        _EDIT_MENU.clear()
        for (title, slot, key) in action_list :
            if title is None :
                _EDIT_MENU.addSeparator()
            else :
                _EDIT_MENU.addAction(title, slot, key)
        _LAST_KEY = key
    _EDIT_MENU.setEnabled(True)

def hide_edit_menu():
    _EDIT_MENU.setEnabled(False)

Any subordinate panel that supports Edit-menu actions such as Copy, must have a focusInEvent() method in which it calls set_up_edit_menu() with a unique key (a letter like "E" for Edit) and a list of triples each of which represents an edit menu action, such as ('Copy',self.copy,QKeySequence.Copy). And I put such code into noteview and editview, and tomorrow will put it in wordview.

In the first version of the above code the final lines were not _EDIT_MENU.setEnabled(True/False) but rather _EDIT_MENU.setVisible(True/False). That produced an amusing result: a hard loop. What was happening was this.

  1. Click in the Notes panel.
  2. Notes panel gets focusInEvent, calls mainwindow.set_up_edit_menu()
  3. Mainwindow changes Edit menu from invisible to visible
  4. Apparently that gives the Edit menu the focus! So...
  5. Notes panel gets focusOutEvent, calls mainwindow.hide_edit_menu()
  6. Mainwindow changes Edit menu from visible to invisible
  7. Qt says, "oh, you didn't want the focus after all? I'll give it back to this noteview widget, then."
  8. Repeat from step 2.

The reason for the complication of passing a key is that I know there will be frequent times when a panel such as the Edit panel will lose the focus to either a different app entirely, or to a panel that doesn't set the edit menu. So if the menu is already set up, just enable and it return.

Tuesday, January 6, 2015

Managing the Edit menu, right and wrong ways

Absent any news from Nuitka, I sat down to resume work on a deferred piece of the Word panel, namely, giving it a custom Edit menu. The noteview module has its own Edit menu, and Word needs its own.

Here's the thing. In Qt menus work like this. A QMenu is populated with a set of QMenuAction objects. Each action has the properties of:

  • a name string, such as "Copy" or "Save";
  • an optional icon;
  • an Enabled flag, which if False means the action is visible but inert, and probably shown differently, e.g. is "grayed-out";
  • and most significantly, a slot (function) to be called when the action is selected.

Often the slot is an existing, standard method of a standard widget. For example, a widget based on QTextEdit might set up its Edit:Copy action to call its own standard copy() method.

Hah! But what happens when an app offers multiple, different, independent widgets? As for example, PPQT offers an Edit panel where the document is edited, but also a Notes panel where the user can write a different document. If there is just one Edit menu and it contains just one Copy action, which QPlainTextEdit object's copy() method is called when the user selects Edit:Copy?

For PPQT it gets worse because the Word panel also wants to support Edit:Copy to copy the list of one or more words currently selected in the Words table. That is a QTableView object which doesn't have a built-in copy() method.

So as the user hops around, clicking the keyboard focus into one panel and then another and then another, how does the Edit:Copy action (not to mention Cut, Paste, etc) keep up?

In V1, it didn't keep up very well. The Edit menu actions operated only on the Edit panel. The keyboard accelerators such as ^c, ^v, and ^x operated kind-of correctly. The Qt application tracked the keyboard focus and directed those actions between at least the Edit panel and the Notes panel. They did not get passed into the Words panel. I spent a lot of time trying to capture ^c via a keyPressEvent function in the Words panel, and could never find a way to make Qt let me see that key combination. It was short-circuited at some higher level and never entered the key-event stack. Eventually I implemented the kludge of shift-control-c, which I could capture in the Words panel, to perform a Copy from the Words table.

For V2 I decided to use the more sophisticated kludge of giving each panel that wanted one, its own Edit menu. (The following is for historical interest only; it was a bad idea.) There is only one Menu Bar, created by the main window. It gave access to the menu bar with a module-level global and a get_menu_bar() function.

In the Notes panel, for instance, it creates its own Edit menu and adds it to the global menu bar, with its Visible set to False. When Notes receives a focusInEvent() call, it sets its Edit menu visible to True; and when it gets a focusOutEvent() it sets it back to False. Thus, I thought, the app-wide menu bar might have several Edit menus simultaneously, but only one, or sometimes zero, of them would be visible.

This seemed to be working with Qt 5.3, except it was implemented only for Notes.

When I moved to Qt 5.4 and started running the program, I noticed some oddball messages popping up in the console: "void QCocoaMenu::insertNative(QCocoaMenuItem *, QCocoaMenuItem *) Menu item is already in a menu, remove it from the other menu first before inserting". Wha? I googled that and found two Qt bugs that produced that message, but they seemed to be associated with crashes and with other circumstances. I tabled that issue temporarily.

Then today I started adding the code to wordview.py to create and display its own Edit menu, parallel to the one set up by noteview. And surprise, the number of those messages suddenly increased. Clearly, I was causing them with my duplicate Edit menus. Qt 5.4, among its other changes, has started using the Cocoa UI of Mac OS. It appears that Cocoa does not like having multiple menu items with identical names in the menu bar.

So now I need a better kludge. I am going to go back to square one on the Edit menu. I will implement an Edit menu setter-upper function in mainwindow. It will be called with a list of QMenuAction items. It will call the clear() method of the one-and-only Edit menu, repopulate it with the actions passed by the caller, and make it visible. The Edit, Notes, and Words panels will each, upon focusIn, call the main window to repopulate the Edit menu. And on focusOut, they will call the mainwindow to tell it to hide the Edit menu. In this way there will be only one Edit menu, so Cocoa will be happy, but it will have actions that point to slots in the panel that currently has the focus.


Incidentally, while working over this problem, I discovered the cause of another mysterious bug. I described this back in June in a post about the Notes panel under the heading "UI Issues". When the keyboard focus left the Notes panel, its selected text changed to gray as it should in an inactive editor; but when the focus returned, its selection did not resume the default bright-yellow hue. I actually put in code to force the correct palette.

I put that code in the focusInEvent() and focusOutEvent() methods, which I had added in order to show and hide the Edit menu.

Today I realized: I never passed the focus events on to the parent class! Each of those events should end with a call to super() to pass the event up the chain. No wonder the palette never changed on focus-in—the parent never saw the event!

Monday, January 5, 2015

No Nuitka movement; QWebEngine mail; actually used ppqt2

No new info from Kay. Had hoped for a fresh shot at compiling Cobro, but nothing today.

After several days of silence on the QWebEngine email list, a Jocelyn Turcotte replied to my query about a timeline for missing function. Some of the reply is a bit disturbing.

You can find some of that information at: http://doc.qt.io/qt-5/qtwebenginewidgets-qtwebkitportingguide.html

Actually this porting guide raised some new questions. One that I passed back to her was this: According to a passing reference in its doc, QWebEnginePage is supposed to support a contextMenuEvent() method. However, QWebEnginePage derives from QObject, not QWidget. QObject doesn't have a contextMenuEvent(). QWidget does, but QWebEnginePage doesn't inherit from it. So it is not clear that it can actually support QEvent calls of any kind! At least, not using the normal QEvent overrides like contextMenuEvent or keystrokeEvent.

Some of the features like private browsing should already be available in Qt 5.5.

Good, but that's 6 months away. And -- which features are committed to that release?

We currently have an internal development board at https://trello.com/b/5G9c1rkb/qtwebengine that you can follow and you can use the Qt bug tracker to vote and report issues or missing important features.

I looked at at this "board"; it is not a proper forum but more a virtual corkboard where developers post "cards" with the tasks they are doing. No sorting or searching and no obvious place for comments.

I have to note that QtWebEngine isn't a 1-to-1 replacement of QtWebKit, principally because some very advanced features can't be set in stone through a public API without risking huge future maintenance efforts as Chromium evolves under our feet. It is quite a dragon to ride.
The public API aims at solving the general use case of embedding web contents in Qt applications, but if you are developing a more powerful application I encourage you to look at the code and dig further to get access to functionalities that you need, keeping in mind that the further you dig, the more likely you'll have to update your code between QtWebEngine updates.

Well, that remark provoked a quick reply from another user, expressing disappointment that QWebEngine was punting on being a complete Webkit replacement, leaving developers with a choice of a "more powerful" API that is frozen (doesn't support HTML5, for example) and a less-featured one that can't be used for anything sophisticated. (Supposing a custom context menu is sophisticated.)

And finally: there was a book I'd post-processed that got kicked back to me for minor fixes in October. I finally got around to finishing that, and I used ppqt2 to do the editing. Worked fine.

Saturday, January 3, 2015

Success with a Nuitka crash; Stymied by stderr

The Nuitka compile of Cobro died looking for the Tk and Tcl framework folders in the wrong place, because otool reports incorrect paths. I put symlinks in /Library/Frameworks pointing to the folders in /System/Library/Frameworks, and ran the compile again. It completed, and produced a 9MB executable. I tried executing it with the --help option, and it ran and responded as expected. Of course this minimal test only gets as far into the __main__ code as the call to parseargs.

Alas, when I ran it properly, it immediately crashed with a segfault. Somewhere in the construction of the numerous UI objects it dies. I sent a rather dejected note with the crash report to Kay Hayen, Nuitka's author. He quickly replied pointing out that one of the final calls in the stack was to "pickle". He said Nuitka uses pickle, and perhaps that is somehow conflicting with PyQt's use of pickle.

PyQt does pull in pickle among the couple of hundred other modules Nuitka lists. But just then I couldn't see a connection. This was Friday afternoon.

Around 4 or 5am today, Saturday, I was half-awake and thinking about this (obsessing about software at 4am is not uncommon for me; well, there are worse things to obsess about when awake in the wee hours...) and it occurred to me that one place PyQt might use pickle is in converting between Python data types and Qt datatypes. And that the program crashes about the point where it would be fetching values from the QSettings. Most of what it pulls from settings is character strings, but it also keeps the SHA-1 hash signature of each comic in settings. These are byte types in Python, but PyQt would have to convert them to QByteArray types. Maybe it uses pickle for this?

So after breakfast I trotted to my computer and composed a ten-line test case: make the QApplication, make the QSettings, push a byte string into the settings and fetch it back.

Nuitka compile. Completed. Run the resulting testcase.exe (Nuitka gives its output binary the .exe suffix even in Mac or Linux).

Boom! Instant segfault, with precisely the same stack trace ending in pickle that the big program produced!

I tell you, I was chuffed. A lovely little test case that crashes in exactly the expected way, first time out of the box. Niiiiice! Shot that off to Kay who I expect will be as pleased as I.

Not quite such a success was my attempt to capture and stifle the stderr output of Cobro. One can quite easily redefine stderr in Python. You just assign something that behaves like a file to sys.stderr. That's all. I did this, assigning a stringIO object to it, and ran the program. Out comes the "can't find DjVu" and "can't find QuickTime" plug-in messages on the console. Nothing captured in the stringIO object.

So the code that emits at least those messages has its own copy of the program's stderr handle. It doesn't go through the one kept by the sys module.

One way to do this would be to have a two-level program. The "cobro" app would be a shell that used the subprocess module to launch the "real" cobro app in a subprocess. Then for sure the shell could capture all the stderr output of the subprocess. Well, probably for sure. Who knows what those CGContextSaveGState messages are capable of?

I really do not want to complicate the program in this way but I might try it just to see if it worked.

Friday, January 2, 2015

Small Nuitka Progress; More on Webkit messages

As mentioned yesterday, the Nuitka compile ended with a futile search for "ldd" on Mac OS. Kay quickly replied with a one-line fix to repair a regression introduced recently. I applied it and reran the compile. This time it got much further still, producing the rather jaw-dropping message,

Total memory usage before running scons: 175.18 GB (188101951488 bytes)

Some seconds later it died on a failure to find '/Library/Frameworks/Tk.framework/Versions/8.5/Tk'. It already has found and collected _tkinter.so from the Python 3.4 framework. The problem seems to be that the full Tk.framework is not in /Library/Frameworks; it is in /System/Library/Frameworks. However, that is not what the otool command reports:

 otool -L ./lib/python3.4/lib-dynload/_tkinter.so
./lib/python3.4/lib-dynload/_tkinter.so:
 /Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (compatibility version 8.5.0, current version 8.5.15)
 /Library/Frameworks/Tk.framework/Versions/8.5/Tk (compatibility version 8.5.0, current version 8.5.15)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

Why would otool tell this fib?

Setting that aside, I ran cobro from source again and observed the nuisance messages more closely, and tried to relate them to what the browser was doing.

The first two messages—one about cannot find Plug-Ins/DjVu, and one about cannot perform a dlopen of the QuickTime Plugin—come out instantly when I click to load the first comic, XKCD. That is quite a simple page with no obvious ads or other plug-in-requiring content.

Then I browse several comics down to Questionable Content. It fully loads (per the progress bar) and I am quietly scrolling it when there is a burst of 20 repetitions of this one: ": CGContextRestoreGState: invalid context 0x1292d1c80. This is a serious error. This application, or a library it uses, is using an invalid context and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update."

I shan't reiterate how offensive I find this text to be. The point is, there is absolutely nothing I can do to control or prevent it. The fact that it happened several seconds after the page had completed loading strongly indicates that the fault is in some content loaded by an ad on the page. The comic image just sits there, but ad frames often roll through animated sequences of content. At some point, an ad pulled in something that triggered these messages.

I can't stop a webcomic from loading ads, indeed I don't want to do so. I designed Cobro as a full browser, and not as a simple image scraper like ComicTastic whose UI I have copied, precisely because I did not want to deprive comic artists of their livelihood. And I can't censor the content the ads load. Can I? I doubt much whether I could insert something like adblock into the Webkit (or WebEngine) browser.

So these bursts of pointless and offensive messages will come out on the console and I have no way to prevent them and cannot possibly respond to them.

After I had browsed several more comics, while Stone Soup (from comics.com) was still (slowly) loading, a new message appeared:

"vtDecompressionDuctCreate signalled err=-8973 (err) (Could not select and open decoder instance) at /SourceCache/CoreMedia_frameworks/CoreMedia-1562.19/Sources/VideoToolbox/VTDecompressionSession.c line 1181 slow render still not complete"

It was followed by 19 more repetitions of the CGContextRestoreGState one.

Alright now, David, start thinking constructively instead of just bitching! These messages are streaming out from the browser code but they are, one supposes, directed to stderr. Surely there is some way to take control of stderr? After all, if this was a GUI app (which it should be) it would not have a stderr and presumably these messages would "fade, unseen by any human eye" (Wordsworth), and serves them right.

I must look into this.

Thursday, January 1, 2015

Hacking at Nuitka

Nuitka is a Python compiler. It processes Python source code and, where the CPython interpreter would byte-compile to a byte-code, Nuitka compiles to C++ source code. The generated source code relies on the Python/C API. It implements the same program, including calls into the built-in CPython interpreter functions, but as a compilable source module—or, what interests me, as a complete, standalone, compiled and linked executable.

And it works! Well, almost. When I feed cobro.py into Nuitka, the output is in two folders. In cobro.dist is a set of all the modname.so modules that the program relies upon, including a whole subfolder of PyQt5 modules. In cobro.build is a set of .cpp source files. When I browse in these, they are clearly based on compiling my code. So that's all good as far as it goes.

Unfortunately Nuitka under Python 3.4 does not run to completion. Nuitka has a strong dependency on SCons, a make replacement. Unfortunately, SCons is written in Python 2.x. Support for Python 3 is in process but not available. So Nuitka compiles my code as described, but then fails because it tries to invoke SCons to (presumably) perform the compile and link step, and SCons fails.

Yet Nuitka claims Python 3 compatibility. It knows that SCons depends on Python 2, so when it is time to execute SCons, Nuitka tries to find a Python 2 executable under which to run it. On my system it couldn't. I tried various ways to get around this and so far have failed. Kay Hayen, the creator of Nuitka, has been very responsive to my emails thus far, and perhaps he will yet figure something out.

But here I am, with some lovely .cpp source files and bunch of .so files, and all I need to do is run a compile and link. Tantalizing! So I tried, several times, without success.

Just now I got note from Kay, saying in part,

That mode of operation [manual compilation] is unfortunately not workable anymore. Nuitka generated source does want to get defines sent, and stuff. I think I couldn't do it myself without a lot of work. That is what we use Scons for So we can have complex arguments...

As a workaround, create a script "/usr/bin/python2" and make it set your environment variables correctly and export them, and then execute Python2 with its arguments. Then Nuitka will pick it up.

I may try this next. Nuitka is frustratingly close to being a complete solution to the issues of packaging. Just imagine being to make a compiled binary executable with no external dependencies. And it runs on Windows, Linux, and Mac OS. If I can get past this SCons hurdle, it could work.

Edit: I did in fact try the above, and it worked, in the sense that it allowed SCons to run without error. The Nuitka compile command now ran much, much longer, 30 seconds or so where before it ended after about 10 seconds.

Unfortunately it then ended with the error, "FileNotFoundError: [Errno 2] No such file or directory: 'ldd'" which is quite correct, there is no "ldd" in Mac OS. It is replaced by "otool -L". So another email to the patient Kay. Tomorrow maybe we get over this hurdle.