Small Nuitka Progress; More on Webkit messages

As mentioned yesterday, the Nuitka compile ended with a futile search for "ldd" on Mac OS. Kay quickly replied with a one-line fix to repair a regression introduced recently. I applied it and reran the compile. This time it got much further still, producing the rather jaw-dropping message,

Total memory usage before running scons: 175.18 GB (188101951488 bytes)

Some seconds later it died on a failure to find '/Library/Frameworks/Tk.framework/Versions/8.5/Tk'. It already has found and collected _tkinter.so from the Python 3.4 framework. The problem seems to be that the full Tk.framework is not in /Library/Frameworks; it is in /System/Library/Frameworks. However, that is not what the otool command reports:

 otool -L ./lib/python3.4/lib-dynload/_tkinter.so
./lib/python3.4/lib-dynload/_tkinter.so:
 /Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (compatibility version 8.5.0, current version 8.5.15)
 /Library/Frameworks/Tk.framework/Versions/8.5/Tk (compatibility version 8.5.0, current version 8.5.15)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

Why would otool tell this fib?

Setting that aside, I ran cobro from source again and observed the nuisance messages more closely, and tried to relate them to what the browser was doing.

The first two messages—one about cannot find Plug-Ins/DjVu, and one about cannot perform a dlopen of the QuickTime Plugin—come out instantly when I click to load the first comic, XKCD. That is quite a simple page with no obvious ads or other plug-in-requiring content.

Then I browse several comics down to Questionable Content. It fully loads (per the progress bar) and I am quietly scrolling it when there is a burst of 20 repetitions of this one: ": CGContextRestoreGState: invalid context 0x1292d1c80. This is a serious error. This application, or a library it uses, is using an invalid context and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update."

I shan't reiterate how offensive I find this text to be. The point is, there is absolutely nothing I can do to control or prevent it. The fact that it happened several seconds after the page had completed loading strongly indicates that the fault is in some content loaded by an ad on the page. The comic image just sits there, but ad frames often roll through animated sequences of content. At some point, an ad pulled in something that triggered these messages.

I can't stop a webcomic from loading ads, indeed I don't want to do so. I designed Cobro as a full browser, and not as a simple image scraper like ComicTastic whose UI I have copied, precisely because I did not want to deprive comic artists of their livelihood. And I can't censor the content the ads load. Can I? I doubt much whether I could insert something like adblock into the Webkit (or WebEngine) browser.

So these bursts of pointless and offensive messages will come out on the console and I have no way to prevent them and cannot possibly respond to them.

After I had browsed several more comics, while Stone Soup (from comics.com) was still (slowly) loading, a new message appeared:

"vtDecompressionDuctCreate signalled err=-8973 (err) (Could not select and open decoder instance) at /SourceCache/CoreMedia_frameworks/CoreMedia-1562.19/Sources/VideoToolbox/VTDecompressionSession.c line 1181 slow render still not complete"

It was followed by 19 more repetitions of the CGContextRestoreGState one.

Alright now, David, start thinking constructively instead of just bitching! These messages are streaming out from the browser code but they are, one supposes, directed to stderr. Surely there is some way to take control of stderr? After all, if this was a GUI app (which it should be) it would not have a stderr and presumably these messages would "fade, unseen by any human eye" (Wordsworth), and serves them right.

I must look into this.

Hacking at Nuitka

Nuitka is a Python compiler. It processes Python source code and, where the CPython interpreter would byte-compile to a byte-code, Nuitka compiles to C++ source code. The generated source code relies on the Python/C API. It implements the same program, including calls into the built-in CPython interpreter functions, but as a compilable source module—or, what interests me, as a complete, standalone, compiled and linked executable.

And it works! Well, almost. When I feed cobro.py into Nuitka, the output is in two folders. In cobro.dist is a set of all the modname.so modules that the program relies upon, including a whole subfolder of PyQt5 modules. In cobro.build is a set of .cpp source files. When I browse in these, they are clearly based on compiling my code. So that's all good as far as it goes.

Unfortunately Nuitka under Python 3.4 does not run to completion. Nuitka has a strong dependency on SCons, a make replacement. Unfortunately, SCons is written in Python 2.x. Support for Python 3 is in process but not available. So Nuitka compiles my code as described, but then fails because it tries to invoke SCons to (presumably) perform the compile and link step, and SCons fails.

Yet Nuitka claims Python 3 compatibility. It knows that SCons depends on Python 2, so when it is time to execute SCons, Nuitka tries to find a Python 2 executable under which to run it. On my system it couldn't. I tried various ways to get around this and so far have failed. Kay Hayen, the creator of Nuitka, has been very responsive to my emails thus far, and perhaps he will yet figure something out.

But here I am, with some lovely .cpp source files and bunch of .so files, and all I need to do is run a compile and link. Tantalizing! So I tried, several times, without success.

Just now I got note from Kay, saying in part,

That mode of operation [manual compilation] is unfortunately not workable anymore. Nuitka generated source does want to get defines sent, and stuff. I think I couldn't do it myself without a lot of work. That is what we use Scons for So we can have complex arguments...

As a workaround, create a script "/usr/bin/python2" and make it set your environment variables correctly and export them, and then execute Python2 with its arguments. Then Nuitka will pick it up.

I may try this next. Nuitka is frustratingly close to being a complete solution to the issues of packaging. Just imagine being to make a compiled binary executable with no external dependencies. And it runs on Windows, Linux, and Mac OS. If I can get past this SCons hurdle, it could work.

Edit: I did in fact try the above, and it worked, in the sense that it allowed SCons to run without error. The Nuitka compile command now ran much, much longer, 30 seconds or so where before it ended after about 10 seconds.

Unfortunately it then ended with the error, "FileNotFoundError: [Errno 2] No such file or directory: 'ldd'" which is quite correct, there is no "ldd" in Mac OS. It is replaced by "otool -L". So another email to the patient Kay. Tomorrow maybe we get over this hurdle.

Webkit, a never-failing source of laughs

So yesterday I wrote how the Qt5.4 Webkit seems to be behaving itself. And it is, compared to the 5.2 and 5.3 versions. It actually displays all the hard comics without screwing up the rendering. And so far it hasn't crashed in any of the common ways I summarized in the previous post.

However, the following is the slightly edited console output of running it today.

[08:10:09 CoBro] python ./cobro.py
2014-12-31 08:10:21.581 Python[68621:14259725] Cannot find executable for CFBundle 0x10858fd00
</Users/dcortesi/Library/Internet Plug-Ins/DjVu> (not loaded)
2014-12-31 08:10:21.589 Python[68621:14259725] Error loading /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin:  dlopen(/Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin, 265): no suitable image found.  Did find:
 /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin: mach-o, but wrong architecture
GVA info: Successfully connected to the Intel plugin, offline Gen75 

That was just for starters. Then as I got down toward the end of the list, after viewing a dozen comics without further messages, I clicked on Gunnerkrig Court and the console filled up with this.

[08:16:58.637] FigLimitedDiskCacheProvider_CopyProperty signalled err=-12784 (kFigBaseObjectError_PropertyNotFound) (no such property) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/FigByteStreamPrototypes/FigLimitedDiskCacheProvider.c line 947
<<<< FigByteStream >>>> FigByteStreamStatsLogOneRead: ByteStream read of 8 bytes @ 4453661 took 0.573590 sec. to complete, 1 reads >= 0.5 sec.
Dec 31 08:17:18 Silver-streak-2.local rtcreporting[68621] : logging starts...
Dec 31 08:17:18 Silver-streak-2.local rtcreporting[68621] : setMessageLoggingBlock: called
[08:17:18.935] itemasync_GetDuration signalled err=-12785 (kFigBaseObjectError_Invalidated) (invalidated item) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/Player/FigPlayer_Async.c line 2870
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : startConfigurationWithCompletionHandler: Cached 0 enabled backends
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : setUserInfoDict: enabled backends: (
 )
Dec 31 08:17:19 Silver-streak-2.local rtcreporting[68621] : initWithSessionInfo: XPC connection invalid
Dec 31 08:23:14 Silver-streak-2.local Python[68621] : CGContextSaveGState: invalid context 0x126544940. This is a serious error. This application, or a library it uses, is using an invalid context  and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update.
Dec 31 08:23:14 Silver-streak-2.local Python[68621] : CGContextScaleCTM: invalid context 0x126544940. This is a serious error. This application, or a library it uses, is using an invalid context  and is thereby contributing to an overall degradation of system stability and reliability. This notice is a courtesy: please fix this problem. It will become a fatal error in an upcoming update.

The above message repeated 15 more times.

Do I need to analyze these things? Is there a single gorram thing I can do to prevent them?

Sigh. OK, one at a time. Inserting newlines so blogger can display them.

2014-12-31 08:10:21.581 Python[68621:14259725] Cannot find executable for
CFBundle 0x10858fd00
</Users/dcortesi/Library/Internet Plug-Ins/DjVu> (not loaded)

Some webcomic wants a "DjVu" plugin. This is what I get for telling the browser, PluginsEnabled(True). Because so many comics rely on Flash, you see. What is "DjVu"? I suppose I could find out, but supposing I did, is there any way I could ensure it would always be available wherever Cobro runs? No. And why oh why cannot Webkit fail silently and just put up a broken-plugin icon? Why does it have to blabber about it on the console?

Then we have this beautiful message, let's break it down by parts.

Error loading /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin:

Another plugin the browser can't find. I don't care! Nobody cares. Just shut up.

dlopen(/Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime Plugin,
265): no suitable image found.

My heart bleeds for you.

Did find: /Library/Internet Plug-Ins/QuickTime Plugin.plugin/Contents/MacOS/QuickTime
Plugin: mach-o, but wrong architecture

I'm still not caring.

GVA info: Successfully connected to the Intel plugin, offline Gen75 

Good job, Webkit! If I knew what "GVA info" was, I'd probably be very impressed. What do you suppose it means, "offline Gen75"?

Then there's this wonderful streak of completely opaque babble:

FigLimitedDiskCacheProvider_CopyProperty signalled err=-12784
(kFigBaseObjectError_PropertyNotFound) (no such property) at
/SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/FigByteStreamPrototypes
/FigLimitedDiskCacheProvider.c line 947
<<<< FigByteStream >>>> FigByteStreamStatsLogOneRead: ByteStream read of 8 bytes @ 4453661
 took 0.573590 sec. to complete, 1 reads >= 0.5 sec.

This should really be bronzed and hung in a place of honor in the Hall of Indecipherable and Utterly Useless Messages. It is telling me about an error -12784, apparently having to do with some undocumented cache not finding some arcane property. But instead of explaining what that error means or how to fix it, it natters on about how it read 8 bytes in one-half second. I can personally read a lot faster than that, so I don't think a 4-core 3.5Ghz machine should be bragging about this.

The next part is just poetry. Lightly edited for aesthetic enjoyment,

: logging starts...
: setMessageLoggingBlock: called
=itemasync_GetDuration signalled err=-12785
   (kFigBaseObjectError_Invalidated)
   (invalidated item) at /SourceCache/CoreMedia/CoreMedia-1562.19/Prototypes/Player/FigPlayer_Async.c line 2870
: startConfigurationWithCompletionHandler: Cached 0 enabled backends
: setUserInfoDict: enabled backends: ( )
: initWithSessionInfo: XPC connection invalid

So helpful! So informative! So much not wanted by anybody... And then we have 16 repetitions of

: CGContextSaveGState: invalid context 0x126544940.
  This is a serious error.
  This application, or a library it uses, is using an invalid context
  and is thereby contributing to an overall degradation of
  system stability and reliability.
  This notice is a courtesy: please fix this problem.
  It will become a fatal error in an upcoming update.

Oh, it's a "courtesy" is it? And I am to "fix" the "problem" which is happening where and caused by what, and fix it how? Spare me your passive-aggressive manipulative bullshit and fix your own fucking bugs, thank you very much.

---

Maybe later today I'll google some of this horse-hockey. Or tomorrow; that'll be a good way to start a new year. Hope yours starts better.

Postponing WebEngine, trying Nuitka

I posted a polite & constructive note to the WebEngine development list asking for a timeline on when some of the missing features might appear. No replies as yet.

Trying Cobro under Qt5.4 I am very pleased to see that the Webkit version is now behaving itself. There were several comics that it would display incorrectly. This failure was always that some small image file on the page would be shown at 1000% zoom and overlay the rest of the rendered page. It made Penny Arcade, for example, unreadable.

In addition there were at least four failure modes I've been documenting over the past few months of daily use. In no particular order,

• Emitting a stream of a hundred or more messages "Critical failure: the LastResort font is unavailable", then a segfault.
• "QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files... Abort trap: 6"
• Segfault 11 somewhere deep in the Mac OS innards, usually with QNetworkConfiguration in the stack trace.
• Segfault nested 50 or more calls deep in Webkit rendering code.

It typically crashed in one of these ways every 3 or 4 times I ran it. That was clearly unacceptable. I didn't mind it, just sigh and restart it, but I could never distribute a program that was so unreliable. I was really looking forward to changing to WebEngine and leaving all those nasty bugs behind. Unfortunately that appears not to be, because WebEngine doesn't support several functions that, although minor, I want to have for proper operation. The ability to command "private" (uncached) browsing, for example. It would be unacceptable to have the residue of browsing comics in one's work-browser's cache. Or the ability to implement the custom context menu I wrote, that lets you right-click on a link and make it open in the default browser. Can't do that under WebEngine because it lacks the feature of Webkit that lets you find out what type of data was under the mouse, and if a URL, get its text.

However, after several uses, the 5.4 Webkit browser has displayed none of the above problems. It handles Penny Arcade and Stand Still Stay Silent perfectly, and they were unreadable with 5.3. And none of the crashes has happened yet either.

So, I'm going to shelve the conversion to WebEngine for now. I've gone ahead to put in a couple of minor feature enhancements that I had in mind for some time, and I will continue reading comics with it daily, and just maybe it will prove reliable.

Meanwhile, I downloaded Nuitka and tried to install it. Unfortunately, although it claims to support all of Python through 3.4, the code executed by its setup.py displayed a number of syntax errors that were clearly due to byte-compiling Python 2 code under Python 3.4—missing parens on a print statement, for example.

So I joined yet another goddam dev mailing list and sent a polite query about this. Maybe tomorrow I'll get a reply from one group or the other.

Tangled up in Git

Here's how I thought git worked. I start a branch, (git checkout -b newbranch) and edit a file and save it. Have not committed anything yet but changed plenty.

Suddenly realize, I need to try this thing out in its earlier version. So I thought I could just do git checkout master and any changes made on newbranch would be swept under the carpet and I'd be back to the master version.

Did that: branched cobro, made a shit-ton of changes to cobro.py to switch it to WebEngine, realized, wait, I need to check how something worked with WebKit. Did "checkout master" thing and... cobro didn't change. Didn't go back to what it was. Still had all the WebEngine changes in it. I ended up doing a git revert HEAD, which lost all the changes, made a copy cobro-webkit.py, and now I get to make those changes again.

I have no idea what went wrong, or indeed if it was wrong or expected behavior. (Possibly it happened because the file is on the dropbox folder, maybe my laptop was fighting the desktop?)

Also I discovered two new things that QWebKit offers and QWebEngine does not. One is setting the web page non-editable. The other is "link delegation" where anytime a link is clicked, it emits a signal and lets you decide if it should go through. So now I have three posts on this theme at the qt-project forums. Sad to say, only one has drawn a reply. Although that reply points to the WebEngine dev mailing list, so maybe I will learn something from that.

First steps with QWebEngine

Needing to kill some time this evening while my wife worked on her website, I sat down and began the process of converting Cobro from QWebView, QWebPage, QWebSettings to use their modern but more verbose counterparts, QWebEngineView, QWebEnginePage and QWebEngineSettings. Things look good but as usual there are some issues.

Thanks to my splendidly modular and well-organized coding style, there is really only one object affected in the whole program, a QWebView derived class. I changed its parent class from QWebView to QWebEngineView, and then began going through all the self-initializing lines I had used to set up the QWebView.

Many of them had direct equivalents. For example, self.page() returns the underlying QWebEnginePage just as it previously returned the QWebPage. One still uses self.settings().setAttribute(something, True/False) for setting properties. However, I quickly discovered that several of the settable properties of QWebSettings are not available in QWebEngineSettings.

In particular, there is no JavaEnabled, no PluginsEnabled, and no PrivateBrowsingEnabled. I put a query about these three on the qt forum. It seems strange they would go to some trouble to make the new API compatible with the old, and yet leave out settings, and fairly significant settings at that.

Also my app implements a custom context menu. In it, I use the following to check to see if the thing that was right-clicked upon is a URL. Call self.page().currentFrame() to access the QWebFrame in which the right click occurred. Ask that for hitTestContent(QPoint) to find out what was under the point of the mouse event. If it is a URL, then I can access the text of that URL, and offer the user to open it in the default browser. This is very useful; Cobro is not a full-function browser, and if there is something on a comic page besides the comic, it is handy to be able to jump to a real browser.

Unfortunately it does not appear that QWebEnginePage offers access to the current frame. And there is no QWebEngineFrame class. So although QWebEnginePage specifically documents that you can implement a custom context menu, it is not clear how one can find out what was under the right-click. I also posted a query about this in the qt forum.

However, I just commented out all the parts that I couldn't immediately translate, and ran the app, and up it came. And displayed a comic very nicely. The progress bar signals worked, the titleChanged signal worked. So that was a useful couple of hours.

Py/Qt5.4 at last

So Phil at Riverbank computing dropped a Christmas present on us all, with PyQt5.4 on December 24th. Today I installed Qt5.4, the latest SIP, and PyQt5.4 on my new iMac and started working on Cobro. This all went very well. The installations went smoothly. I spent a couple of hours reviewing the code of Cobro in Wing IDE. I tidied up some comments. I removed some unnecessary global statements. And then ran it, and was pleased to find that this version can display some comics that it cannot running in 5.2. Apparently some of the many bugs in QWebkit have been fixed. Doesn't matter; my next move (on Monday) will be replace the webkit element with the new QWebEngine ones. Following that, probably Tuesday, I have a couple of minor functional enhancements to add.

Toward the end of the week, or next week, the next step I think will be to try compiling Cobro with Nuitka. In theory, the result of compiling a program like Cobro in Nuitka should be a single, stand-alone, self-supporting executable. If so, that would bypass the need for a packager like pyinstaller.

If the Nuitka experiment doesn't work, I will, as previously planned, start trying to use pyqtdeploy on it.