cxfreeze and nuitka issues converge

Some days back I tried bundling Cobro with cx_Freeze and ran into a problem where the bundled program died trying to link libz from /opt/local/lib. This was baffling, as that path had never existed on this machine. I wrote to the cx_Freeze mailing list and nobody responded.

Yesterday, still trying to solve the baffling segfault in the Nuitka-compiled code, Kay had me use the dtruss trace tool to list the modules opened by the compiled program. This exposed an issue where the compiled program was loading QtCore from the Qt distribution folder. It should be loading it from the local "dist" folder where Nuitka has collected all the referenced modules, and where the compiled executable is placed.

In order to fix this I had to give myself a crash course in the use of otool and install_name_tool. The former is a utility that displays info from a DLL. The latter can be used to change some of the things encoded in a DLL. Specifically, changing a hard-coded link path to a relative one.

In the course of applying "otool -L" to the various modules, I happened to note in its output, that QtCore had an external link to, guess what, "/opt/local/lib/libz.1.so". So that's where that reference came from.

Today, after reaching yet another dead end on the Nuitka problem, I thought I'd fix that and see if cx_Freeze worked. So I went into the Qt distribution folder and tediously checked the links of every DLL. Well, not so tedious; because I wrote a very complicated shell script to find it:

#!/bin/bash
otool -L $1.framework/Versions/Current/$1 | grep "/opt"

Turns out, there were four or five modules that used it. QtWebKit had three links to things in "/opt/local/lib". It must be that I overlooked some install option that would have corrected this when I installed Qt5.4. I've no idea what, but I must look carefully when installing 5.5.

Anyway, with that done, I could now cx_Freeze Cobro and the frozen app ran. For just a little bit. Then it died with a Python error that made me say "aHA!":

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "cobro.py", line 1785, in 
  File "cobro.py", line 1429, in __init__
  File "cobro.py", line 964, in load
TypeError: unable to convert a QVariant back to a Python object

That is the exact point in the program where the Nuitka compilation dies with a segfault: the moment when PyQt is trying to fetch a byte string from a QSettings value. Here is the current Nuitka minimal test case:

from PyQt5.QtCore import QCoreApplication
import sys
app = QCoreApplication(sys.argv)
# This defines the key to the settings file -- Registry key in windows,
# in Mac OS, ~/Library/Preferences/com.bogosity.BOGUS.plist
app.setOrganizationName("BOGUS_NAME")
app.setOrganizationDomain("bogosity.com")
app.setApplicationName("BOGUS")
from PyQt5.QtCore import QSettings
settings = QSettings() # open settings file
# retrieve the key from argv[1]
value = settings.value(sys.argv[1],"?")
print('key',sys.argv[1],'value',value)

A "writer" program has saved a character string under key "c", an integer under "i", and a byte string under "b". Executing reader.exe b produces a segfault, where executing it with arguments of "c" or "i" produce the expected output. I quickly ran cxfreeze on the reader and writer programs and tested them:

[11:11:48 pyinst] dist/reader b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

The exact test that segfaults in Nuitka, yields an error in the frozen module. Note that python reader.py b does not do either; it produces key b value b'\xde\xad\xbe\xef'.

I am pretty sure that the Nuitka-compiled code is segfaulting either (1) while it is attempting and failing to convert the bytestring value, or (2) while it is trying to create the exception. The stack trace makes me favor (1). This is it in part:

0   Python                         0x00000001071532ec PyModule_GetState + 12
1   _pickle.so                     0x0000000108bdab60 Pdata_pop + 32
2   _pickle.so                     0x0000000108be4740 load + 928
3   _pickle.so                     0x0000000108be5879 _pickle_loads + 361
4   Python                         0x00000001070ffed8 PyObject_Call + 104
5   Python                         0x0000000107101ec8 PyObject_CallFunctionObjArgs + 408

What I kinda suspect is that there is a segfault in both cases, but the Python interpreter is catching it and converting it to an exception.

Ah, but now the plot thickens a bunch. The problem could be in the writer! Observer this sequence:

[12:05:42 pyinst] writer.dist/writer
[12:05:49 pyinst] python reader.py b
Traceback (most recent call last):
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object
[12:05:55 pyinst] python writer.py
[12:06:02 pyinst] python reader.py b
key b value b'\xde\xad\xbe\xef'

So the CPython reader call fails, if the value was written by the frozen writer. Running the CPython writer makes the CPython reader ok again. Ok then,

[12:08:48 pyinst] reader.dist/reader b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/Current/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in 
    exec(code, m.__dict__)
  File "reader.py", line 12, in 
    value = settings.value(sys.argv[1],"?")
TypeError: unable to convert a QVariant back to a Python object

OK, the frozen reader cannot convert the value even when written by the CPython writer. Just the same, I believe I need to go back to the nuitka folder for another try.

[12:10:26 nk] python writer.py
[12:10:31 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'
[12:10:33 nk] reader.dist/reader.exe b
Segmentation fault: 11
[12:10:47 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'
[12:10:58 nk] writer.dist/writer.exe
[12:11:03 nk] python reader.py b
key b value b'\xde\xad\xbe\xef'

Hmmm. Let us review.

• CPython writer output can be read by CPython reader.
• Nuitka-compiled writer output can be read by CPython reader.
• Frozen writer output cannot be read by CPython reader.
• Frozen reader cannot convert bytestring from any writer.
• Nuitka-compiled reader cannot convert bytestring from any writer.

I must mull...