Translator options dialog working

So I took my own advice and created a class to represent a dialog option. The author of a Translator codes some of these in his module as global variables, e.g.

import xlate_utils as XU
MAX_LINE_QUERY = XU.Dialog_Item( kind='number', label='Max Line',
                                tooltip='Maximum line width, default is 72',
                                minimum=50, maximum=90, result=72 )
...
OPTIONS_DIALOG = [ MAX_LINE_QUERY, ... ]

The Dialog_Item class definition has only one method, __init__ and it has about 120 lines of code that is mostly validation of the parameters as being the proper types. Errors get logged but some kind of object is always created that the rest of the code can display, if only as a QLabel "Bad Definition Here".

When the Translator is requested, the first thing that happens is to look into the loaded translator module for OPTIONS_DIALOG as a list of Dialog_Item objects. If it's there, a dialog is prepared. Here's one of the test cases in action.

So that all went together nicely. The code to build the dialog, display it, and capture the values set by the user is all quite compact.

Well, that was the fun part. The next part, which is really central to all Translators, is code to parse a document in DP format and reduce it to elemental parts, which can then be fed to any Translator for conversion. "Here's a Chapter Head, convert it. Here's a sub-head, convert it." And so on. But it goes much deeper than that, especially since I want to support nesting of, for example, a Poetry section inside a Footnote or Illustration, and so on. It's non-trivial.

I did this in V1 in a kind of hand-crafted semi-intuitive way. But the only consumer of the output of that parser was my own code for ASCII reflow and HTML conversion. Now I have to look to a generic consumer, a coder who is not me. And I do not want to expose any of the PPQT internals, like the way a document is stored, to the Translator. It needs a very clean, arms-length API. And I would like the parser to have a better foundation in computer science instead of just a big hacky loop with a bunch of state variables, as in V1.

I've been pointed to a partial DP document parser that I need to read. And I have some ideas of how to go at it. But it's a big and critical chunk of work that I need to do next.

A dict has no attrs -- or actually, it does

So I'm starting to code the Translator interface. I've been planning this and making notes on it for months and it's fun to start making it real. Also fun to be back into code-and-test mode after a long spell just flogging installation and bundling problems.

I mean to offer each Translator a simple way to query the user for options. The Translator module uses a simple, static, declarative API to describe what it needs to know from the user. (I talked about this earlier but I've made the API simpler and nicer since.) When the user calls for that translation, I'll whomp up a QDialog on the fly with the necessary widgets—QCheckbox, QSpinbox, QLineEdit—show them to the user, and stow the user's input back where the Translator can refer to them. I'm almost ready to start testing this support except I ran into something that's making me rethink the details of the API. And realize the frustrating limitations of the Python namedtuple class.

What I'm currently asking the coder to do is to describe each dialog item as a dict, for example to ask for a yes/no choice,

OMIT_TABLES = {
    "Type" : "Checkbox",
    "Label" : "Omit tables?",
    "Tooltip" : "Check this if the translation should skip /T tables",
    "Result" : False
}

That's a nice enough API. Except for the visual effect of all the quotes, which make it look like it needs a shave! So I was writing the code to validate one of these. I dare not assume my client Python coder has done it right, so I need to check everything. Does it have all, but only, the keys it should have? Is everything that is supposed to be a string, a string?

I'd been writing code to interrogate the imported Translator module, which is a Python namespace. You use hasattr() and getattr() for this. So in writing the code to check that one of these dicts was all correct, I wrote things like if hasattr(item, 'Result')... which seemed very natural, but darn, it didn't work.

The hasattr() call didn't throw an error and complain that it wasn't applicable to dictionaries. It just returned False on every test I wrote. OK, I understand that the right way to know if a dict has a certain key is to write if 'Result' in item.... But why didn't hasattr() complain, if it didn't "do" dictionaries?

The answer, I think, is that everything in Python is an object of a class. The hasattr() function interrogates objects, and a dict is an object of class dict. It actually has some attributes such as __repr__ and the like. But its keys are not attributes. It's just that I am trying to use a dict as if it were a "record" in the Pascal sense, which isn't its intended use.

So I thought to myself, is there something that is more like a record, or some way I could make this API more like an old assembler macro call? Well, there is the namedtuple. Using a namedtuple I could do something like this:

## the translator-coder would import a support module with...
from containers import namedtuple
dialog_item = namedtuple('dialog_item',['type','label','tooltip','result'])
## in the Translator module the coder could then write,
OMIT_TABLES = dialog_item(type = 'checkbox',
                          label = 'Omit tables?',
                          tooltip = 'Check this if the translation should skip /T tables',
                          result = False)

Which is an even nicer interface, has less of the fuzzy look. Bonus for me, there's less to check, as there's no question of wrongly spelled keys. But a big problem, there's no way to omit any keys, either. A namedtuple "factory" like dialog_item above will throw an error if it is called with one of the defined keys not supplied. That's not good, because for example, the tooltip should be optional. And some dialog types need additional fields (like "min" and "max" on the Number type) that should be omitted for the other types.

Well, heck. All that namedtuple() is doing, is declaring a class. It's a meta-class, it generates class definitions. So I could with a bit of thought, just code up a class definition with its own initializer that did allow attributes to be optional. Which would, one, be cleaner than making the coder write all those quote-marks; and two, allow me to do validity checking at declaration time.

So back to the drawing board on this API.

"Watson" reads this blog...

Following a link from a thread on Reddit, I found out I could put a sample of my writing into a linguistic analyzer powered by the IBM "Watson" computer (or is it an algorithm?)

Upon reading 500 or so words from the previous post, "Watson" concludes that:

You are inner-directed, shrewd and can be perceived as critical. You are authority-challenging: you prefer to challenge authority and traditional values to help bring about positive changes. You are independent: you have a strong desire to have time to yourself. And you are reserved: you are a private person and don't let many people in. Your choices are driven by a desire for revelry. You consider achieving success to guide a large part of what you do: you seek out opportunities to improve yourself and demonstrate that you are a capable person. You are relatively unconcerned with tradition: you care more about making your own path than following what others have done.

Not sure where that "desire for revelry" comment comes from. Otherwise—fair enough...

Also: I put in samples from a couple of other blogs (J.T. Eberhardt's and Dana Hunter's) and the results were extremely different from the above. So its analysis may not be "true" but it is certainly non-trivial.

Finally! All platforms bundled.

Thursday, my Elance contractor delivered hunspell for both 32- and 64-bit Windows and Python 3.4. That dropped in and worked fine, and PPQT ran nicely from the command line. So then I could begin working with PyInstaller on Windows 7. Yesterday and today I discovered and circumvented four bugs in it, all unique to running under Windows, or the combination of Windows and Python 3.

First off there were two of the "hook" files that used a wrong path to import the PyInstaller windows "utils" module. Clearly at some recent point that module was moved within the PyInstaller folder, and somebody forgot to update all the hook files.

Next, it ran but the bundled app couldn't start, "module SIP not found". Now, every PyQt5 module needs SIP (the C++ shim that PyQt uses to cross from Python to the Qt binaries), and every one of the several PyQt5 "hook" modules that was being called named "SIP" as a hidden-import. Why wasn't it being bundled? I did not resolve this question, but I did circumvent the problem simply enough: I just added --hidden-import=sip to the PyInstaller invocation line. That was all it took to make the bundled app run, and wasn't that a lovely sight?

While investigating that, I tried to use the pyi-archive_viewer script that is included with PyInstaller. It lets you examine a bundled app to see what was actually included in it. Or it should; but I quickly found that it couldn't execute one of its basic functions, because it was trying to compare a user input string against a class member that was in bytes format. In Python 2, that worked. In Python 3 it doesn't, because Python 3 requires a clear distinction between strings of bytes and strings of characters, which are Unicode. It's one of the most common issues when converting from Python 2 to Python 3, and this comparison had been overlooked. I reported it and applied a quick one-line source change to get around it.

Once I patched that point in the archive viewer, it immediately turned up another error: when it tried to open a sub-archive it threw a run-time error exception because some "magic number" that it used as a signature didn't match. I traced this far enough to see that the magic number calculation had a three-level if statement, in principle saying "if this is Python 2, do it this way; elif this is Python 3 and the version is less than or equal to 3.3, do it that way; else it's Python 3.4 or above and do it this other way." I'm pretty confident I'm the first person to try this code on Windows and Python 3.4, so I just opened an Issue pointing to that code. Having gotten around the missing SIP problem, I no longer needed the archive viewer so I moved on.

One more step, then. I could bundle to a folder; but could I bundle to a single file .exe? Preferably one using my cute little Marvin icon? So I ran PyInstaller with that option—and it threw an exception. Oh, pooh. The exception was another very typical Python 3 compatibility problem, "str type does not support buffer protocol". This error gets thrown whenever you try to feed a string type to a file that has been opened with the "b" raw-bytes mode. In Python 2 you could do that because both str and byte types were aliases for a C char *. In Python 3, bytes still means that, but str means "16- or 32-bit Unicode characters" and you can't just feed them into a bytes file. You have to tell Python how to encode the characters into a byte-stream, for example by coding bytes(str_var.encode('UTF-8')).

I traced the error to a call to the win32api module. PyInstaller was trying to update a portion of the Windows "manifest" (whatever that is) using the UpdateResource Win API call. The win32api module is another open-source project; PyInstaller is just using it. And that module was accepting a string type as an argument to this UpdateResource method, and then (it appears) trying to feed that string into a file opened as bytes, and causing an exception. The bug is in that module. But I circumvented it by changing the code of PyInstaller so that it passed the string encoded to bytes.

And with that monkey-patch thrown on, it ran and produced a lovely single-file PPQT2.exe file with cute little Marvin icon.!

So now I have successfully bundled the app for all three platforms and put them up on my Public dropbox folder. There is nothing but nerves standing between me and announcing the availability of the alpha test publically. I will probably wait until Monday to do that.

Importing translators!

Ran up a little test app to make sure I know as much as I thought I knew about dynamic importing. Everything I know about this, I learned from working on PyInstaller. It has a library of "hooks", small modules that modify the loading process for a specific module. When it finds an import for modname, PyInstaller looks in its hooks folder for a file hook-modname.py. If there is such a hook, it loads that file of Python code into a namespace and looks at the namespace for certain things, such as a global "datas" that can be a list of data files to be loaded when any bundled app imports modname, or a function "hook" that it can call to edit the importation of modname.

It was knowing about this general pattern — load source into namespace, interrogate namespace attributes, call functions in namespace — that made me confident that I could support a variable number of "translator" modules, and even permit users to add new translators in the field.

In the actual app, the File menu will have a sub-menu "Translators". This sub-menu will be prepared at startup. The main window will call a function that populates a QMenu with names of translators. Here is the approximate code of that process.

The outer function will get the Extras path (as set in the Preferences) and look in it for a folder "Translators". It makes a list of all items in that folder and passes each to the following.

    def add_xlt_source( fpath ) :
        if not os.path.exists( fpath ) : return
        if not os.access( fpath, os.R_OK ) : return
        fname = os.path.basename( fpath )
        if not ( fname.endswith( '.py' ) or fname.endswith( '.pyc' ) ) : return
        # It exists, is readable, and ends in .py[c]. Try to load it into
        # a Python namespace.
        xlt_loader = importlib.machinery.SourceFileLoader( fname, fpath )
        print( 'getting namespace', fname)
        xlt_namespace = xlt_loader.load_module()
        # if it is a proper Translator, it has a global MENU_NAME
        if hasattr( xlt_namespace, 'MENU_NAME' ) :
            act = submenu.addAction( xlt_namespace.MENU_NAME )
            act.setData( xlt_namespace )
            act.triggered.connect( run_xlator )
            submenu.setEnabled( True )

The key is the one statement xlt_namespace = xlt_loader.load_module(). This performs an import. It executes all the statements in that source file. (Some of those might raise exceptions, so probably that statement should be in a try/except block.) The returned value is a Python namespace that represents everything declared in that module: its global variables, its classes, and its defined functions.

One can interrogate the namespace with hasattr(). In this case, a Translator has to define a global that is a string (this should be tested!) to use as the menu choice that invokes that translator.

If the module passes this test, the code makes a QAction with the name from the module and adds that action to the sub-menu. The menu action's "triggered" signal is pointed at a function to handle invocation of that translator, and the namespace itself is stored in the action as arbitrary data.

Here's the current stub of the run_xlator function.

        space = self.sender().data()
        print( space.MENU_NAME, getattr( space, 'DATA', '(no data)' ) )

This is a "slot" invoked from the "triggered" signal that is generated when the user selects that item on the menu. It must be part of a QObject-derived object. It can call QObject.sender() to get a reference to the object that created the signal, which in this case can only be the QAction from the sub-menu. The QAction has a data() method that returns the namespace that was stored in it with setData(). That's everything defined in the module that was loaded, so here we print two global values, one we know exists, and one that is optional.

For test purposes I've set up two files in the Translators folder. One is not a Translator,

'''
Test module that is NOT a translator.
'''
print('Non-translator module executing anyway!')

The other one is.

'''
test translator module
'''
MENU_NAME = 'Wahoo!'
DATA = 'Some Data'
print('Wahoo executing!')

Here's the output of a test.

getting namespace not_a_translator.py
Non-translator module executing anyway!
getting namespace xlt_wahoo.py
Wahoo executing!
Wahoo! Some Data  (from run_xlator)

Keep the blog alive...

I follow Carl Claunch's Rescue1130 blog. Besides enjoying the technology he writes about, I have admired his persistence in blogging daily, often 7 days a week. He has a full-time job that involves frequent travel, plus volunteer work at the Computer History Museum, and this absorbing hobby of restoring the 1130, yet he has been posting something every day.

Well, this week he has suddenly went silent. He posted Friday, but not Saturday or Sunday or... wait, that's it. Today is Monday. I found myself actually worried; is he well? Which is stupid, the man has probably just taken a weekend off (no doubt to the relief of his long-suffering family).

Then I realized with some guilt, that after a fair spell of posting most weekdays, I went silent. What about my loyal readers? Are they concerned for my health, or irked at my laziness? Blogging is a responsibility!

Right, so the most significant thing I've been doing the past week is contributing, in a clumsy and halting way, to the PyInstaller project. After a long period of relative quiescence, it suddenly sprang to vigorous life in the past month. Several contributors began posting issues and pull requests to fix their issues. The lead maintainer, Hartmut, became extremely active in response, commenting on the issues and pull requests, rejecting some, accepting others. Most importantly, he did the job of rebasing the dormant Python3 branch onto the current Develop head, so it picked up the maintenance it had missed.

So I tried to use it, and found several minor bugs which I fixed and got a complete working PPQT2 bundle for both Mac OS and for Ubuntu 14.10 (32-bit and 64-bit).

Then I put those fixes into a pull request, but it wasn't right, so Hartmut very patiently directed me in how to make it right, and after maybe three tries, it worked and those fixes are now in the official Python3 branch of PyInstaller, yay me.

Only bundling for Windows remains and I can announce an alpha version of PPQT2. The Windows bundle was a major hurdle for V1, mainly because PPQT requires the Hunspell spell checker. Unlike the other packages PPQT needs (regex, natsort, sortedcontainers) which are pure Python, pyhunspell is a Python-to-C++ wrapper over the API to the Hunspell library. Which means its main component is a C++ source that has to differ between Python2 and Python3, because the Python-to-C API changed between versions. There's a user-patch that supposedly does that; but it is not at all clear whether that patch applies to the current source. And then the source has to be compiled with MSVC at a particular level (2010, 64-bit) to match the level used to compile the official Python 3.4 release. I understand all these words but have zero experience doing anything like this.

What I did for V1 was to go on elance and hire it done. Money very well spent. And I'm doing exactly that again; I posted the elance job this morning and have one inquiry already. This time I will make sure that the updated code and the DLL get sent to the maintainer so others can use them.

When that's all wrapped up, hopefully by mid-week, I should be able to run PyInstaller on Windows 7 and get a clean bundle of PPQT2.

Meantime, I need to get to work on the remaining functionality. That means, for comfort and convenience, being able to code on my laptop. Much as I love my 27-inch iMac, I can't take it to a coffee shop. The laptop had fallen behind the desktop system in Qt and PyQt versions. So what else I've been doing this morning is installing Qt5.4.1, and the latest SIP, and PyQt5.4.1, on this laptop. The very lengthy PyQt make is chugging away as I write.

Last week the task of editing the V2 "suggested workflow" document brought me face to face with a flock of usage issues I had been postponing. My intent is that PPQT2 will be a front end that works smoothly with the "Ppgen" markup system that is becoming popular at the U.S. PGDP site, and with the "Fpgen" markup convention that is used by DP Canada. And that means looking ahead and asking myself, how do those markup systems handle things like block quotes, right-aligned strings, blocks of centered lines, and tables? Because I don't want to direct my users into doing things that would cause conflicts, if they decide to move to one of those markups. OTOH, I want to get the users to use a syntax that will be easy to code in the Translators that will convert to those markups -- or directly to ASCII or HTML, if they use the direct Translators I will provide.

That's been an interesting exercise and is not complete yet. It's kind of an annoying task for a couple of reasons. One is it forces me to deal with these systems that are accomplishing exactly the same damn results, but could they use any kind of standard markup to do it? Oh no of course not; they had to invent their own bloody markup language.

Another is that those markups are not superbly well documented. I was a professional technical writer and naturally have high standards for this. So I have to bind and gag my inner editor so I can just read the damn stuff and not waste time trying to rewrite it as I go.

So where are we, exactly?

Been quiet for a while, sorry. I spent several days doing the initial stages of post-proofing a moderately complex book, using the current PPQT2. In the process I found and fixed several minor bugs. By the end it was working quite smoothly, fully the equal of V1. I can use this tool. And will, if I can just get it finished and shipped.

Concurrently with using the program, I was documenting it. I already set up the help file, but that is a summary organized around the UI: menu by menu, panel by panel. An equally important, perhaps more important document is the task-oriented "suggested workflow" document. That takes the reader through a step-by-step process of post-processing a book, showing how to apply the features of PPQT to perform each.

There was a suggested-workflow for V1, of course. For V2 the initial work steps are the same with only minor changes due to changes in the menu structure. But being an obsessive scribbler I had to rewrite them anyway to be easier to read and more terse.

But a number of changes come in the sequence of later steps. V1 has its own ASCII reflow and HTML conversion built-in. For V2, both of those functions are handed off to Translators (that have yet to be written to an API yet to be implemented). I expect that the most-used Translators will be ones that convert to Ppgen and to Fpgen, markup languages unique to DP and DP-canada respectively. Those markups are used to feed into batch conversion apps that produce the ASCII, the HTML, the EPUB. So the documentation has to assume that the user is aiming toward a smooth conversion to another markup -- not aiming toward generating the final product within PPQT2. (Of course, I expect the user will continue to edit the Ppgen/Fpgen document in PPQT2. There are several advantages to doing so. But the responsibility for generating HTML, or for properly formatting an ASCII etext, falls on that package and on its documentation.)

Anyway, this changed my approach in documenting the tasks of post-processing. But I got that pretty well done.

On the shipping front, which means bundling a Python app as a self-contained package, there has been some progress. Hartmut, the top maintainer of PyInstaller, stepped up and took over the task of rebasing the Python3 code branch onto the current Develop branch. Or so it seemed...

Today I applied the latest PyInstaller to generating a Mac app, and it worked splendidly! If you want to try it, here's the link. So that was good.

Then I moved to Ubuntu. Downloaded the Python3 branch and installed it, ran it, and hit exactly the same problem (the bundled app dies looking for "orig-prefix.txt". If I hack around that, it hits another problem. Both come out of the same little stretch of code in compat.py and hook-site.py. When I compare these two source files between the Develop and Python3 branch, they are very different. Why? I thought when Python3 had been rebased onto Develop, these issues would disappear. I put a query on the PyInstaller list and stopped for the day.

Dang, so close.