Friday, June 19, 2015

HTML Translator going together fast

I spent several hours coding up an HTML translator and have it 75% complete. Another coding session will finish it; then I'll have to test it a wee bit. Finished by Tuesday, I reckon.

I only found one awkward spot in the design of the Translator API. An Illustration markup like this:

[Illustration:fig_55.png|fig_55.jpg Fig. 55 The whang-dangle is..

Produces a sequence of "events" as follows:

OPEN_ILLO with the two filenames
OPEN_PARA
LINE "The whang-doodle..."

So at the time I am generating the code for an image div, I have only the filenames. (Passing the filenames after the colon is a PPQT extension. If the user doesn't do that, the Translator has no way to get them.) The HTML Translator can use this to build a nice opening string of:

<div class=image>
<a href='images/fig_55.jpg'>
<img src='images/fig_55.png' /></a>

However, it would be a useful service if it could also generate alt= and title= attributes from the starting text of the caption, for example alt="Fig. 55 The whang-dangle.... But that can't happen because the opening text of the caption will not arrive for two more events.

I thought about postponing actual output of the image markup until the first paragraph arrived, but that would make the code just horribly ugly. On every OPEN_PARA event you'd have to ask, is this the start of a caption?

I put in a similar but lesser kludge in translating a Footnote. The OPEN_FNOTE event has the footnote key value, so it can generate the target anchor and the back-link to the footnote reference. Unfortunately all that goes inside the first paragraph of the note:

<div class='footnote'>
<p><a id='Footnote_X'></a><a href='FNref_X'>[X]</a>

So here is the <p> being put out ahead of the coming OPEN_PARA event. So I had to create a switch footnote_starting and test it in the OPEN_PARA code, and when it is on, not generate anything and turn it off.

But by and large, the HTML Translator just kind of fell together. You just consider each of the 34 event codes in sequence, write a little bit of code, and repeat.

Tuesday, June 16, 2015

Just about ready for official release

I committed all the translator material to github. Except (OMG!) it appears I forgot to add the two new modules, translators.py and xlate_utils.py. OK, done. So that is all wrapped up and documented very well if I do say so (and as a retired tech writer, I think I know when an API is properly documented). I'm very pleased at how I used a formal syntax to define and verify the document structure. The code of translators.py is clean and well-organized. The mechanism for defining and displaying an "Options Dialog" is simple and (I hope) understandable.

There are parts of the xlate_utils.py module that I'm not quite so proud of. The tokenize() function is going to be fairly heavily used and I confess there are parts of it that are rather ad-hoc not to say downright kludgy. And not superbly well tested, yet. However, in coming days I'll be writing an HTML translator that will test it.

That done, on Monday and Tuesday I turned to my list of issues and resolved most of them. One that came out better than I originally expected was this: PPQT2 expects input files to be encoded UTF-8. The user can get a Latin-1 file correctly opened by renaming it to a suffix of .ltn, but if that is not done, the file will be input through the UTF-8 codec. Some special characters will not decode right and will be replaced with Unicode \ufffd, the "replacement character". And if this isn't noticed right away, and the file is saved, there's permanent loss of data.

So I very much wanted to catch this error early and warn the user. But how? I researched the methods of QTextStream, QFile, and QTextCodec. I know that while QTextStream is executing a readAll() call, it must use a QTextCodec.toUnicode() function. That function is capable of returning a count of invalid characters, but there doesn't seem to be any way to find it out.

It looked as if the only ways I could use to find out if the file decoded properly would be either, one, to read it with Python, in which case the readall method would throw an exception; or two, to use QTextStream.readAll() into a string and search the string for replacement characters. Either method would require me to change the API between the main window and the Book, or else to read the possibly-large document file twice.

Then it dawned on me that the QPlainTextEditor has a perfectly good find() method. All I had to do was, in the Book just after it has loaded the editor from the file, to call the editor's find() to look for a replacement character. One hopes the search fails. But if it does not, I can notify the user with a warning message, including the character position of the first replacement character. I made the warning message detailed and also included a pointer to the Help topic.

Another long-standing issue, more of a major loose end to clean up, was logging. There are lots and lots of log messages being issued all over the program. But the logging output was going nowhere. I had initially thought that I would add argument parsing, and use it to support --log-path= and --log-level= parameters. But that's dumb; I'm packaging the Windows and Mac OS versions as clickable apps, with no command-line input. And the Linux version doesn't have to be launched from a command line. So I did some study and reading and chose writable locations for log files based on the platform: /var/tmp for Linux, ~/Library/Logs in Mac OS, and \Windows\Temp in Windows. Oops, I just realized I committed the code for that, but I should also update the Help file to document it. Or maybe put it in the README for each version?

Anyway: tomorrow is Museum day. Thursday I need to spend most of my free time studying the docs for a new volunteer gig. I have an online training session for that at 4pm that day and I want to be prepared. But Friday I will start coding an HTML Translator. Should have that done early next week. By the end of next week I should have PPQT2 packaged up ready to announce. Can't wait.

Saturday, June 13, 2015

Translator API complete

So I continued to code in bursts over the week away in Seattle, and today at home I was able to put in several hours and finished the Translator API, including documenting it. This makes PPQT2 just about functionally complete. What I need to do over the next week—to be honest, probably two weeks—is:

  • Commit all this work and check it in to github,
  • Write a real Translator to serve as an example—I think probably an HTML one as it would be the easiest,
  • Clean up all the "issues" I have been posting for myself on the github page,
  • Create 32-bit Win7 dev and test VMs to replace the 64-bit ones I've been using,
  • Bundle the package and release it—as a beta? Or final? No, Beta.

That done, I will spend the early part of July awaiting bug reports, writing another Translator, either fpgen or ppgen, and finally implementing drag-out tabs. By mid-july I intend this project to be done.

Sunday, June 7, 2015

Test Translator running

My coding time has been limited the past few days and will continue to be so as we fly off to Seattle for a few days to visit relatives, although I'll get in some. I hope by this time next week the complete Translator interface will be working.

What is working now is: document parsing, finding Translator modules, building the submenu, responding to the selection of a Translator from the menu, displaying a Translator's option query dialog, and calling the Translator's initialize(), translate() and finalize() entries with appropriate arguments. A special demo Translator named "Testing" exists and accepts calls at those entries. All it does is produce lines of output documenting what it is called with, including displaying each document "event" it gets.

So far, the output of Testing (or any other Translator, should one exist, but none do) is only dumped with print statements. But a whole lot of machinery had to work in order to get this far.

One bug I had to work through was a classic "lost reference" bug. In most Qt interfaces, when you build a widget and give it to another widget to manage, the manager widget becomes a "parent" and retains the child widget, so it continues to exist. That's not the case with menus. A QMenu does not take ownership of a QAction or a sub-QMenu. I forgot that. I modified the main window to call the translator support module to build the submenu of translators, but didn't save the reference returned. Just handed it to the File menu's addMenu() method.

The result was an interesting bug: the Translators... sub-menu would appear the first time you opened the File menu. Then the second or third time (or sometimes the first time), there would be no submenu. It would disappear. It all depended on when the Python garbage collector got around to demolishing the submenu QMenu object. That took half an hour of trying and thinking before I finally twigged, did a classic Doh! face-palm, and did a one-line change to have mainwindow save a reference to the submenu.

Another problem was my lack of experience with Python generators. A generator is a function that contains a yield statement, but what is tricky is, it is not that function you call in your for-loop; it is that function's return value. And the function call gets parenthesized arguments, while the iterator does not. I almost had it right but had to review the python docs and recode at two different points.

I used a generator in my "scanner" that overrides the built-in token scanner for the YAPPS parser. My scanner class took an iterator—which is anything that responds to the next() built-in function—as an initialization parameter. It calls next(iterator) to get the next line of the document being parsed.

For unit-test purposes, it was initialized with a Python StringIO object loaded with a triple-quoted literal. A StringIO responds to next() with the next line of stuff. But for the real thing, I needed to pass an iterator that yields the next line of the real document.

Months ago I coded such an iterator into the editdata module. It's a simple function:

    def all_lines(self):
        tb = self.begin() # first QTextBlock in document
        while tb.isValid():
            yield tb.text()
            tb = tb.next()

There's the magic yield statement. But when I passed editdata.all_lines to initialize the scanner, I got an error about cannot iterate a method. What I had to pass to the scanner in place of a StringIO was not all_lines but all_lines(), the returned value of calling the generator. That's the iterator that can respond to a next() call.

I made the exact inverse goof in quickly whipping up the Testing Translator. The translate() method of a Translator is passed an iterator that returns "events" it is to process. I was correctly passing it the result of calling my event-generating function with a yield statement. But in the Translator where it invoked the iterator, I coded for (code, text, stuff, lnum) in event_iterator() and got a different error message, about "cannot call an iterator". Had to remove the parens, is all.

I want to modestly point out that when a Translator gets such a Python error, it is caught and displayed with some helpful info in an error dialog to the user. That code's working too.

What's not working? Two tricky bits. I've promised in the Translator API doc that there will be a tokenize function that breaks a line of text up into mini-events so that the Translator doesn't need to duplicate the logic to extract such things as superscripts, subscripts, footnote anchors, and markups like italics. That will take a bit of thinking to make clean and bulletproof.

And the final step of doing translation: taking the text that the Translator writes into a MemoryStream object, combining it with metadata extracted from the source Book object, and creating a new Book object with the translated text and at least some of the source metadata. That's going to take some fiddly code, probably involving JSON editing.

And finally, back in the mainwindow code, installing a new Book and making it visible for editing. That's just a special case of the File>New operation, hopefully.

Saturday, June 6, 2015

Making Translators real -- and why I hate Markdown

"Smelling the barn"—an expression that horsey people use to describe the sudden enthusiasm of a horse when it nears the end of a long ride. Do horses really do that? Whatever; I'm starting to smell the end of this project. I'm alternating between writing the code that will call a Translator and updating the document that tells the Translator's author what will happen. And it all feels like it's coming together at last. I have serious hopes of delivering the 1.0 product by the end of this month!

But the API doc... I started writing it in Markdown. It's simple; it's designed for documenting code; it's also slightly less stupidly inconsistent than reST. But then I tried to process it for display. Aagghhh!

One would expect that John Gruber, the inventor of Markdown, could write an accurate preview widget—wouldn't one? So the first place I tried my document was at his "markdown dingus". And it is terrible! Although the sidebar clearly states that a four-space indent means a code block, my document has numerous indented code blocks and Gruber's "dingus" renders them all in normal text font. It preserves the line breaks in these sections, at least, but does not preserve the white-space indents.

Worse, it does not automatically escape either underscores or HTML codes within a code block. So where the code contains an underscore in a variable name, the underscore disappears and the "code" suddenly turns italic. And where there's a bit of HTML in the code block, it takes effect. It's just a mess. Shame on you, John Gruber.

There's a pretty decent Markdown extension for Chrome. It renders my code blocks fine. But I'm just confused because for Gruber's widget I have to escape underscores inside variable names, but then the Chrome widget renders the backslash. I'm going to assume that the Chrome extension is accurate (whatever "accurate" means when discussing Markdown). But Markdown is a mess, a very unsatisfactory medium. Ordinary manual HTML, like I do in this blog, would be less confusing for sure.

Tuesday, June 2, 2015

Implementing translators 1: API spec

With the document parse coded and tested, I turned my attention to coding the function that will actually invoke a Translator when one is selected from the menu. I spent a couple of hours tidying and rearranging the code of translators.py and coding the framework of the function. My approach to this is to write a comment prolog saying what I'm going to do, and revising that over and over as I realize why I can't do that, at least not in that sequence, so I'm actually going to do this, etc. etc.

The first steps are known and I could code them: find if the Translator had a user-options dialog and presenting it; parsing the document. But then: time to start actually calling the Translator. This brought me face-on to the question of what exactly is the interface to the Translator. I had a detailed sketch of a design from weeks ago but knew more now. So I changed modes and spent about 3 intense hours rewriting the API spec. I made several on-the-fly decisions that simplified it from before. Really, any competent Python programmer—who also understands the full range of things that can occur in a DP document—and who also understands in detail the syntax of the target document format—should have no difficulty writing a Translator.

When I put it that way, it suggests a Venn diagram of three circles for which the little triangle where all three overlap might be rather small. But for the select group of people in that happy spot—no problem.

Monday, June 1, 2015

Parsing a document, 7: testing (and ranting at Enum)

I am now beginning to test my document-parsing code and things are going very well. It amounted to about 250 LOC (plus the code generated by YAPPS, another 200 or so). For first execution of brand new code things went well. After I picked off 6 or 8 stupid coding errors (like: defining some regexes as class members and forgetting to use "self." to reference them) and a couple of small logic errors, it is happily parsing a simple document.

One problem I ran into that took a bit of finagling was this. The generated parser comes from a "grammar" file. I've shown some preliminary grammar code in previous posts. One tricky production is the one for heads:

rule HEAD:      EMPTY {{print('head...')}} ( PARA {{ print( "...3") }}
                            | EMPTY EMPTY PARA+ EMPTY {{ print("...2") }}
                            )

The items in {{double braces}} are Python statements which YAPPS will insert into the generated parser code at the point where parsing reaches that part of the production. In that code the statements are print() calls. But what I really needed was this:

rule HEAD:      EMPTY {{ open_head() }} ( PARA {{ close_head(3) }}
                            | EMPTY EMPTY PARA+ EMPTY {{ close_head(2) }}
                            )

In other words, call functions of mine that will set up a start-heading work unit, and, when the type of heading is known—only after processing the paragraph(s) of text within the heading—back-patch the open-head unit with the type of head it turned out to be, and append the close-head unit.

Well, that code died with an exception because "function open_head() not found." Wut? I was importing the parser with:

from dpdocumentsyntax import DPDOC

which should make the parser class part of the active namespace where the functions like open_para() were defined. But no. I tried several ways to work around this. You can include blocks of code in the generated parser, but if I defined the helpers like open_para() there, they could not see the globals like the WORK_UNITS list they had to modify. Eventually I had to do it in a not very pretty way,

import dpdocumentsyntax
dpdocumentsyntax.open_para = open_para

That is, manually inserting those definitions into the imported namespace.

Anyway, as it parses, the code builds a list of "work unit" objects that will eventually be fed to a Translator as "events". A typical sequence of work units, or events, would be,

  • Open head(2) (Chapter head)
  • Open paragraph
  • Line (text: "CHAPTER ONE")
  • Close paragraph
  • Close head(2)
  • Open paragraph
  • Line (text)
  • Line (text)
  • Close paragraph

And so forth. There are all told 30 different possible "events" and I expect to pass each to a Translator with a code signifying what kind of event it is, e.g. Open Paragraph, close BlockQuote, or Open Illustration Caption, etc. So how should these codes be defined? Obviously there must be names for them, like OPEN_PARA, CLOSE_FNOTE and so forth. And obviously these will be in a module the Translator can include, perhaps so:

from xlate_utils import EVENTS

Then the coder can make decisions by comparing to EVENTS.OPEN_PARA and the like.

Looks like a job for an Enum, right? The Enum "type"—it isn't a type—was added to Python in version 3.4, and having played with it, I cannot fathom why they bothered. It has to be the most useless piece of syntax ever. But check this out.

from enum import Enum
class ECode( Enum ):
  VAL1 = '1'
  VAL2 = '2'
ECode.VAL1
<ECode.VAL1: '1'>
'1' == ECode.VAL1
False
edict = { ECode.VAL1: 1, ECode.VAL2: 2 }
edict
{<ECode.VAL2: '2'>: 2, <ECode.VAL1: '1'>: 1}
edict['1']
Traceback (most recent call last):
  File "<string>", line 1, in <fragment>
builtins.KeyError: '1'
ECode.VAL1 < ECode.VAL2
Traceback (most recent call last):
  File "<string>", line 1, in <fragment>
builtins.TypeError: unorderable types: ECode() < ECode()

Now, for something completely different:

class HCode( object ) :
  VAL1 = '1'
  VAL2 = '2'
HCode.VAL1
'1'
HCode.VAL1 == '1'
True
hdict = { HCode.VAL1 : 1, HCode.VAL2 : 'to' }
hdict
{'1': 1, '2': 'to'}
hdict[ '2' ]
'to'
hdict[ HCode.VAL1 ]
1
HCode.VAL1 < HCode.VAL2
True

What I'm saying is that a simple class definition accomplishes everything that the "Enum" class does, and also has "real" values that can be compared and ordered. There is the one tiny drawback that a user could assign to HCode.VAL1 but that can, I believe, be prevented by adding a decorator.

So I will be providing the 30 event codes as a class EVENTS that is really a class and performs what a C header file does: give names to arbitrary literals.