Looking ahead to Pandoc

This is to order my thoughts about output formats and markup systems, and to gather links to these in one convenient place.

History

PPQT is intended to support the work of volunteers finishing etexts for Distributed Proofreaders, aka PGDP. PGDP was one of the first "crowd-sourced" volunteer sites on the internet, organizing thousands of volunteers to find the typos in OCR images of public-domain texts, one page at a time. At the end of the process, a different set of volunteers, the "post-processors" or PPers, have the job of splicing together the individually-proofed pages of each book to make one smooth etext. That's the task that PPQT aimed to assist.

The original PGDP workflow ended with an ASCII etext, no more. There are hundreds (thousands?) of PGDP-proofed etexts at Project Gutenberg. By 2002 or so, most PPers also prepared HTML versions of their texts. And in recent years there's been demand for other formats such as EPUB.

Markup Systems

A text passing through PGDP gets formatted with a particular markup style documented in the Formatting Guidelines. Although PGDP did not label the guidelines as a "markup system" that is what they constitute: a set of rules for representing a book's typography and layout in a plain text document. PGDP never gave their markup system a catchy name; let's call it DPM.

dpm

DPM can be compared to other plain-text markups such as Markdown and reStructured Text. It comes off quite well in these comparisons. The other markups were devised by (mostly) programmers for use in (mostly) documenting code, they don't support typography beyond emphasis, and layout beyond code-blocks. Some of the things that DPM supports and others do not include footnotes, poetry (in the sense of being able to specify line breaks and indentation), and simple right-alignment of text, as in a citation within a block quote.

fpn

In recent years, PGDP volunteers motivated in part by the need to auto-convert etexts to new formats such as EPUB (and in part, I'm sure, by simple N.I.H. syndrome), have devised new markup styles. One is fpn devised by Robert Frank (rfrank at PGDP) and announced in February 2014. This markup uses different syntax to support the features of DPM, and adds a number of minor features. In general Robert Frank favored a terse syntax reminiscent of 1980s TROFF syntax. It would not be difficult to convert a DPM-marked text to one that is marked up with basic fpn using search and replace; for example chapter heads in DPM are marked with four newlines, and in fpn with a leading .h2.

fpgen

A bit earlier, in July 2013, the independent volunteers of PGDP Canada announced their own new markup style, fpgen. Documented in the DP-canada WIKI, fpgen is also the work of an "rfrank", in this case Roger Frank, who tended to favor an XML-like bracketed syntax. Again it would not be difficult to convert a DPM text to an fpgen one; for example a DPM chapter head marked with four newlines would become <heading level='1' id="ch01">Head Text</heading>

my-dpm (blush)

I am not immune to N.I.H. and the temptation to define markup syntax. In PPQT version 1 I supported a number of extensions to DPM, including right-aligned text and a syntax for tables. I designed these features based off of PPQT's model, Guiguts, which had its own simple extensions of DPM. For example, in DPM a block quote is

/Q
Quote text...
Q/

Guiguts extended this to allow specifying the first, left, and right indents so:

/Q[8,4,12]
Quote text with 8-char first indent, 4-char left indent, 
and 12-char right indent...
/Q

My version supported in PPQT V.1 allowed instead,

/Q F:8 L:4 R12
Quote text with 8-char first indent, 4-char left indent, 
and 12-char right indent...
Q/

Guiguts had a simple ASCII table markup; I extended it with additional syntax for column alignments and widths. I also added /R..R/ for right-aligned text and /C..C/ for centered text.

What to support with PPQT?

In a way, the choice of markup hardly matters, because the markup disappears before the book reaches its destination at Gutenberg.org. A marked-up document is a transient state between the original OCR text and the final etext/html/EPUB files. So the choice of markup is merely a convenience for the PPer. It is a way for her to encode decisions about how the book should be formatted: these lines are a poem, these lines are a table; this is emphasized text, etc.

However, the choice of markup is controlled by the software used for creating the final output. Robert Frank has a Python program to convert an fpn text to EPUB and HTML. Roger Frank of PGDP Canada has a, guess what, Python 3 program to convert fpgen to EPUB and HTML.

And both Guiguts and PPQT V.1 have code to convert DPM to HTML.

What should PPQT V.2 do? Should it contain code to convert fpn or fpgen to some other output format? Should it retain the V.1 HTML converter? Or should it be markup-agnostic?

Agnosticism

By markup-agnostic I mean, have only the features needed to make a clean job of finalizing an etext,including:

• Support for image display alongside text,
• proofer's notes saved in the metadata,
• an extensive find/replace,
• the character and word tables (so important for spell-check and finding other missed errors),
• the Footnote panel with essential aids for cleaning and renumbering footnotes,
• automatic calling of gutcheck or (better) bookloupe and a tabular display of the resulting diagnostics,

And just stop there, and say: ok, PPer, now you have a smooth DPM text, you can go on to use the editor and regex find/replace to convert this to any markup you like, and save the file, and process it using software from whomever.

Translators

Another option would be to offer automated translation from DPM to fpn and/or fpgen. And further, it would be possible to foist the job of coding those translators off on the people who want those markups. I've already floated this as an idea to DP Canada: that they could write the fpgen-erator to some API that I could provide.

Enter Pandoc

And then, there's Pandoc. Pandoc is a universal markup-translator. It reads texts in a variety of markup styles, and it writes output in an even wider variety including EPUB, LaTex, and PDF. It is widely used and widely praised, and in principle, could completely replace the programs written by both rfranks, generating any possible desired output format from code that is widely used and supported by an active community.

All that is needed is a way to get a post-processed etext into Pandoc. Unfortunately although Pandoc accepts a number of markups, that list does not include dpm, fpgen or fpn.

Pandoc does offer two general input formats. One is its own internal format represented in JSON format. The other is its own extended Markdown. This markup—let's call it pem for Pandoc's Extended Markdown—supports everything that dpm supports. It does not support quite everything that fpn supports, and I'm not sure about whether it is a superset of fpgen or not.

I can picture PPQT supporting a batch conversion of dpm to pem, for example as a command under the File menu: File > Save to Pandoc, and this as a replacement for the old HTML conversion step. This would convert a dpm text to a pem text and write it. However that wouldn't be much use to a PPer who didn't have access to Pandoc, because only Pandoc supports pem.

If I can work out how to distribute a Pandoc executable with PPQT in any platform, I can also imagine having automated "Save to Epub" and "Save to HTML" commands, that would generate a pem stream and feed it down a pipe to a Pandoc command, with the output to the designated file.

What About HTML?

PPQT V.1 has only one aid for HTML, the HTML Preview panel. When editing an HTML document, you can get a rendering of it in a QWebFrame. It's only a bit more convenient than saving the file and opening it in a separate browser (you avoid having to do a ctl-s in PPQT, click in the browser, click the reload button, then click back to PPQT to edit).

But should V.2 have any HTML support at all? The point in editing HTML inside PPQT was that the auto-converted HTML is awful to look at and needed a lot of hand-tweaking and customizing. But when HTML conversion is pushed off to an external program, whether that is Pandoc or an effort by one of the rfranks, is there any point in editing the resulting HTML output? Or is one supposed to do confine one's tweaking to the marked-up fp[ge]n file, and treat the HTML as a write-only output?

And supposing PPQT retained its HTML preview panel, should it also have a preview panel for EPUB? (Is that possible?) It would be kind of slick if, when you opened a file with an html suffix, you automatically got an HTML preview panel on the right, while if you opened a file with a .mobi suffix, you got an EPUB preview panel...

Also given HTML support of any kind, what about W3C Validation? Back when I assumed V.2 would have its own HTML conversion, I also speculated that it should support an automatic upload to the validation site, automatic download of the error list, and display of that in a panel such that you could click on a diagnostic and jump the editor to the referenced line. Is that still useful when HTML is being generated by an external program? Or is validation and W3C conformity now the responsibility of the external program?

I welcome the comments of any of my readers on these issues.

Now you see it, now you don't

First I added about 4 more tests to the mainwindow script shown in a video in the prior post. One of these revealed an interesting bug. The test was, to use File>Open and pick a file that was already open. The result should be to cause the existing edit view of that file to pop to the front (not to open a second copy of the file).

That worked as it should, but then I combined it with another test. The other test was to choose, not somebook.txt but somebook.txt.meta. This should check that somebook.txt exists, and if it does, open it (treating the book and its .meta file as identical). That test also worked as it should. But when I combined the two, opening somebook.txt.meta when somebook.txt was already open, I got two open copies of somebook.txt. Huh?

It was a matter of when the different checks were made, of course, and a small rearrangement of code fixed it. But it's fun taking an attack stance toward my own code and breaking it.

Then I moved on to the notes panel, and the first test of a disappearing Edit menu and it works a treat. I had to modify the mainwindow so that it offered a module method for retrieving the Menu Bar. Then the notes widget could do this:

        ed_menu = QMenu(C.ED_MENU_EDIT,self)
        ed_menu.addAction(C.ED_MENU_UNDO,self.undo,QKeySequence.Undo)
        ed_menu.addAction(C.ED_MENU_REDO,self.redo,QKeySequence.Redo)
        ...cut, copy, paste...
        self.edit_menu = mainwindow.get_menu_bar().addMenu(ed_menu)
        self.edit_menu.setVisible(False)

Adding a menu to the menu bar returns a "menu action", and a QAction has a property visible which, if it is set to False, means the action, or in this case the menu, is not shown. Further down in the notes panel code is:

    def focusInEvent(self, event):
        self.edit_menu.setVisible(True)
    def focusOutEvent(self, event):
        self.edit_menu.setVisible(False)

And it works just loverly. Click in the Notes panel, the Edit menu appears. Click in the edit view, it disappears. When I finish building notes view I will go back into Edit view and give it an Edit menu of its own, with quite a few more actions.

Mainwindow test and Sikuli movie

Finally completed a Sikuli script to drive the mainwindow code through much of its function. In the course of this, found several bugs, but that's what testing is for he said through gritted teeth. Then used Screen Cast O Matic to make a 1-minute movie of the test running, because Sikuli is kind of fun to watch.

Roadmap

Where to now? Well, the next piece of code is the Notes panel. When that is in place I will have two text editors that will contend for the use of the Edit menu, and I can experiment with my tentative plan for operating multiple Edit menus.

After that, the Find panel, which if you know PPQT V.1, is a rather complex chunk of UI, and a lot of its code is dependent on [Py]Qt4 features and has to change. No more QStrings, no more QRegExps. All Python strings and the regex library. So that'll be a bit of work.

With Notes and Find, V.2 will be functional enough that a person could use it for editing a book. However, I don't expect to get to that point in this month of June. Unfortunately (for PPQT but not for me!) I'm going to be on holiday all July, so I would guesstimate completing Find around mid-August.

Well, that was easy -- I think

The issue with the focus-in was bugging me during the night. Lying in the dark with my eyes closed, debugging. Not productive! But got up this morning and modified the event handler in my editview, the one that up until a little while ago read:

    def eventFilter(self, obj, event):
        if event.type() == QEvent.KeyPress :
            return self._editorKeyPressEvent(event)
        if event.type() == QEvent.FocusIn :
            self.focusser()
        return False

Slight digression here. Because the actual QPlainTextEditor is part of a layout created with the Qt Designer, there is no way to add any kind of event handler to it. In particular no way to add an overriding keyPressEvent() method, which this editor needs in order to support things like the go-to-bookmark and zoom-font keystrokes. Also no way to provide an overriding focusInEvent() handler which I needed as a signal that it was time to display all the "panels" associated with that particular book (or so I thought).

So instead, the editview module, after initializing its Designer-written UI, installs an "event filter" on it, the above code. An event filter is a "man in the middle" that returns True if it has dealt with the event, or False to say, send it along to the original address for handling. All events directed to the UI widgets—the editor and the line-number and page-name widgets—pass through this code first. It picks off the key presses and looks for special keys, returning False for those it didn't handle.

It also used the FocusIn event as a sign it should call the focussing function it was passed by the Book when instantiated. (Which calls the main window, which displays all the related panels, long story.)

That's what I found yesterday wasn't reliably working. When the editview was created and assigned to a QTabWidget tab, it got a FocusIn. Then it got no more of those until the user manually clicked in the editor. Even if you brought another tab to the front, hiding this one, then brought it back: no events unless it had received at least one real click. You could tell it was in this state because the QPlainTextEdit widget did not display a blinking cursor. After a click, it would. Or, I found out, after a tab key. If you press tab an unpredictable number of times, eventually the new editview would get a FocusIn. I'd tried adding calls to self.setFocus(Qt.TabFocusReason) in several places during initialization, to no avail.

So what I did this morning was add to the above code a call to a routine I'd written some time back, to display the event stream. Here's the event sequence, after opening two books, but before clicking in either of them. Just clicking the tabs to alternate between the two editviews, the events are:

2two_test.txt event type  Show 
2two_test.txt event type  UpdateLater 
2two_test.txt event type  Paint 
2two_test.txt event type  Hide 
3three_test.txt event type  Show 
3three_test.txt event type  UpdateLater 
3three_test.txt event type  Paint 
3three_test.txt event type  Hide 

After having manually clicked in each widget, the one last clicked upon gets FocusOut and FocusIn events, but the others don't. All told, it seems that the FocusIn event is nowhere near as consistent or reliable as I'd assumed. There are no doubt various window-system-related issues that I'm not aware of. (And given that thought, very probably there are platform differences as well!)

What the event trace printout does show to be reliable, is the Show and Hide events. So I simply changed my event filter to say, if event.type() == QEvent.Show... and things began to work the way I expect them to.

Me 'n Sikuli are good; Recursive bug

So it turned out that Sikuli actually had a feature that made all my laborious capture code yesterday unnecessary. Completely so. And on top of that the feature is discussed in their top-level "hello world" tutorial. Maybe I'm losing it. Anyway you can capture parts of another app very nicely, interactively. Which I started doing today and got about a third of the way through a mainwindow test before I got distracted by a couple of bugs that turned up. Which is what testing is for, ok.

One bug was quite interesting although hard to describe. The symptom was that, when two books were open and one opened a third, suddenly the second and third books were sharing the same set of "panels". Same image display, primarily, as that's the only functioning panel, although the other placeholder panels were also swapped. So after opening book 3, it and book 2 had the identical image displays, even though they should show different images. Verrrry peculiar.

Well, long story short, after about a half hour of debugging I realized what had to be happening was that the "focus me" function, which is called when any book's edit window gets the focus in order to display that book's other panels, was being called recursively! It would be half-way through focusing the newly opened book when it got called again to focus the newly opened book. A little more tracing and I figured out why that happened: when the new book's edit window got added to its tabset, it received a focus-in event from Qt, so it called the focus-me routine which was still unfinished. Hilarity ensued.

That was easy to fix, once figured out. But another is still pending. This is another issue of getting the Qt focusIn event. As just recounted, one of those events is delivered to an edit window the instant it is added to a tab set. And later on, when you click on the tab for a book's edit window, it should get another focus-in event. And it does—but only if it has had at least one real user mouse-click.

I put a print in the focus-event handler to prove it. Here's the sequence: open three books. Each one gets its edit panel added to the edit tabset. And each one at that time gets a focus-in event.

Now, click back and forth on the three tabs. Each edit panel is revealed appropriately. But they do not get a focus-in event. Click all the tabs you want; they are displayed but they don't get the event, and therefor they don't call focus-me and their image and other panels don't get displayed.

Now click the mouse in one of the panels. It gets a focus-in event and from then on it also gets a focus-in event every time its tab comes up. No manual click: no events. One manual click: events.

I've tried variations on calls to QWidget.setFocus() during the creation of the edit view, but it doesn't have any effect. Very mystifying.

Tidying up, but trouble with Sikuli

Spent some time cleaning up test files, editing and re-running prior unit tests, and generally making the Tests folder nice. Picked off a couple more minor bugs.

Then started the main effort, a Sikuli script to really test all the mainwindow features accessed by the File menu. I've already been using these in casual tests in coding but I want to see them all exercised in formal fashion. I thought Sikuli could do this, after my previous success in using it with the edit view window. Wow, that was April. How time flies.

So it turns out Sikuli isn't so helpful when testing menus in the Mac OS menu bar. The problem is the interactive capture of target app menus. Running in the Sikuli IDE you want to say "click on this" and it gives you a view of the whole screen minus the Sikuli windows, and you are supposed to be able to drag over it to select where to click. Unfortunately, the left-hand end of the menu bar still has the Sikuli application's rather lengthy name and no File menu. So, where to tell it to click?

Well, it works to drag over a rectangle about where the File menu will be when the target app is in front, so executing that script does cause the target's File menu to open. But now I want to tell Sikuli, "find( - image of open file menu here - )". But when I go back to the Sikuli IDE and activate the "find" dialog and it tells me to select a piece of the screen—the File menu has disappeared.

This wasn't a problem when capturing the pop-up menu in the edit view window. A pop-up context menu stays open until dismissed. But the File menu from the Mac OS Menu Bar does not stay open when you switch away from the app that owns it. So I'm not seeing how to get Sikuli to verify the contents of my File menu or click on items in it. Put a question in and we'll see.

Unit testing

Spent a pleasant 4 hours writing a detailed unit test driver for utilities.py, the module into which I have chucked everything to do with QFile and QFileDialog and friends. The testing turned up several bugs, which is what it's all about.

This was the module for which I developed my variation on defaultdict, of which I was quite proud. Proud enough that I also posted it in reddit/r/python. And today in exercising the functions that use that clever class, I discovered that it wasn't working! Well, it did work in the sense that it always returned correct answers, but it didn't in the sense that it wasn't doing any caching of results. It always calculated the result anew for every key, quite the opposite of a cache.

This was due to my misunderstanding what the Python docs were saying about defaultdict. In my defense I'll say that the Python writeup is not all that clear. But still, just stepping through the code a couple of times to verify it did what I thought, would have revealed instantly that it didn't. Anyway, I went and updated both this blog and the reddit posting.

That was one bug. Several others were simple typos and mis-codings in code that hadn't be executed yet, typically the code that logged errors. Then I found what looks a lot like a bug in PyQt 5.3. I've posted it to the mailing list and we'll see. The test case I posted is so simple it's hard to imagine how it could be an error by me.