Monday, December 28, 2015

Holiday in the Endless Sky

So for fun I've been playing Endless Sky. It's a nicely made re-implementation of the old EV Nova (which appears to be still available, although I'm not sure it would run in a current Mac OS).

Part of the early grind of Endless Sky, as it was for EV Nova, is to haul commodities from star to star, buying low and selling high, until you accumulate enough credits to buy a better ship. Each star has prices for ten different commodity types, and one of the first questions the player asks is, what route has the highest profit? (In EV Nova, there was one fabulously profitable one-hop route, the only problem being that you were almost guaranteed to be attacked by pirates every time through it.)

Endless Sky's galaxy is defined in a simple text file that is part of the game package (here's the source). I looked at it and thought, hmmm. The galaxy is an undirected graph; the commodity prices are properties of the nodes of the graph. This looks like a job for Python! So one morning I sat down in and in less than two hours, I had code to read and store the galaxy as a graph, and it worked first time! What follows is a slightly upgraded version, so it has about 2:30 invested in it.

import sys
try:
    filename = sys.argv[1]
    map_file = open( filename, 'rt', encoding='utf-8', errors='ignore' )
except:
    print('usage: mapper <file path to the map file>')
    exit(-1)

# map trade names to indexes in a system trade list
trade_indices = {
    "Clothing" : 0,
    "Electronics" : 1,
    "Equipment" : 2,
    "Food" : 3,
    "Heavy Metals" : 4,
    "Industrial" : 5,
    "Luxury Goods" : 6,
    "Medical" : 7,
    "Metal" : 8,
    "Plastic" : 9
}
trade_names = { d : n for n, d in trade_indices.items() }

# The galaxy is a dict of { 'system-name' : System } It is an undirected
# graph in which the edges are system-names in each system's links:
#    for neighbor_name in Galaxy['some-name'].links:
#        neighbor_system = Galaxy[neighbor_name]

Galaxy = dict()

class System( object ):
    def __init__( self ):
        self.trade = [0] * 10 # prices of 10 commodity types here
        self.links = set() # names of connected systems
        self.habitable = False # can trade here?

import regex
# The following matches to a pattern of either
#   verb namestring [nnn]
# or
#   verb "name string with spaces" [nnn]

rx_verb = regex.compile( '''^\s*(\w+)\s+((\w+)|['"]([\w\s]+)['"])(\s*\d+)?\s*$''', flags=regex.IGNORECASE )

# If the current line is "verb name [nnn]" return (verb, namestring, [nnn|None]).
# Otherwise return (None, None, None)

def match_line( line ):
    match = rx_verb.match( line )
    verb = None
    namestring = None
    number = None
    if match is not None :
        verb = match.group(1)
        namestring = match.group(3) if match.group(4) is None else match.group(4)
        number = match.group(5)
    return (verb, namestring, number)

# Read the map, build the Galaxy

in_system = False

for line in map_file :

    ( verb, name_string, number ) = match_line( line )

    if not in_system :
        if verb == 'system' :
            # Beginning a system block. Create a system and file it.
            new_system = System()
            Galaxy[name_string] = new_system
            in_system = True
            habitable = False
            gov_not_Hai = True
        continue

    # Process "link", "trade", "government" and "object" statements
    if verb == 'link' :
        # new_system links to name_string. Note the map has reciprocal links,
        # we do not need to add a reverse link. In fact we couldn't, because
        # name_string may not be in the galaxy yet.
        new_system.links.add( name_string )
        continue

    if verb == 'trade' :
        # new system trade line found, store the price in the system list
        commodity_index = trade_indices[ name_string ]
        commodity_price = int( number )
        new_system.trade[ commodity_index ] = commodity_price
        continue

    # Check "object name" statements. The regex only matches to "object
    # name", not to the more common "object" statements. If a system has an
    # object with a name, it is habitable. Except for Algiebra, which has
    # a named moon but cannot be used for trade.
    if verb == 'object' :
        habitable = 'Watcher' != name_string #anywhere but Algiebra
        continue

    # Look for government starting with Hai, because those systems are not
    # accessible until later in the game (this could be conditioned on a
    # command-line option)
    if verb == 'government' :
        gov_not_Hai = not name_string.startswith( 'Hai' )

    # if the current line is blank, we are at the end of this system.
    if 0 == len(line.strip()) :
        new_system.habitable = habitable and gov_not_Hai
        in_system = False
        continue

    # Note the official map *ALWAYS* has a blank line at the end of a
    # system block i.e. preceding a 'system' statement. If the map file
    # is mis-edited to not have such a blank, this code will merge the
    # trade etc. lines from the following system to the previous one.

# All map lines processed, the Galaxy is built.

With the galaxy as a graph, it took another hour to brute-force the best trade route of one, two, three or four hops. No longer route because the starter ship and the typical freighter can't go more than four hops without landing.

# Return the set of all system names that are n hops out from the given
# system.
def n_hop_targets( system, n ):
    if n == 1 :
        return set( system.links )
    all_targets = set()
    for target in system.links :
        all_targets |= n_hop_targets( Galaxy[ target ], n-1 )
    return all_targets

# Compare the trade price lists from two systems. Return the index
# of the most profitable commodity and the profit margin.
def compare_prices( buy_list, sell_list ):
    profits = [ buy-sell for buy, sell in zip( buy_list, sell_list ) ]
    p = max( profits )
    return profits.index(p), p

# Find the most profitable trade route between a given system and any of a
# set of systems it can reach at some number of hops. Input is the system
# itself, and set of target names.
# return the system name and (outbound buy, profit, inbound buy, profit)

def best_trade_route( system, target_set ):
    home_line = system.trade
    out_p = in_p = 0 # outbound and inbound profits
    out_c = in_c = None # outbound, inbound indexes
    where = None # target system
    for dest_name in target_set :
        dest_sys = Galaxy[ dest_name ]
        if dest_sys.habitable :
            dest_line = dest_sys.trade
            d_c, d_p = compare_prices( home_line, dest_line )
            h_c, h_p = compare_prices( dest_line, home_line )
            if (out_p + in_p) < (d_p + h_p) :
                out_p, out_c = d_p, d_c
                in_p, in_c   = h_p, h_c
                where = dest_name
    return (where, out_c, out_p, in_c, in_p )

def best_n_hop(n):
    best_name = None # name of winning start point
    best_targ = None # name of its trade partner
    best_info = (0, 0, 0, 0)
    best_rt = 0 # round-trip profit
    pdict = dict()
    for name, system in Galaxy.items() :
        if system.habitable :
            where, out_c, out_p, in_c, in_p = best_trade_route(
                system,
                n_hop_targets( system, n )
                )
            if (out_p + in_p) > best_rt :
                best_name = name
                best_targ = where
                best_info = ( out_c, out_p, in_c, in_p )
                best_rt = out_p + in_p
                pdict[best_rt] = (best_name,best_targ,best_info)
    print( '\n\nBest', n, 'hop trade route is' )
    print( best_name, 'to', best_targ )
    print( 'outbound take', trade_names[ best_info[0] ], 'earning', best_info[1] )
    print( 'inbound bring', trade_names[ best_info[2] ], 'earning', best_info[3] )
    print( 'round-trip profit is', best_rt )

best_n_hop(1)
best_n_hop(2)
best_n_hop(3)
best_n_hop(4)

So before lunch I knew that I should trade Metals and Luxury Goods between Alphard and Delta Velorum. I took a certain amount of pleasure in having all that work quickly and easily.

My pleasure was rather lessened when I googled "Endless Sky best trade route" and found a forum post with exactly that route, found by some non-programmer weeks ago.

Oh, well.

Friday, October 23, 2015

A Mathematical Basis for Karma

So I've been auditing Jordan Peterson's course "Personality and Its Transformations". Peterson is a wonderful lecturer and while he sometimes drives off into the weeds of multiple digressions, he often just lights up one's skull with insights. (see note below)

Here is a concept that he tossed off in a couple of sentences, just a throw-away line really, around the 59:00 point in Lecture 14. He says,

I also don't think that the connections between people and the society are as abstract and distant as we think, because you might think, well, what the hell difference could it possibly make, you know, the way I behave? Well, you're a node in a network. You're not an individual connected by a linear line to another individual connected by a linear line to another individual, in a line that's seven billion people long. That would make you nothing: just pull you out and the line would reclose, and that would be the end of that. You're a node in a network, and the network's communicating. And we know for example, that you are roughly going to interact with, in some serious way, a thousand people in your lifetime as a minimum, minimum estimate. So, and all those people know a thousand people, so that's a million people that are one person away from you, and two people away from you is a billion people, and as soon as you get to three, well, that's far more people than there are. So, you know, you are only three or four or five connections away from everyone. And so it is very very difficult to know exactly how your behaviors and misbehaviors echo and ripple. And we know that people can be tremendous forces for good; we know that because you see people like that from time to time; and we certainly know plenty about the reverse. So God only knows what role you play in determining, you know, whether the part and the whole of mankind goes seriously wrong or seriously right.

So, he's Canadian, you know? Well, Canadian quirks aside, my mind was caught by that casual remark that you or I will influence at least 1000 people, and they 1000, etc. Which means, one's influence spreads by a power law. What of one's influence, how would it fall off? To work it out formally, let

  • E be the total of your influence on one other person
  • K be the number of persons you influence in each period Y
  • N be a number of periods

The value of Y can vary; Peterson was talking about your whole lifetime, but it could stand for one year without changing anything.

The number of people your influence reaches is KN. Say you influence K people in one year, they reflect or echo your influence on K people each in the next year, that's K2, K3 in the second year, and so on. This rapidly becomes a large number, as exponential functions are wont to do.

Meanwhile, the effect you have on people is being diluted as EN. Now this is like, but unlike, an epidemiological simulation. Simulating an epidemic, a node in the network is either infected or not infected, and a simulation of an epidemic has little nodes changing from green to red with no in-between. But one person's influence E on another person is only fractional; when you express an opinion, or behave in some way, another person will adopt that opinion or echo that behavior only weakly or with low probability. So perhaps E is a small fraction, 0.01 maybe, i.e. 1 person in a hundred will actually do exactly as you did or said. (It would really be a family of values, one for each kind of influence you might have; the influence of your speech patterns with one value, your habit of kicking puppies another value, your clothing choices another and so on ad infinitum; all relatively weak yet nonzero.)

The point is, your effect on others is EN, 0.01 on the people you interact with, but 0.012=0.0001 on the K2 people they influence, and so on. It gets smaller, but it never goes to zero—and also, it reaches a whole bunch of people.

In another essay, I wrote something to the effect that you could think of your actions as "seeding your world" with good things or bad things, with health or illness, calm or anger. Here is a mathematical basis for that, and (I think) the real basis for the Buddhist notion of karma.


Note from 2018: In recent times, Jordan Peterson has emerged as a rather distasteful public personality, much criticized by people whose opinions I respect. But I don't think this more recent criticism invalidates the very particular idea that I quote above.

Tuesday, August 18, 2015

Ppgen translator done, some bugs found

I finished the ppgen translator this afternoon. In order to verify it works I had to download and run the ppgen program itself, which proved quite simple. You just go get it from its github page, a big single Python module, and run it. I started to read it but decided instead to treat it as a black box. While I respect the effort that RFrank continually pours into supporting it, there are things about its design, and its documentation, that irk me as a professional writer and programmer. So if I start to read it I will just be picking nits and thinking of how it should be done, and that's unproductive of my time. Worse, I could sucked into helping maintain it. Run away!

Anyway, during the testing I found some things that couldn't be accounted for in my Translator code (which turned out to be pretty simple, less than 300 lines with lots of comments). Investigation led to two small bugs in the PPQT Translator support itself. There was one logic error that resulted in generating a spurious blank line preceding any no-reflow section. I am not sure why I never noticed that until these tests.

The other had to do with the YAPP-generated document parser. The way it was written, the following perfectly normal input,

...end of a paragraph.

<tb>

Start of next paragraph...

was wrongly parsed as if the second paragraph was a section head. In the DP document format a section head is marked by two preceding empty lines. There was only one preceding blank line here, why was it being parsed as a head?

Every production (other than the thought-break, which was a late addition to the parser) ended with EMPTY? to absorb any empty line that followed them. For example, a no-reflow section was defined as XOPEN (LINE | EMPTY)* XCLOSE EMPTY?. So if the user wrote

/X
stuff...
X/

New paragraph...

the blank line after the X/ line would be absorbed into the NOFLOW section. Because of my doing that in all cases, the syntax of a HEAD3 was just EMPTY PARA, or one empty line and a paragraph. When I belatedly remembered the thought-break markup and added it, I forgot to define it as absorbing an optional empty line after it. That was easy to add, a two-line fix. With the other bug, a total of 4 or 5 lines changed. But that mandates repackaging the whole app again, sigh. Although that isn't really so bad, a few hours of work, most of which is spent waiting for files to upload to or download from the dropbox, so I can be doing other things.

That will be Thursday. Before I do it I'll review the issues list, there may be a couple of other easy fixes I should do.

Saturday, August 15, 2015

Translators: Updating HTML, adding PPgen

By PPQT2 to the Woodshed

I used PPQT2 to post-process quite a bulky project, Hawkins Electrical Guide Vol. 3. This is the kind of PP project I've always enjoyed, with varied document structure (not just chapters of paragraphs) and many images. In this case, over 200 images, on which I spent many hours in Photoshop making clear, clean yet very compact .png files.

I had been working on this book ever since PPQT2 was at all usable, a year ago or so. After finishing the ASCII and HTML translators, I could finalize this book using PPQT2. Which I did, and uploaded it, and ran into an eagle-eyed and extremely conscientious PPVer, who kicked it back to me with a list of over 50 issues to correct. Properly taken to the woodshed, I was!

Some of the issues related to the generated HTML, and in the process of correcting them I realized some ways in which the HTML translator could do a better job. Also I had spent some time absorbing the various EPUB advice pages in the DP Wiki, and realized the impact that EPUB has on the post-processor's view of HTML.

EPUB Rant

Parenthetically, DP has a confused relationship to EPUB. Project Gutenberg now routinely does a batch conversion of the submitted HTML book using something called EPUBmaker, and it does a number on one's HTML. In prior years I, like many PPers, have spent lots of time on tweaking the HTML to make the ebook look very much like the printed book. But Epubmaker ruthlessly throws away most of that, leaving a flat, boring, ugly etext.

Double-parenthetically, part of the problem is the many restrictions of the EPUB format itself. It doesn't allow floats—so forget about sidebars, side-notes, and running text around small images. It doesn't support pop-up title=texts when you hover the mouse on an element—so forget about showing the original spelling of a typo, or showing the transliteration of a Greek or Cyrillic word. It imposes ridiculous constraints on images; nothing wider than 600px and no image files larger than 200K. Like other stupidly-designed standards, it takes the historical limitations of the ebook readers of 2005 and codifies them for all time. Do you think a retina iPad can't display an image larger than 600px? Or a Kindle Fire? The EPUB standard is very much like the many state laws that codified the design of auto headlights in the 1950s, based on the then state of the art, the sealed-beam unit. So when European cars started using replaceable halogen bulbs, they could not be imported to the U.S. because their headlights were not sealed-beam units. It took decades to get the laws changed so imported cars didn't have to have inferior U.S. headlight units retrofitted before they could be sold. EPUB does exactly the same thing, locking us into an already-outmoded technology. Close inner parenthesis.

DP's response to EPUB has been scattered and slow. There are several different Wiki pages about it, giving conflicting advice and often referring to forum threads that are years old. But the bottom line is, the PPer today who spends any time on how the HTML looks is wasting her energy. The majority of PG downloads are for EPUB, not HTML, and all your pretty CSS will be stripped out by Epubmaker. Close outer parenthesis!

HTML changes

With all this in mind, I went back to the HTML translator and made changes. I simplified the CSS in the header block a lot, removing many options and comments on appearance. I changed the method of encoding visible page numbers from the Guiguts method to a method that was recommended in one of the EPUB Wiki pages, as possibly able to survive Epubmaker.

Another change was from percentage widths to fixed widths. PPQT lets the user specify margins in ASCII space units, for example /Q First:6 Left:4 Right:4. These translate nicely in the ASCII output. But for HTML, I had been converting them to percentages of a 75-character line, so that Right:4 became margin-right:5%. But percent widths are relative to the container, so 5% is less in a nested container than at the outer level.

There was already a historical conversion of 2 ASCII spaces == 1 HTML "em" unit; this had been in use for poetry line indents for years in the Guiguts HTML conversion, and my HTML Translator did the same thing for poetry. Well, why not for all widths? So I changed it to use em units for everything, and Right:4 becomes margin-right:2em; which is the same regardless of context.

Ppgen

The afternoon after I posted the updated HTML Translator I was congratulating myself on the PPQT2 design that makes the Translators into separate files, and how easy it was to update just that file without having to repackage the whole app. And then about how nobody has expressed any interest in doing any other Translator. And how there really ought to be a Ppgen one.

I've had a Chrome window open for months, with about six tabs open pointing to different Ppgen docs. (Which, parenthetically, are badly organized and incomplete.) Well, crap, I said to myself, let's see how hard it would be. I pulled up a copy of my skeleton Translator file and started filling in the 30-odd entries in the "computed go-to" list of API "events". And it went very well. A majority of events are either null, or can be handled by a single literal string without any functional logic. For example, the OPEN_H2 event just squirts out .h2.

By end of the day I had almost all of it coded, lacking only the table-related events, and I pretty well see how to implement them.

So early next week I reckon I will be able to announce a trial Ppgen Translator. I'll have to hedge the announcement with many caveats, mostly because I do not have the actual Ppgen batch tools installed so I can't actually test that my translation produces usable output. But if people don't like it, they can fix it. It's just a small Python source file; be my guest.

And when that's finalized, people will be able use PPQT2 to complete Ppgen-based projects. Which might increase adoption.

Saturday, August 1, 2015

What to do, where to go next?

I've used this blog with the very clever name (well, perhaps not so clever these days, since nobody uses paper manuals any more, and if you've never seen a software manual on printed paper, you might not get the reference) -- used it, I say, to document whatever enthusiasm is monopolizing my attention. Before 2010, I used it as a place to store occasional essays on whatever was bubbling around in my brain. (Here's a really well-written piece, if I do say so myself, from 2009, on The Too-Small God. Here are actual numbers on the economics of a plug-in hybrid car.) For several months in 2010 I used it to record the process of rebuilding my recumbent bike. For the last two years I've used it as a diary as I developed PPQT2.

Well, PPQT2 is pretty well done now. There are several issues still on the github page, some of which would require significant days of effort to close. But I don't feel any urgency to do that work. The existing app is adequate for my personal needs. The "user community" aside from me can be numbered on one hand, I think; and they are very quiet.

So: whence the blog? Probably it will be very quiet for a while. If you have been following it for the PyQt5 stuff, thank you for reading along! I hope you got something useful from it. I don't expect to be doing much with PyQt now, but if I do, I'll post about it. So I suppose you should keep it in your RSS reader. Just move it down to the bottom of the list, next to those other blogs that you used to follow but which have gone quiet of late.

(I have several like that in my RSS reader. You know, that could make an interesting blog post...)

Thursday, July 23, 2015

Audio discoveries and problems

So I thought I would package Sidetone using PyInstaller, the latest version of which works so well with PPQT2. But strange things happen in the bundled version. The call to QAudioDeviceInfo.availableDevices(), which works perfectly running from source, returns an empty list to the bundled app. So both comboboxes are empty. Very appropriately, the empty comboboxes never generate a currentIndexChange signal, so the app never does anything (alos very appropriate).

I added code that, when the available list comes back empty, would get a one-item list of the device returned by QAudioDeviceInfo.defaultInputDevice() or QAudioDeviceInfo.defaultOutputDevice(). Because, the Qt docs assure me, "All platform and audio plugin implementations provide a default audio device to use." Which they do, but the devices being returned to the bundled app are invalid devices. They are QAudioInput or QAudioOutput objects, but they also return True from the .null() method, and when the app tries to start them, it generates a stderr message about trying to use a null device.

The code continues to run from source, but with this glitch. On my laptop—which is running the same levels of Mac OS, Python, Qt, and PyQt—when I unplug the USB headset, the app automatically switched to the built-in mic and speaker, and began an entertaining feedback warble. So I thought, OK, there must be some signal, some indication that a USB audio device has gone away. What is it?

But back on the desktop system, where I am doing the coding, things are different. There, when I pull the USB plug out, the app purrs on as if nothing had happened. I added code to intercept the stateChanged signal from the active devices and print the state. It goes from 3 (idle) to 0 (active) and stays there happily after the plug is pulled. And the system doesn't switch to the built-in devices. It is possible to select the built-in devices in Sidetone, and produce quite remarkable feedback effects, but it doesn't happen automatically on the desktop system.

I thought, OK, I'm plugging the headset into a USB hub. What if I put it directly into the back of the iMac? Something did change: the sound developed that "picket-fence" rattle indicating buffer under-run. I had to put the buffer size back up to 512 to eliminate it. Just speculating; the built-in USB delivers data faster than when the headset is on a hub, connected to the built-in hub? Don't care.

What did not change when I plugged into the built-in hub was the behavior when I pulled the plug out. No state change.

So I'm a kind of baffled on two points. One, how to know when the user yanks the plug on the device being used; and two, what is different about a bundled app than one running from source.

Neither is really important to my intended use (personal and casual). I'm cool running from source and I don't expect to be yanking the plug out in normal use. But I'd like to know. If you have any idea, please jump in with a comment.

Tuesday, July 21, 2015

Sidetone, first draft working

It turns out that yes, you can take input from a mic and put it into headphones, using Qt. The minimal process is this.

Acquire a list of available audio devices for input or output. For example,

self.input_info_list = QAudioDeviceInfo.availableDevices( QAudio.AudioInput )

The list items are QAudioDeviceInfo objects.

Populate a combobox (popup menu thing) with the names of the available devices, for example,

self.cb_inputs.addItems(
            [ audio_info.deviceName() for audio_info in self.input_info_list ]
            )

Present the two comboxes, one for input devices and one for output, and await a selection on either. Now it gets complicated, because on the currentIndexChanged signal from either combox, you maybe have not created any devices, or you've created one but not the other, blah blah. Anyway say you are creating an input device. You get new_index an index into that list of device info objects.

        audio_info = self.input_info_list[ new_index ]
        # Create a new QAudioInput based on that.
        preferred_format = audio_info.preferredFormat()
        self.input_device = QAudioInput( audio_info, preferred_format )
        self.input_device.setVolume( 1.0 )
        self.input_device.setBufferSize( 384 )

Now you have an input device. That last step, setting the buffer size, is import, as will be discussed in a minute.

Now the user selects an output device from that list.

        audio_info = self.otput_info_list[ new_index ]
        preferred_format = audio_info.preferredFormat()
        self.otput_device = QAudioOutput( audio_info, preferred_format )
        self.otput_device.setVolume( self.volume.value() / 100 )
        if self.input_device :
            self.input_device.start( self.otput_device.start() )

The very last line is what connects the input device to the output. The value of self.otput_device.start() is the QIODevice that the output device uses. Calling the input device's start() method and passing a QIODevice tells it, this is your target, the sink for your input data. The input device starts putting data into the QIODevice, and the output device takes it out and reproduces it.

In principle the exact reverse should also work, i.e. self.otput_device.start( self.input_device.start() ), but for some reason that leads to strange audio artifacts.

Anyway, the first time I ran this, without setting the buffersize, the sidetone was there but (as with the XCode demo program I wrote about) there was an echo, as if I were talking into a rather large barrel. It turns out the default buffer size is 4096 bytes. I changed the buffer size to 2048 and the echo became less. Then to 1024. Then to 512, with an improvement each time. At a buffer of 256, the audio stream developed a flutter or rapid "picket-fence" noise. Setting it back to 384 removed the noise. The sidetone still has a detectable echo, a ring, but it is tolerable.

Next I have to try it out on my laptop, which is where it will be used, simultaneous with the 3CX VOIP app that I am required to use. If Sidetone can co-exist with 3CX I'll be a happy camper.

If you'd like to play with Sidetone, it is right here on github.

Sidetone project 1

With PPQT pretty much out of my hair (until my OCD drives me back to do another translator or such) I turned to another project, a small one I call "Sidetone".

Sidetone is the sound of one's own voice in a telephone handset. I say handset because I learned the term when working for PacBell in the 60s. Picture a telephone handset:

You listen at the one end; you speak into the other; and a small amount of the sound of your own voice is repeated in the listening end. It makes the phone sound "live". Before a connection is made, or after the connection drops, there's no sidetone and your voice sounds flat, muffled, dead.

I'm starting to spend shifts taking calls on the Recovering From Religion hotline for which I wear a Plantronics headset. And it offers no sidetone. It's annoying. Of course I can hear myself, my voice is transmitted through the air and through my skull bones. And I know the mic is working because I can go to the Sound Preference pane and see the VU indicator bouncing as I speak. But I sound muffled to myself—no surprise, since I have padded things over my ears. I want sidetone! Checking around the web, I find some indications that in Windows, it is possible to get the audio driver to provide sidetone. It's an obvious feature for the system's audio to offer. Given you have a single USB or BlueTooth device that has both an input side and an output side, how hard would it be to direct an attenuated copy of the input signal back to the output? But this is not a feature offered by the Mac OS Sound Preference panel.

So I started picturing a simple little Mac app that would (somehow) take audio in and dribble a little sidetone back out. I thought maybe I could write a real "grown-up" app using XCode. I knew that Mac OS had something called Core Audio; and I know XCode has lots of example programs. So I tried to find some example that I could maybe manipulate to do what I wanted.

And I did, it's CAPlayThrough. But Oh. My. Word. what a monster. CAPlayThrough.cpp alone is over 800 lines (excluding the lengthy don't-sue-us prolog) and there are four other .cpp files and a bunch of .h files as well.

And the capper? It doesn't work very well! I had XCode build it and run it, and told it to take input from the mike and write output to the earphones. Speak into the mic and I sound as if I were in a good-sized barrel or a small cave; there is a latency of at least 0.1 second. That's not good for sidetone; sidetone has to be near-zero latency.

Then I poked around the Python docs and pypi for a while. Audio support is not one of the "batteries included" in Python. There are several libraries to interface to PortAudio, an open-source package. But it wasn't immediately clear how well PortAudio was integrated into Mac Core Audio. And the PyAudio interface package, like some other audio modules I looked at, depends on numpy. Which tells me they are storing and retrieving audio samples as numpy arrays. Probably I'm being unfair but that smells of latencies to me.

Then it occurred to me to look at good old (Py)Qt. What does Qt have to offer in the audio arena?

Quite a lot, it turns out. I'm not sure at this point if it is going to be possible to do what I want, but the facilities are certainly simple. The following program, displayed in full, lists all the available audio devices and their characteristics.

from PyQt5.QtMultimedia import QAudioDeviceInfo,QAudio
from PyQt5.QtWidgets import QApplication

app = QApplication([])

def show_info( mode, dev_list ):

    print('Found {} {} devices'.format( len(dev_list), mode) )

    for audio_info in dev_list :
        print('\n\nname:', audio_info.deviceName())
        preferred = audio_info.preferredFormat()
        print('\tpreferred channel count:', preferred.channelCount())
        print('\tpreferred sample rate:', preferred.sampleRate() )
        print('\tpreferred sample size:', preferred.sampleSize() )
        #print('\tsupported sample rates')
        #for rate in audio_info.supportedSampleRates() :
            #print( '\t\t{}'.format(rate) )
        #print('\tsupported sample sizes')
        #for size in audio_info.supportedSampleSizes() :
            #print( '\t\t{}'.format(size) )

dev_list = QAudioDeviceInfo.availableDevices( QAudio.AudioInput )
show_info( 'input', dev_list )

dev_list = QAudioDeviceInfo.availableDevices( QAudio.AudioOutput )
show_info( 'output', dev_list )

That took about 15 minutes to put together, most spent in the Assistant finding the names of the classes. Here's some sample output.

Found 2 input devices

name: Built-in Microphone
 preferred channel count: 2
 preferred sample rate: 44100
 preferred sample size: 24

name: Plantronics .Audio 648 USB
 preferred channel count: 2
 preferred sample rate: 44100
 preferred sample size: 16

Found 2 output devices

name: Built-in Output
 preferred channel count: 2
 preferred sample rate: 44100
 preferred sample size: 24

name: Plantronics .Audio 648 USB
 preferred channel count: 2
 preferred sample rate: 44100
 preferred sample size: 16

So this looks very encouraging. I'm going to try to cobble together some actual audio input and output next.

Friday, July 17, 2015

Patting myself on the back with both hands

This week I created the ASCII translator. It came to just over 800 lines of code—with lots of block comments and blank lines, so probably under 500 lines of actual Python. I wrote most of it on Monday and finished the first draft on Tuesday. Wednesday was museum day, but yesterday I sat down to test it. There were perhaps ten minor errors of the sort where I spelled a variable name two different ways, or forgot to initialize some global variable at the right time. There were maybe three places where I had to say, "hmmm, that can't work that way, need to recode". But by supper time all the features had been successfully tested except for table formatting. Mind you, this includes the Knuth-Pratt justification code, ported over from the V1 module but lightly recoded.

This morning I began testing table formatting. That's the process of reading something like this,

/T r6 l50 r8
I | CHAPTER ONE: THE MIRROR CRACKS | 3 |
II | CHAPTER TWO: THE GLAZIER IS CALLED AND COMES TWO HOURS LATE | 24 |
III| CHAPTER THREE: DIM REFLECTIONS ARE CAST | 33 |
T/

and producing output like this,

     I | CHAPTER ONE: THE MIRROR CRACKS                     |       3 |
    II | CHAPTER TWO: THE GLAZIER IS CALLED AND COMES TWO   |      24 |
       | HOURS LATE                                         |         |
   III | CHAPTER THREE: DIM REFLECTIONS ARE CAST            |      33 |

Note that the long table cell was "reflowed" to fit the specified width of the column. Knuth-Pratt reflowed, of course.

Well, it all went together very quickly. It actually mostly worked out of the gate. I spent more time getting it to issue appropriate error messages than I did making it format correctly.

So that's done. (OK, I might go back in and implement "Bottom" alignment for table cells, which could be used in the example above.) But a major piece of code by anyone's standards, I think, written and tested in four days. I told the wife, "You know, I rock." She said, "Sure you do."

What with that and having knocked off all the easily-fixable issues from the github list, it is time to make a new release. We're looking at a busy weekend, so it will probably be Monday when I do that.

There remain two pieces of PPQT2 that I might work on. One is a Ppgen Translator, although I think I would first try to persuade RFrank or one of his minions to do it. The other is to go back and actually implement the drag-out windows using the research with which I started this blog not quite 2 years ago. I don't feel a lot of urgency about either. I really want to move on to some other projects.

Edit: After I wrote the above, I thought about how to actually implement bottom cell alignment. And there really wasn't much to it. I opened up the file and had it working in about 15 minutes.

/T r6 l40 rB8
I | CHAPTER ONE: THE MIRROR CRACKS | 3 |
II | CHAPTER TWO: THE GLAZIER IS CALLED AND COMES TWO HOURS LATE | 24 |
III| CHAPTER THREE: DIM REFLECTIONS ARE CAST | 33 |
T/

Note the "B" in the spec for the third column.

     I|CHAPTER ONE: THE MIRROR CRACKS          |       3|
    II|CHAPTER TWO: THE GLAZIER IS CALLED AND  |        |
      |COMES TWO HOURS LATE                    |      24|
   III|CHAPTER THREE: DIM REFLECTIONS ARE CAST |      33|

I really do rock, you know.

Thursday, July 9, 2015

So back to work...

I released the mostly-baked PPQT2 to a reception that was friendly although very muted. In particular, nobody indicated any interest whatever in writing a Translator.

Then I spent a couple of days completing the basic work on a large and complex post-processing book. That is, I did all the steps that in my own "Suggested Workflow" document should precede translating the book to some other markup like HTML.

In the course of that, I found some issues with the sequence of events in the Suggested Workflow, so revised that. I also found a few minor usability issues with the app and added them to the Issues list on Github.

Then it was time to try translating a real, and large, document to HTML, complete with many Illustrations, a few Footnotes, and many Block Quotes and Unsigned Lists and a few Tables. So, all the stuff that a Translator should recognize.

The first step of Translating is parsing the document, and this threw up many errors. Some were legitimate; others should not happen, but do happen because the automated document syntax parser needs to be tweaked. Several more Issues went onto the stack. After I either fixed of circumvented those, the HTML translator actually got called, and it revealed two problems.

The first was a puzzling crash while processing a footnote. It turned out that I had mis-coded a Footnote in the document. This error was not being caught by the document structure parse, with the result that bad data was being passed to the translator. I had to tighten up a regex in the parser so it would not recognize an ill-formed footnote. It would just become a line of text.

The next problem was that the alt= and title= properties of most of the images were broken. The cause turned out to be obvious. Whatever text follows the [Illustration: markup, presumably the first line of the caption, is passed to the Translator along with the Open Figure event code. The point was to let the HTML translator use that first caption text as the alt=/title= string.

Unfortunately for most of the figures in this book, the opening of the caption looks like

[Illustration: <id="Fig_563"><sc>Fig</sc>. 563. Some bumble rumble thing

The <id="Fig_563"> is my optional markup for a link target; it is taken from the Ppgen markup. However, its presence results in building an img statement with:

<img src="images/f563.png" alt="<id="Fig_563"><sc>Fig</sc>. 563. some..."

I fixed this by adding a new utility function to the xlate_utils module: flatten_line(text) returns a text string with everything stripped out of except words and spaces. Then I had the html translator pass its Open Figure preview text through that, so that it would write HTML like this:

<img src="images/f563.png" alt="Fig 563 some bumble rumble thing"

Yes, flatten_line() even strips periods and other punctuation out. That's intentional, because after all, quote characters are punctuation too.

With these changes, the HTML translator is working quite nicely. It certainly produces an HTML book that is ready to be edited in html, have its CSS tweaked and so forth.

What next? Several things. First of all, there are 16 open Issues on the github site. Over the next few days I plan to fix at least ten of them. Then, I am going to write the ASCII translator I promised myself. When I have that, I will be able to complete post-processing of the book I'm working on. When that Translator works, I will put up an updated release of PPQT2 and make another plea for Translators, specifically for Ppgen and Fpgen ones. They are needed, and I am really not the right person to write one, as I lack the kind of deep knowledge of those markups that would make it easy. If necessary, I might write a very rudimentary one of each so I can tell the maintainers of those markups, there, now finish it please.

After that—which will happen by 30 July—I will dust off my hands and walk away from PPQT, returning only to fix serious bugs.

Thursday, July 2, 2015

Announced to a ringing silence

So I posted about PPQT2 in both the DP forum and the PGDP-Canada post-proofing forum. Almost no reaction, although I know a few people downloaded it because one of them had a problem (probably an incomplete download) and two others jumped in to say it worked for them.

For the time being I'm working at the book that I am actively post-proofing. In another week or so I will get to where I'd like to translate it to ASCII, at which time, if nobody else has written one, I'll write an ASCII Translator.

I have strong hopes, however, that the chaps who maintain Ppgen for DP and Fpgen for DP-Canada will step up and do Translators for their respective markups.

Tuesday, June 30, 2015

All Built, Waiting to Announce

I ran the PyInstaller builds on all four platforms, Mac OS, Windows 7 32-bit, and Ubuntu 32- and 64-bit. The process in each case is

  • Build the app and run it for a sanity check
  • create the PPQT2 folder containing the app, the README, the COPYING.TXT, and the extras folder
  • zip that folder with output to ~/Dropbox/Public/PPQT2
  • do something else while Dropbox loads the ~50MB file to the cloud
  • suspend that development VM
  • activate the test VM of the same platform (identical OS but no Python or Qt installed)
  • do something else while Dropbox updates that VM's copy of /Public
  • copy the zip file to the local desktop and unzip it
  • take a deep breath and hold it
  • run the app and exhale Yessssss! when it comes up and all features work.

Repeat for the next VM.

I was relieved to find that a 32-bit Win7 app runs just fine on a 64-bit Win7 installation. I was also relieved that the dev version of PyInstaller for Python3 worked perfectly in all three OS's. This is such a change from just a few months ago when I went through several weeks of agony trying different ways to bundle an app, including cxFreeze, pyqtdeploy, and nuitka.

What changed everything was that Hartmut Goebel and the other key maintainers of PyInstaller suddenly returned to activity with a long (and continuing) flurry of updates and fixes. The long-idle Python3 fork was merged to the mainline and got some key updates, and everything got good again. I feel a lot of gratitude toward those guys.

So now what for PPQT2? Right now I am waiting for the right moment to announce its availability on the DP forum. I think that will be Wednesday night; then I will be available to respond to forum comments promptly for four days straight. Meantime I am thinking hard about how to announce it, especially how to convey the importance of third-party contribution of Translators.

I have a couple of enhancements still to code. I promised I would write an ASCII Translator, just because I don't want my Knuth-Pratt paragraph justification code to be lost. And of course there will be some bug reports to deal with. But with any luck this project will be finito by the end of July.

Saturday, June 27, 2015

Major oops! revealed in windows test

So I had intended to release PPQT2 on Windows 7 64-bit only. But a query from an alpha user made me rethink that, and I decided better to build and release it on Windows 7 32-bit, which I trust and hope will work also on a 64-bit system. Unlike Linux, where I have to create versions for both widths.

Anyway, that meant setting up a 32-bit Win7 system. A couple weeks ago I went on eBay and bought a Win7 Professional 32-bit DVD. And installed it in a VM and provisioned it for development -- a process I meant to blog about today. But...

When I got to the point where I could run PPQT from source, I did so, and did a superficial checkout of various features. And immediately noticed that the Edit panel appeared to be using a non-monospaced font, probably Arial or whatever the Win7 default font is. Opened the preferences dialog, looked at the Edit Font choice popup menu. There were my expected Liberation Mono and Cousine font choices, along with several mono fonts from the local environment, like Consolas and Courier New. But selecting either Liberation Mono or Cousine from the menu and clicking Apply had no effect. The Edit panel remained showing Arial. Selecting a local font like Consolas did have immediate effect, so the whole mechanism of choosing and applying a font was working. But the two preferred fonts were not.

Hmmmm.

They were working perfectly well on Mac OS and Linux.

Weren't they?

Well, they seem to work.

To review, these two fonts are good looking mono fonts that both have a very wide repertoire of Unicode glyphs. I want them available to all PPQT users. To make that happen, I carefully built a resource file naming them, along with some minor graphic icon pngs, and compiled it with pyrcc5 into a Python file, resources.py. As a result, this code at the head of fonts.py:

_FONT_DB = QFontDatabase()
_FONT_DB.addApplicationFont(':/liberation_mono.ttf')
_FONT_DB.addApplicationFont(':/cousine.ttf')

...should prepare a local font database that includes all locally-available fonts plus those two, loaded from the Qt resources. They should then show up in the following code which prepares the list of families from which the "choose an edit font" popup is built.

# Return a list of available monospaced families for use in the preferences
# dialog. Because for some stupid reason the font database refuses to
# acknowledge that liberation mono and cousine are in fact, monospaced,
# insert those names too.

def list_of_good_families():
    selection = _FONT_DB.families() # all known family names
    short_list = [family for family in selection if _FONT_DB.isFixedPitch(family) ]
    short_list.insert(0,'Liberation Mono')
    short_list.insert(0,'Cousine')
    return short_list

Read the comments... I think I may know what that "stupid reason" was, now. Because, although those two family names were in the menu, choosing them had no effect. The Qt font database is carefully designed so that it always returns a font when you ask for one, falling back to some default if it can't give you what you ask for. Clearly in Windows, it couldn't provide the "Cousine" family and was falling back to the system default.

But not in Mac OS or Linux...

But you know, in both those dev systems, I believe I had installed both fonts locally...

Could it be that the font database had never been supplying those fonts from the resources module?

OK, so where did I import that module? I apply Wing IDE's "search in files" widget to look for "resources" in all project files.

I never did import it. I built the ****ing resources file and never imported it to the running app!

Added one line, import resources, to PPQT2.py and reran the app. Bingo! I can now set Liberation Mono or Cousine as Edit fonts!

It's a bit of a mystery how I managed to get along so far without encountering any problem from the missing resources. But I don't care. Just glad to have found it before shipping.

Thursday, June 25, 2015

A tough bug

So before I ship PPQT for real, I thought I better take one more look at an elusive problem. I had noticed that sometimes, keying ^f in the Edit panel did not bring the Find panel to the front as it is supposed to do. But the behavior seemed to be intermittent.

Recall that PPQT has on the left, a tabset of Edit panels, one per open document. And on the right, a tabset of various functional Panels like the Notes panel, Images panel, and Find panel. Each open document has its own set of function panels. So when you switch to a different document's Edit panel, the right half of the window is repopulated with the function panels for that document.

So what would happen is this. Say there are two documents open and document A is active (its Edit panel is visible). And on the right, the Images panel is displaying a scanned page from A. The user is typing and wants to find something, so keys ^f. What should, and usually does, happen is that the Find panel replaces the Images panel and the cursor focus moves to the Find text field.

What sometimes happened was that instead, the Images panel remained, but a narrow blue keyboard-focus rectangle appeared over it, outlining the position of the Find text field on the (invisible) Find panel.

It took a half-hour of repeated experiments to work out the failing conditions. There had to be at least two documents open. You had to switch between them in a certain order.

Internally, what I supposed was happening was that the Edit panel was trapping the ^f keystroke in its key event handler, and immediately issuing a signal named editKeyEvent, which was connected to a slot in the Find panel code. Using a breakpoint I verified that control got to the Find panel, and that it would call into a main window function to get itself displayed. The main window code is simple,

    def make_tab_visible(self, tabwidg):
        ix = self.panel_tabset.indexOf(tabwidg)
        if ix >= 0 : # widget exists in this tabset
            self.panel_tabset.setCurrentIndex(ix)
            return
        mainwindow_logger.error('Request to show nonexistent widget')

It asks the active tabset if it knows this widget; if it does, please make it active. What was happening was that the Find panel making this call was not the Find panel in the active tabset. It was the Find panel widget of the other document.

Wut?

This sent me off on a long tail-chase on the signal/slot setup. Somehow, I thought, the Edit panel for one book must have connected its ^f keystroke event signal to the wrong Find panel. But that code was all solid.

In the course of investigating the hook-up of the Edit signal to the Find panel, I noticed that there was another way for that signal to be emitted. It might come from the keyPressEvent handler of the edit panel. But it might also come from the Find... action of the Edit menu. Oho!

Months ago I implemented variable Edit menus. Each Edit panel has its own Edit menu, in which the menu Action for Find... (with a ^F keyboard accelerator option) was hooked to call a little function that emitted that Edit panel's signal to that Edit panel's Find panel. The Word panel and Notes panel also have their own Edit menus. This was all to get around Qt problems that plagued version 1, where different panels feuded over the ownership of the single global Edit menu. Now the Words panel has its own Edit menu that does things that relate to Words panel facilities, etc.

When any of these panels get a focusIn Event, they immediately call the main window asking it to put up their own custom Edit menu. When I implemented this, I put in a little gimmick to save some time.

_LAST_KEY = None
def set_up_edit_menu(key, action_list) :
    global _EDIT_MENU, _LAST_KEY
    if key != _LAST_KEY :
        _LAST_KEY = key
        _EDIT_MENU.clear()
        # code to populate the Edit menu from action_list
    _EDIT_MENU.setEnabled(True)

Each caller would pass a simple letter key, and the main window could avoid clearing and populating the menu when, as often happens, focus goes in and out of the same panel over and over.

The problem was, every Edit panel called with a key of 'E'. Ooops! That meant that when the focus moved from one document's Edit panel to another's—without a stop between in some other panel that had an Edit menu—the Edit menu would not be repopulated. It would still have the Actions defined by the first document. And that included a signal to the first document's Find panel when ^f was keyed or when Edit>Find was selected. So the signal would go to the wrong Find panel widget. It would see it wasn't visible, so it would call the main window; the main window would not find that widget in the current tabset; no Find panel would be displayed; but the Find panel would then call for keyboard focus to its Find text field, resulting in a focus rectangle over the top of the Images panel.

The fix was to make the key be something unique to the caller, and Python supplies a unique id via the id() built-in function. For a bonus, the callers no longer have to provide that key as an argument.

def set_up_edit_menu(action_list) :
    global _EDIT_MENU, _LAST_MENU
    if id(action_list) != _LAST_MENU :
        _LAST_MENU = id(action_list)
        _EDIT_MENU.clear()
        # populate the Edit menu from action_list
    _EDIT_MENU.setEnabled(True)

Wednesday, June 24, 2015

Learning about callable types

Another technique I used in the HTML translator is the Python equivalent of a "computed go-to". The Translator API passes a series of "events" and any translator has to deal with them something like this:

    for (code, text, stuff, lnum) in event_generator :
        # deal with this event

There are 34 possible event codes. A naive way of "dealing with this event" would be to write an if..elif..elif stack 34 items high. If you put the most frequent codes at the top this would not be too bad in performance, but it makes for rather unwieldy code to edit. A better way is to to have a dict in which the keys are the 34 code values, and the values are the actions to be performed for a given key. I've provided a skeleton of a Translator module with code like this:

    actions = {
        XU.Events.LINE          : None,
        XU.Events.OPEN_PARA     : None,
        XU.Events.CLOSE_PARA    : None,
...
        XU.Events.PAGE_BREAK    : note_page_break ,
...
        XU.Events.CLOSE_TABLE   : "</table>" ,
        }

    for (code, text, stuff, lnum) in event_generator :
        action = actions[ code ]
        if action : # is not None or null string,
            if isinstance( action, str ) :
                BODY << action # write string literal
            else :
                action() # call the callable
        # else do nothing

In this version, items in the dict can be one of three things:

  • None or a null string, meaning either "do nothing" or "not implemented yet"
  • A string literal to be copied to the output file immediately
  • A reference to a callable which will do something more involved, perhaps formatting the text value in some way before writing it.

(The callable functions named in this action dict were the "flock of little helper functions" that I referred to in yesterday's post—the ones that needed access to variables initialized by their parent function, and which had to become globals due to Python's eccentric scoping rules.)

For one part of the HTML Translator, I had a similar action dict but in it I wanted to have four possible types of actions:

  • None, meaning "do nothing"
  • A string literal to be copied to the output file immediately
  • A reference to a callable that would do something more involved such as setting or clearing a status flag.
  • A lambda that yielded a string value based on the text value but didn't actually contain a file-output call.

No problem, thought I. The above loop logic can easily be stretched to deal with this along these lines:

   for (code, text, stuff, lnum) in event_generator :
        action = actions[ code ]
        if action : # is not None or null string,
            if isinstance( action, str ) :
                BODY << action # write string literal
            elif type(action) == types.LambdaType :
                BODY << action() # invoke lambda, write its value
            else : # type(action) == FunctionType
                action() # call the callable
        # else do nothing

Surprise! It didn't work. Why? Turns out, although the types module has distinct names FunctionType and LambdaType, they are equal. You cannot distinguish between a reference to a lambda and a reference to a function based on type().

That kinda makes sense, in that we are told repeatedly that a lambda is just shorthand for an anonymous function. But it would have been handy to tell the difference.

In the end, for this part of the code (it was not the main translate loop but one with fewer possible codes) I made each value of the action dict a tuple ('f', funct_name) or ('s','literal string'). That allowed me to distinguish between a lambda that generated a string, and a function that didn't. But the whole thing felt like a kludge and I believe I will go back and recode that particular loop as an if/elif stack instead.

Tuesday, June 23, 2015

Learning about Python scoping

Variable scoping is a big topic in some programming languages, or was. "Scoping" refers to the rules about how the system resolves a variable name to its value. Here's an example of the problem I ran into while writing the HTML translator.

def parent():
    par_var = True
    def child():
        if par_var :
            print('yes')
        else :
            print('no')
    child()

When you execute parent(), what happens? Specifically, how is the child function's reference to par_var resolved to a value? My naive expectation was that Python would look first in the scope of child(), and fail; then look in the scope of parent(), and succeed, finding a value of True and printing "yes". Which it does! Expectations confirmed! But make this small change:

def parent():
    par_var = True
    def child():
        if par_var :
            print('yes')
            par_var = False
        else :
            print('no')
            par_var = True
    child()
    child()
    child()

What do you think? If you execute parent(), will it perhaps print yes, no, and yes? Nope. It will not run at all! It will immediately terminate with an error, "builtins.UnboundLocalError: local variable 'par_var' referenced before assignment".

What!?!

In the first example, Python had no problem resolving the child's reference to the parent's variable. But with this small change—assignment of values to par_var—the scoping rule changed. Now Python searches in the scope of child(), fails, and stops looking. It doesn't look out in the enclosing scope.

A little research turns up agreement that yes, this is the rule: if a variable is assigned a value in the scope of a function, Python assumes (in fact, insists) that the variable is local to that function. The only exception is if you specifically include a global statement. So let's do that:

def parent():
    par_var = True
    def child():
        global par_var
        if par_var :
            print('yes')
            par_var = False
        else :
            print('no')
            par_var = True
    child()
    child()
    child()

Does it work now? Nunh-unh. "builtins.NameError: name 'par_var' is not defined". The global statement modifies the scoping rule, all right. But it does not simply say, "global to me"; it says "global in the sense of being at the top level of this namespace/module". Which it is not, in that example. The only way to make the above code work is to move the first assignment to par_var outside the body of the parent function.

So in Python, it is possible to have a variable that is "relatively global"—not local but declared in some a containing scope—but only if that variable is read-only. As soon as the inner function attempts assignment, the variable must be either purely local, or purely global.

This is kind of wacky. It made me have to revise a bunch of code I'd written, where a parent function declared a whole batch of little helper child functions, and shared the use of the parent's variables. All the variables had to move out to the module level and get ALLCAP names. Also, I have this little evil thought: what if the child function does not assign to par_var but instead passes it to another function, and that function assigns to it? par_var might have to be a list or other mutable collection to make that work, but... hmmm.

Whatever, that's done. HTML conversion works nicely and I am happy to say, is really quick. It takes less time to translate a document and create a new document, than it does to load the source document in the first place. Later this week I will be packaging PPQT for release.

Friday, June 19, 2015

HTML Translator going together fast

I spent several hours coding up an HTML translator and have it 75% complete. Another coding session will finish it; then I'll have to test it a wee bit. Finished by Tuesday, I reckon.

I only found one awkward spot in the design of the Translator API. An Illustration markup like this:

[Illustration:fig_55.png|fig_55.jpg Fig. 55 The whang-dangle is..

Produces a sequence of "events" as follows:

OPEN_ILLO with the two filenames
OPEN_PARA
LINE "The whang-doodle..."

So at the time I am generating the code for an image div, I have only the filenames. (Passing the filenames after the colon is a PPQT extension. If the user doesn't do that, the Translator has no way to get them.) The HTML Translator can use this to build a nice opening string of:

<div class=image>
<a href='images/fig_55.jpg'>
<img src='images/fig_55.png' /></a>

However, it would be a useful service if it could also generate alt= and title= attributes from the starting text of the caption, for example alt="Fig. 55 The whang-dangle.... But that can't happen because the opening text of the caption will not arrive for two more events.

I thought about postponing actual output of the image markup until the first paragraph arrived, but that would make the code just horribly ugly. On every OPEN_PARA event you'd have to ask, is this the start of a caption?

I put in a similar but lesser kludge in translating a Footnote. The OPEN_FNOTE event has the footnote key value, so it can generate the target anchor and the back-link to the footnote reference. Unfortunately all that goes inside the first paragraph of the note:

<div class='footnote'>
<p><a id='Footnote_X'></a><a href='FNref_X'>[X]</a>

So here is the <p> being put out ahead of the coming OPEN_PARA event. So I had to create a switch footnote_starting and test it in the OPEN_PARA code, and when it is on, not generate anything and turn it off.

But by and large, the HTML Translator just kind of fell together. You just consider each of the 34 event codes in sequence, write a little bit of code, and repeat.

Tuesday, June 16, 2015

Just about ready for official release

I committed all the translator material to github. Except (OMG!) it appears I forgot to add the two new modules, translators.py and xlate_utils.py. OK, done. So that is all wrapped up and documented very well if I do say so (and as a retired tech writer, I think I know when an API is properly documented). I'm very pleased at how I used a formal syntax to define and verify the document structure. The code of translators.py is clean and well-organized. The mechanism for defining and displaying an "Options Dialog" is simple and (I hope) understandable.

There are parts of the xlate_utils.py module that I'm not quite so proud of. The tokenize() function is going to be fairly heavily used and I confess there are parts of it that are rather ad-hoc not to say downright kludgy. And not superbly well tested, yet. However, in coming days I'll be writing an HTML translator that will test it.

That done, on Monday and Tuesday I turned to my list of issues and resolved most of them. One that came out better than I originally expected was this: PPQT2 expects input files to be encoded UTF-8. The user can get a Latin-1 file correctly opened by renaming it to a suffix of .ltn, but if that is not done, the file will be input through the UTF-8 codec. Some special characters will not decode right and will be replaced with Unicode \ufffd, the "replacement character". And if this isn't noticed right away, and the file is saved, there's permanent loss of data.

So I very much wanted to catch this error early and warn the user. But how? I researched the methods of QTextStream, QFile, and QTextCodec. I know that while QTextStream is executing a readAll() call, it must use a QTextCodec.toUnicode() function. That function is capable of returning a count of invalid characters, but there doesn't seem to be any way to find it out.

It looked as if the only ways I could use to find out if the file decoded properly would be either, one, to read it with Python, in which case the readall method would throw an exception; or two, to use QTextStream.readAll() into a string and search the string for replacement characters. Either method would require me to change the API between the main window and the Book, or else to read the possibly-large document file twice.

Then it dawned on me that the QPlainTextEditor has a perfectly good find() method. All I had to do was, in the Book just after it has loaded the editor from the file, to call the editor's find() to look for a replacement character. One hopes the search fails. But if it does not, I can notify the user with a warning message, including the character position of the first replacement character. I made the warning message detailed and also included a pointer to the Help topic.

Another long-standing issue, more of a major loose end to clean up, was logging. There are lots and lots of log messages being issued all over the program. But the logging output was going nowhere. I had initially thought that I would add argument parsing, and use it to support --log-path= and --log-level= parameters. But that's dumb; I'm packaging the Windows and Mac OS versions as clickable apps, with no command-line input. And the Linux version doesn't have to be launched from a command line. So I did some study and reading and chose writable locations for log files based on the platform: /var/tmp for Linux, ~/Library/Logs in Mac OS, and \Windows\Temp in Windows. Oops, I just realized I committed the code for that, but I should also update the Help file to document it. Or maybe put it in the README for each version?

Anyway: tomorrow is Museum day. Thursday I need to spend most of my free time studying the docs for a new volunteer gig. I have an online training session for that at 4pm that day and I want to be prepared. But Friday I will start coding an HTML Translator. Should have that done early next week. By the end of next week I should have PPQT2 packaged up ready to announce. Can't wait.

Saturday, June 13, 2015

Translator API complete

So I continued to code in bursts over the week away in Seattle, and today at home I was able to put in several hours and finished the Translator API, including documenting it. This makes PPQT2 just about functionally complete. What I need to do over the next week—to be honest, probably two weeks—is:

  • Commit all this work and check it in to github,
  • Write a real Translator to serve as an example—I think probably an HTML one as it would be the easiest,
  • Clean up all the "issues" I have been posting for myself on the github page,
  • Create 32-bit Win7 dev and test VMs to replace the 64-bit ones I've been using,
  • Bundle the package and release it—as a beta? Or final? No, Beta.

That done, I will spend the early part of July awaiting bug reports, writing another Translator, either fpgen or ppgen, and finally implementing drag-out tabs. By mid-july I intend this project to be done.

Sunday, June 7, 2015

Test Translator running

My coding time has been limited the past few days and will continue to be so as we fly off to Seattle for a few days to visit relatives, although I'll get in some. I hope by this time next week the complete Translator interface will be working.

What is working now is: document parsing, finding Translator modules, building the submenu, responding to the selection of a Translator from the menu, displaying a Translator's option query dialog, and calling the Translator's initialize(), translate() and finalize() entries with appropriate arguments. A special demo Translator named "Testing" exists and accepts calls at those entries. All it does is produce lines of output documenting what it is called with, including displaying each document "event" it gets.

So far, the output of Testing (or any other Translator, should one exist, but none do) is only dumped with print statements. But a whole lot of machinery had to work in order to get this far.

One bug I had to work through was a classic "lost reference" bug. In most Qt interfaces, when you build a widget and give it to another widget to manage, the manager widget becomes a "parent" and retains the child widget, so it continues to exist. That's not the case with menus. A QMenu does not take ownership of a QAction or a sub-QMenu. I forgot that. I modified the main window to call the translator support module to build the submenu of translators, but didn't save the reference returned. Just handed it to the File menu's addMenu() method.

The result was an interesting bug: the Translators... sub-menu would appear the first time you opened the File menu. Then the second or third time (or sometimes the first time), there would be no submenu. It would disappear. It all depended on when the Python garbage collector got around to demolishing the submenu QMenu object. That took half an hour of trying and thinking before I finally twigged, did a classic Doh! face-palm, and did a one-line change to have mainwindow save a reference to the submenu.

Another problem was my lack of experience with Python generators. A generator is a function that contains a yield statement, but what is tricky is, it is not that function you call in your for-loop; it is that function's return value. And the function call gets parenthesized arguments, while the iterator does not. I almost had it right but had to review the python docs and recode at two different points.

I used a generator in my "scanner" that overrides the built-in token scanner for the YAPPS parser. My scanner class took an iterator—which is anything that responds to the next() built-in function—as an initialization parameter. It calls next(iterator) to get the next line of the document being parsed.

For unit-test purposes, it was initialized with a Python StringIO object loaded with a triple-quoted literal. A StringIO responds to next() with the next line of stuff. But for the real thing, I needed to pass an iterator that yields the next line of the real document.

Months ago I coded such an iterator into the editdata module. It's a simple function:

    def all_lines(self):
        tb = self.begin() # first QTextBlock in document
        while tb.isValid():
            yield tb.text()
            tb = tb.next()

There's the magic yield statement. But when I passed editdata.all_lines to initialize the scanner, I got an error about cannot iterate a method. What I had to pass to the scanner in place of a StringIO was not all_lines but all_lines(), the returned value of calling the generator. That's the iterator that can respond to a next() call.

I made the exact inverse goof in quickly whipping up the Testing Translator. The translate() method of a Translator is passed an iterator that returns "events" it is to process. I was correctly passing it the result of calling my event-generating function with a yield statement. But in the Translator where it invoked the iterator, I coded for (code, text, stuff, lnum) in event_iterator() and got a different error message, about "cannot call an iterator". Had to remove the parens, is all.

I want to modestly point out that when a Translator gets such a Python error, it is caught and displayed with some helpful info in an error dialog to the user. That code's working too.

What's not working? Two tricky bits. I've promised in the Translator API doc that there will be a tokenize function that breaks a line of text up into mini-events so that the Translator doesn't need to duplicate the logic to extract such things as superscripts, subscripts, footnote anchors, and markups like italics. That will take a bit of thinking to make clean and bulletproof.

And the final step of doing translation: taking the text that the Translator writes into a MemoryStream object, combining it with metadata extracted from the source Book object, and creating a new Book object with the translated text and at least some of the source metadata. That's going to take some fiddly code, probably involving JSON editing.

And finally, back in the mainwindow code, installing a new Book and making it visible for editing. That's just a special case of the File>New operation, hopefully.

Saturday, June 6, 2015

Making Translators real -- and why I hate Markdown

"Smelling the barn"—an expression that horsey people use to describe the sudden enthusiasm of a horse when it nears the end of a long ride. Do horses really do that? Whatever; I'm starting to smell the end of this project. I'm alternating between writing the code that will call a Translator and updating the document that tells the Translator's author what will happen. And it all feels like it's coming together at last. I have serious hopes of delivering the 1.0 product by the end of this month!

But the API doc... I started writing it in Markdown. It's simple; it's designed for documenting code; it's also slightly less stupidly inconsistent than reST. But then I tried to process it for display. Aagghhh!

One would expect that John Gruber, the inventor of Markdown, could write an accurate preview widget—wouldn't one? So the first place I tried my document was at his "markdown dingus". And it is terrible! Although the sidebar clearly states that a four-space indent means a code block, my document has numerous indented code blocks and Gruber's "dingus" renders them all in normal text font. It preserves the line breaks in these sections, at least, but does not preserve the white-space indents.

Worse, it does not automatically escape either underscores or HTML codes within a code block. So where the code contains an underscore in a variable name, the underscore disappears and the "code" suddenly turns italic. And where there's a bit of HTML in the code block, it takes effect. It's just a mess. Shame on you, John Gruber.

There's a pretty decent Markdown extension for Chrome. It renders my code blocks fine. But I'm just confused because for Gruber's widget I have to escape underscores inside variable names, but then the Chrome widget renders the backslash. I'm going to assume that the Chrome extension is accurate (whatever "accurate" means when discussing Markdown). But Markdown is a mess, a very unsatisfactory medium. Ordinary manual HTML, like I do in this blog, would be less confusing for sure.

Tuesday, June 2, 2015

Implementing translators 1: API spec

With the document parse coded and tested, I turned my attention to coding the function that will actually invoke a Translator when one is selected from the menu. I spent a couple of hours tidying and rearranging the code of translators.py and coding the framework of the function. My approach to this is to write a comment prolog saying what I'm going to do, and revising that over and over as I realize why I can't do that, at least not in that sequence, so I'm actually going to do this, etc. etc.

The first steps are known and I could code them: find if the Translator had a user-options dialog and presenting it; parsing the document. But then: time to start actually calling the Translator. This brought me face-on to the question of what exactly is the interface to the Translator. I had a detailed sketch of a design from weeks ago but knew more now. So I changed modes and spent about 3 intense hours rewriting the API spec. I made several on-the-fly decisions that simplified it from before. Really, any competent Python programmer—who also understands the full range of things that can occur in a DP document—and who also understands in detail the syntax of the target document format—should have no difficulty writing a Translator.

When I put it that way, it suggests a Venn diagram of three circles for which the little triangle where all three overlap might be rather small. But for the select group of people in that happy spot—no problem.

Monday, June 1, 2015

Parsing a document, 7: testing (and ranting at Enum)

I am now beginning to test my document-parsing code and things are going very well. It amounted to about 250 LOC (plus the code generated by YAPPS, another 200 or so). For first execution of brand new code things went well. After I picked off 6 or 8 stupid coding errors (like: defining some regexes as class members and forgetting to use "self." to reference them) and a couple of small logic errors, it is happily parsing a simple document.

One problem I ran into that took a bit of finagling was this. The generated parser comes from a "grammar" file. I've shown some preliminary grammar code in previous posts. One tricky production is the one for heads:

rule HEAD:      EMPTY {{print('head...')}} ( PARA {{ print( "...3") }}
                            | EMPTY EMPTY PARA+ EMPTY {{ print("...2") }}
                            )

The items in {{double braces}} are Python statements which YAPPS will insert into the generated parser code at the point where parsing reaches that part of the production. In that code the statements are print() calls. But what I really needed was this:

rule HEAD:      EMPTY {{ open_head() }} ( PARA {{ close_head(3) }}
                            | EMPTY EMPTY PARA+ EMPTY {{ close_head(2) }}
                            )

In other words, call functions of mine that will set up a start-heading work unit, and, when the type of heading is known—only after processing the paragraph(s) of text within the heading—back-patch the open-head unit with the type of head it turned out to be, and append the close-head unit.

Well, that code died with an exception because "function open_head() not found." Wut? I was importing the parser with:

from dpdocumentsyntax import DPDOC

which should make the parser class part of the active namespace where the functions like open_para() were defined. But no. I tried several ways to work around this. You can include blocks of code in the generated parser, but if I defined the helpers like open_para() there, they could not see the globals like the WORK_UNITS list they had to modify. Eventually I had to do it in a not very pretty way,

import dpdocumentsyntax
dpdocumentsyntax.open_para = open_para

That is, manually inserting those definitions into the imported namespace.

Anyway, as it parses, the code builds a list of "work unit" objects that will eventually be fed to a Translator as "events". A typical sequence of work units, or events, would be,

  • Open head(2) (Chapter head)
  • Open paragraph
  • Line (text: "CHAPTER ONE")
  • Close paragraph
  • Close head(2)
  • Open paragraph
  • Line (text)
  • Line (text)
  • Close paragraph

And so forth. There are all told 30 different possible "events" and I expect to pass each to a Translator with a code signifying what kind of event it is, e.g. Open Paragraph, close BlockQuote, or Open Illustration Caption, etc. So how should these codes be defined? Obviously there must be names for them, like OPEN_PARA, CLOSE_FNOTE and so forth. And obviously these will be in a module the Translator can include, perhaps so:

from xlate_utils import EVENTS

Then the coder can make decisions by comparing to EVENTS.OPEN_PARA and the like.

Looks like a job for an Enum, right? The Enum "type"—it isn't a type—was added to Python in version 3.4, and having played with it, I cannot fathom why they bothered. It has to be the most useless piece of syntax ever. But check this out.

from enum import Enum
class ECode( Enum ):
  VAL1 = '1'
  VAL2 = '2'
ECode.VAL1
<ECode.VAL1: '1'>
'1' == ECode.VAL1
False
edict = { ECode.VAL1: 1, ECode.VAL2: 2 }
edict
{<ECode.VAL2: '2'>: 2, <ECode.VAL1: '1'>: 1}
edict['1']
Traceback (most recent call last):
  File "<string>", line 1, in <fragment>
builtins.KeyError: '1'
ECode.VAL1 < ECode.VAL2
Traceback (most recent call last):
  File "<string>", line 1, in <fragment>
builtins.TypeError: unorderable types: ECode() < ECode()

Now, for something completely different:

class HCode( object ) :
  VAL1 = '1'
  VAL2 = '2'
HCode.VAL1
'1'
HCode.VAL1 == '1'
True
hdict = { HCode.VAL1 : 1, HCode.VAL2 : 'to' }
hdict
{'1': 1, '2': 'to'}
hdict[ '2' ]
'to'
hdict[ HCode.VAL1 ]
1
HCode.VAL1 < HCode.VAL2
True

What I'm saying is that a simple class definition accomplishes everything that the "Enum" class does, and also has "real" values that can be compared and ordered. There is the one tiny drawback that a user could assign to HCode.VAL1 but that can, I believe, be prevented by adding a decorator.

So I will be providing the 30 event codes as a class EVENTS that is really a class and performs what a C header file does: give names to arbitrary literals.

Friday, May 29, 2015

Parsing a document: 5, first coding bits

I have started the process of integrating a generated parser into my still-growing translators.py module. The big task is to override YAPPS's default "scanner" class with one of my own. The default scanner takes a string or a file and gives its associated parser characters on request. But in my case, the characters are distillations of the lines of the document. It turns out I only need to override one method, grab_input().

The interface between the parser and the scanner is not exactly a clean one. The scanner maintains a member "input" which is a string, and a member "pos" which is an index to the next unused char of the string. The parser increments the scanner's pos member as it matches tokens. When it has caused pos>=len(input), it calls grab_input(). That method is supposed to adjust pos and input so that pos<len(input).

In my case, I will usually set pos=0 and set input to a single character, the code for the current line's contents. There are a few cases where I put more than one code in input.

I have this about 3/4 coded, including the code to save the non-empty lines as "work units" ready to hand to a Translator. When the parse of the document is complete, there will be work units for all the lines in a list. The parse having succeeded, I can take the work units and shove them at the translator one at a time.

I was slowed down a bit today. I started adding an enum to the code, and discovered that my laptop was still on Python 3.3, so "import enum" didn't work. So I had to stop and install Python 3.4. But then I realized, oh doggone it, now I don't have any of my third-party libs like regex or hunspell, so I had to install them. Or mostly just copy them from the 3.3 site-packages to the 3.4 one. But it took some time.

I still need to fiddle with the document syntax, mostly in order to insert bits of Python code at significant transitions. Then I can let the parser discover things for me. For example, when the parser knows it is starting a heading, I can generate a "Open Heading" work unit, and when the parser finds out which kind of heading it is, I can update that work unit.

Anyway, tomorrow or Monday I will have this to a state where I can actually execute it. Hopefully by the end of the week I'll be able to finalize the Translator API and start coding a test Translator.

Thursday, May 28, 2015

Parsing a document: 4, yapps2 results

I succeeded in defining a grammar for an extended form of DP document structure in YAPPS2, compiling it, and running it. The output is a fairly simple program of about 200 LOC. The grammar is based on the idea that I will "tokenize" the document to one character per line, with each character representing the structural value of the line: start of a section, end of a section, empty, or text (and an extra "E]" at the end of a footnote, illustration or sidenote, as discussed a couple days back).

Then feed the string of characters to this generated parser. Either it will "recognize" the string as valid, or it will produce a syntax error with enough info that I can tell the user the approximate location where it goes wrong.

Normally a generated parser needs added code to process the string being parsed. But in this case all I want is validation that the structure is correct. Then I can feed the lines to a Translator in sequence. The Translator coder does not need to worry about structure; the Translator can be nearly stateless.

OK, so here is the complete syntax.

%%
parser dpdoc:

    token END: "$"
    token LINE:     "L"
    token EMPTY:    "E"
    token XOPEN:    "X"
    token XCLOSE:   "x"
    token ROPEN:    "R"
    token RCLOSE:   "r"
    token COPEN:    "C"
    token CCLOSE:   "c"
    token TOPEN:    "T"
    token TCLOSE:   "t"
    token POPEN:    "P"
    token PCLOSE:   "p"
    token FOPEN:    "F"
    token IOPEN:    "I"
    token SOPEN:    "S"
    token BCLOSE:   "\\]"
    token QOPEN:    "Q"
    token QCLOSE:   "q"
    token NOPEN:    "N"
    token NCLOSE:   "n"

    rule NOFILL:    XOPEN ( LINE | EMPTY )* XCLOSE EMPTY? {{ print("nofill") }}
    rule RIGHT:     ROPEN ( LINE | EMPTY )* RCLOSE EMPTY? {{ print("right") }}
    rule CENTER:    COPEN ( LINE | EMPTY )* CCLOSE EMPTY? {{ print("center") }}
    rule TABLE:     TOPEN ( LINE | EMPTY )* TCLOSE EMPTY? {{ print("table") }}
    rule POEM:      POPEN ( LINE | EMPTY )* PCLOSE EMPTY? {{ print("poem") }}
    rule PARA:      LINE+ ( EMPTY | END )  {{ print("para") }}

    rule HEAD:      EMPTY {{print('head...')}} ( PARA {{ print( "...3") }}
                            | EMPTY EMPTY PARA+ EMPTY {{ print("...2") }}
                            )

    rule QUOTE:     QOPEN ( PARA | POEM | RIGHT | CENTER | QUOTE )+ QCLOSE EMPTY? {{ print("quote") }}
    rule FIGURE:    IOPEN ( PARA | POEM | TABLE | QUOTE )+ BCLOSE EMPTY? {{ print("figure") }}
    rule SNOTE:     SOPEN PARA+ BCLOSE EMPTY? {{ print("sidenote") }}
    rule FNOTE:     FOPEN ( PARA | POEM | TABLE | QUOTE )+ BCLOSE EMPTY? {{ print("fnote") }}
    rule FZONE:     NOPEN ( HEAD3 | FNOTE )* NCLOSE EMPTY? {{ print("zone") }}

    rule NOFILLS:   ( NOFILL | RIGHT | CENTER | TABLE | POEM )

    rule goal: EMPTY* ( NOFILLS | PARA | HEAD | QUOTE | FIGURE | SNOTE | FNOTE | FZONE )+ END

The print statements in double braces are for debugging; they go away in the final. They could be replaced with other Python statements to actually do something at those points in the parse, but as I said, all I want is to know that the parse completes.

The generated code defines a class "dpdoc". When instantiated it makes a parser for these rules. One passes a "scanner" object when making the parser object. By default it is a text scanner defined in the YAPPS2 runtime module, but mine will be quite different.

The HEAD rule caused some issues. Initially I wrote,

rule HEAD2: EMPTY EMPTY EMPTY PARA+ EMPTY
rule HEAD3: EMPTY PARA

But YAPPS rejected that as ambiguous. It only looks ahead one token. When it sees EMPTY it can't tell if it is starting a HEAD2 or a HEAD3. Eventually I found the solution in the Yapps doc, as shown above.

Tomorrow I have to figure out how to organize the text line information as I "tokenize" it, so as to have it ready to feed into a Translator. And start to implement that tokenizer.

Edit: actually now I see an error, the HEAD3 call in the FZONE rule. That's a relic; there is no HEAD3 production since the HEAD ambiguity was resolved. Bad test coverage! Not sure how to resolve this. May need to actually rely on output from the parser.

Tuesday, May 26, 2015

Parsing a document: 3, trying out YAPPS2

I have winnowed down the list of 34 (yes you read that correctly) parser generators to a very short list of ones that (a) are pure Python, (b) are documented in a readable and complete fashion, (c) appear to allow the user to tinker with the tokenizer—as opposed to being locked-in to parsing strings or files. That group is:

I've spent the last two afternoons reading parser docs until my eyes bleed and their features are starting to run together. It would be a fascinating exercise, and useful to the Python community, to spend a few weeks really sorting out those offerings and put together a paper with comparative code examples and timings and such. I don't have time to do it well even for the short list above. Maybe someday.

Anyway to commence I thought I'd generate a parser using YAPPS2, and trace through the code of the generated parser and really get a handle on what it does. So I downloaded it. First thing to note is that the download link in PyPi doesn't really go to a download page, but to a page that points in a confused manner several directions: to a Debian package, another Debian package "with improvements", and to a Github repo that is supposedly Python 3 compatible. But it isn't. But there's a link to a set of patches for the Github code that fixes quite a few Python 3 issues, notably print statements. But it wasn't complete; very shortly after applying it I ran into an unfixed "except Exception, e" and soon after, another unfixed print statement. So it's an adventure getting it going.

But I got it to where I could begin to try the first example in the manual. Which is clearly very old, because this is supposedly the YAPPS2 manual, but the example has you "import yapps"—it has to be "import yapps2" now. And that did not work, but immediately stopped with an undefined name. Exploring, it turns out that the code is such that the hand-execution shown in the manual (start python and type "import yapps2; yapps2.generate('filename')") cannot possibly work. A critical statement "from yapps import grammar" is only executed when yapps2.py is run from the command line.

OK the generate step now reads the "calc" (basic calculator) example definition and writes a small and readable python program. Which upon execution reveals several more Python 2/3 issues, including use of raw_input and some more print statements. But when I manually fixed those, it actually worked, reading expressions, parsing them, and printing the results.

My brain is a bit fried at this point; gonna take a nap now; tomorrow is Museum work day; resume this on Thursday.

Parsing a document: 2, parser generator or own code?

So the question still open is, should I use a parser generator to make a dedicated parser that could validate the document structure according to the syntax I described yesterday? Or, should I just hack out my own recursive-descent parser?

Yesterday's post has a link to a table with 20 or so different Python-based parser generators. I'm still going through them one by one, reading their documentation, trying to decide if (and how) I could use them.

A parser generator is basically a stand-alone Python program that reads a syntax description and writes a module of code that can parse that syntax. One hopes the parsing would be fast, and when an error is found, it is reported with some accuracy. Another requirement for me, is that the generated parser be pure Python with no dependencies on C modules. The generator itself can have such dependencies because it would not be distributed; only the generated parser would be part of the product.

There are some advantages to using a generated parser. First, accuracy. One would expect that the parser would correctly parse exactly the specified syntax. Any errors would be in the syntax itself, not the parser. Second, modifiability. If for some reason the syntax needs to be changed in any way, it's just a matter of editing a syntax file and generating a new parser, and that's that. No code changes. Finally, error reporting. In principle the parser can report the location of an error, or at least the location it finds the error, accurately.

The alternative is to write my own parser, guided by the syntax. It would loop over the lines of the document, pushing things onto a stack when it sees an opening tag like /* and popping them off when it sees a closing tag. It would implement the rules of what can appear inside what by testing against some sort of table. It would probably implement things like "swallowing" the E? optional empty line after certain productions using some hack or other to save time.

This approach is tempting just because I know from experience that the huge majority of documents are nearly flat with very little nested structure and few errors. So an actual parse is in a way, overkill. But a hand-made parser is also the mirror image of the generated parser: the syntax rules are distributed through 500-odd lines of code and hard to change, as well as impossible to certainly verify. Error reporting might or might not be as good.

I've got to finish surveying the parser generators, pick the likeliest one, and do maybe a few small experiments to understand how to use it, before I decide.

Monday, May 25, 2015

Parsing a document: 1, a document structure syntax

First thing today I sat down and worked out a BNF-style notation for the document structure I want to support. This is the structure that is only implicit in the DP "Formatting Guidelines" but with block structure augmented.

As a parsing problem this is unusual in that most parsers and parser documentation are focused on parsing tokens that are substrings in a line, for example, the tokens within a statement like foo=6+bar*3. In defining the structure of the DP document the tokens are not characters but entire lines. For example one "terminal" production—comparable to the id "foo" in the preceding statement—is a no-reflow section defined as

/*
...any amount of lines ...
*/

My first thoughts were based on the idea that I could, for purposes of validating the structure, just reduce the entire document to a string of letters, one letter per line. Suppose for example that

  • an empty line --> E
  • /* --> X
  • */ --> x (note, lowercase)
  • etc for others
  • a nonempty line not otherwise classified --> L

Given that, a rule for a no-fill production would be X(L|E)+x. The only other such structure that the guidelines allow is /#...#/ for a block quote. This means "reflowed text that is indented". The guidelines never seem to have envisaged what else might appear inside a block quote.

I would add support for /C...C/, no-reflow but centered; /R...R/, no-reflow but right-aligned; /T..T/ for tables, really just no-reflow sections but needing special handling a Translator; and /P...P/, Poetry, in which each line is treated as a separate paragraph, but leading spaces are retained and a line can be reflowed if it is too long for the current width, but then with a deep indent for the "folded" lines. Now: can any of these appear inside a block quote? Inside each other?

An additional problem arises with three block sections that the guidelines treat in a different way: Illustrations, Sidenotes, and Footnotes. In each case the block begins with left-bracket and a keyword. The block can end on the same line or on a later line; the end of the block is a line that terminates with a right-bracket. But the content of that line before the right bracket is part of the text.

[Illustration: Fig 32: A short and snappy caption.]
[Illustration: Fig 33: A ponderous and lengthy and
especially, long caption that might even include...
...wait for it...
/#
Yes! A Block Quote!
#/
And who knows what else?]

This causes a problem as compared to the other block sections: their delimiters are whole lines, where these blocks are delimited by parts that appear on the same line(s). It turns out that for easiest processing, one would like to treat them as if they were broken out on separate lines with an extra empty line. For example, the one line [Footnote B: Content of this note.] is best encoded as if it were

[Footnote B:
Content of this note.

]

And of course if I were scanning the document and building this string, I could do just that: put out at least four characters for a Footnote: perhaps F to start it, then characters representing its content line(s), then a E and a right-bracket.

Empty lines cause some concern because, unlike the usual computer grammar that treats newlines as just more whitespace, they are semantically meaningful. A paragraph is one or more non-empty lines that terminates with an empty line (or the end of the file, or the right-bracket of a Footnote or Illustration...). A level-2 head (a.k.a. Chapter title) begins with four empty lines, may contain multiple paragraphs and ends with two empty lines. A level-3 or subhead begins with two empty lines and terminates with one.

Also, users are instructed to precede markup openings like /# with an empty line, and to follow a markup close like #/ with one. But that means the both the paragraph and any markup section "eats" its following empty line, so that in fact a Head-2 is signaled by three (not four) empty lines, one surely having been eaten by the preceding construct whatever it was.

Well, that said, with all the above caveats, here is a draft document syntax.

# The nofill/right/center/table sections may only contain L and E, 
# to have any other letter like P in them shows an error.
# All the multiline sections absorb a following empty line but
# don't insist on it.

rule Nofill : X[LE]+xE?
rule Right  : R[LE]+rE?
rule Center : C[LE]+cE?
rule Table  : T[LE]+tE? # Table cells can't have Poetry, etc

# Poems can only have lines and blank lines, no /C etc.
# If you want a centered Canto number or right-aligned attribution,
# insert P/ and restart the poem on the next stanza.

rule Poem   : P[LE]+pE?

# A paragraph absorbs the following empty line

rule Para   : (L+E) | (L+$) # $ == end of file

# Assert: every Head2/3 is preceded by some other element that
# eats a terminal E.

rule Head2  : EEE(Para)+E
rule Head3  : E(Para)

# A block quote is allowed to contain text, right/center aligns,
# Poetry or A NESTED QUOTE. Arbitrarily ruling out no-fill and Tables.

rule Quote  : Q(Para|Right|Center|Poem|Quote)+qE?

# A side-note should be just a phrase but who knows? Anyway,
# only Paras.

rule SNote  :  S(Para)+]E?

# Figure captions may contain Quotes, Poems, or Tables. No other
# figures or Footnotes.

rule Figure :  I(Para|Poem|Table|Quote)+]E?

# Footnotes same.

rule FNote  :  F(Para|Poem|Table|Quote)+]E?

# A footnotes "landing zone" can have a Head3 and FNotes, or nothing

rule NoteLZ :  N(Head3|FNote)*nE?

With this more or less nailed down I started reading up on parser generators in Python. There is a helpful table of them in the Python wiki and by the end of the day I'd gotten through reading the docs on maybe half of them. More on that tomorrow.