And the missing puzzle piece was...

Yesterday I struggled with the problem of how to direct the clicked() signal of a QPushButton. Although the symptoms were confusing, it seemed fairly clear that I was unintentionally connecting the default signal, whose signature is "clicked(bool)", when what I wanted was the no-parameter version, "clicked()".

In the old code, using the PyQt4 syntax, the signature of the desired signal was specified using the SIGNAL macro, SIGNAL("clicked()"). In the new signal/slot API, the doc shows how to specify a signal with a different parameter list, by "indexing" the signal name with a type, e.g. signalname[str].connect(...). What the doc didn't show was how to select the no-parameter signal.

This question was quickly answered on the PyQt mailing list by Baz Walter: you "index" the signal name with an empty tuple, signalname[()].connect(...). And that worked just fine, so my user-button connecting code now reads,

        for j in range(FindPanel.USER_BUTTON_MAX):
            self.user_buttons[j].clicked[()].connect(
                lambda b=j: self.user_button_click(b)
                )
            self.user_buttons[j].user_button_ctl_click.connect(
                lambda b=j : self.user_button_load(b)
                )

I worked out for myself what I was doing with an expression like

    lambda b=j: self.user_button_click(b)

A lambda expression lambda args : expr is exactly equivalent to

    def anonymous (args):
        expr

so

    lambda b=j: self.user_button_click(b)

is the equivalent of

    def anonymous (b=j):
        self.user_button_click(b)

In other words, I am specifying an argument with a default value! At execution time in the for-loop, the current value of j is substituted, for example

    def anonymous (b=17):
        self.user_button_click(b)

When the no-argument version of the signal is specified, the anonymous function is called with no arguments and the default index value is provided. When I unintentionally invoked the bool-passing signal version, the boolean it passed was taken as the argument b, and passed along instead of the default.

This does not explain to me why the current value of j is not also substituted when I write it this way:

    lambda : self.user_button_click(j)

This ought to mean,

    def anonymous ():
        self.user_button_click(17)

What happens instead is that every user-button clicked passes 23, the last-defined value of j in that loop. Which suggests to me that it is actually referencing the variable j. Oh wait, I could test that... Uh-huh! When I code it this way:

        for j in range(FindPanel.USER_BUTTON_MAX):
            self.user_buttons[j].clicked[()].connect(
                lambda : self.user_button_click(j)
                )
            self.user_buttons[j].user_button_ctl_click.connect(
                lambda b=j : self.user_button_load(b)
                )
            j = 'gotcha!'

and then click a user button, guess what the parameter to user_button_click is. Yup. "gotcha!"

So when a variable appears in the argument part of a lambda expression, its value is substituted for its name (b=j becomes b=17), but when a variable appears in the expression part, it is parsed as a reference to that variable, just as if the expression was in open code.

I'm glad it works this way or I'd have to think of a different scheme for directing the signals from multiple buttons to a single method (I'm sure there are other ways). But it seems inconsistent.

Edit: Actually, this is the result of Python's documented handling of default arguments. From the doc,

Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call.

So the conversion of b=j into b=17 (or whatever number) is exactly what should happen. So this is not some kind of hack, but rather an involved but acceptable way of getting a literal value into a lambda.

Connecting signals for an array of buttons

The Find panel has an array of 24 UserButton objects, which are custom QPushButtons. Two signals from each button are significant. The built-in clicked() signal causes one action to happen. A custom signal, generated during a control-click, causes a different action to happen. In version 1, under PyQt4, the code to connect these signals looked like this.

        for i in range(UserButtonMax):
            self.connect(self.userButtons[i], SIGNAL("clicked()"),
                                lambda b=i : self.userButtonClick(b) )
            self.connect(self.userButtons[i], SIGNAL("userButtonLoad"),
                                lambda b=i : self.userButtonLoad(b) )

The slot for each signal is a lambda that just calls the relevant function passing the integer index of the button. So in the userButtonClick(self,button) method, button is the index of the button.

It took a while to work out how to generate a succession of lambda functions, each passing a different number. I'm still not sure exactly why

lambda b=i : self.userButtonClick(b)

passes the literal value of i as it was when the expression was evaluated, while

lambda: self.userButtonClick(i)

does not. Lambdas are magic. Anyway, here is the very similar code for version 2, using the new and improved PyQt5 signal syntax.

        for j in range(FindPanel.USER_BUTTON_MAX):
            self.user_buttons[j].clicked.connect(
                lambda b=j : self.user_button_click(b)
                )
            self.user_buttons[j].user_button_ctl_click.connect(
                lambda b=j : self.user_button_load(b)
                )

The only difference between those, besides trivial changes of nomenclature, is the use of .clicked.connect instead of SIGNAL("clicked()"), I swear. I have looked at those until my eyes were bloodshot and they are the same.

But of course they don't work the same.

Well, actually, the custom button does work. When I control-click on a user-button, the self.user_button_load() method is called with its index number from 0 to 23, just like it ought to be. But a normal click calls the self.user_button_click() method with an argument of... wait for it... False. Not an int in 0..23, and the same for every button.

Now, right away I noticed that although in V1 I was invoking SIGNAL("clicked()"), in fact the clicked signal of QPushButton is documented as passing a boolean representing its checked state. Which would be False, as these are (by default and also by explicit code) not checkable buttons.

If that's the case, the only issue is how in the PyQt5 API to select the no-parameter overload of that signal. The documentation shows how to select an overloaded signal definition with a different parameter list, but not how to select one with no parameter.

But then I thought, OK, suppose that value does represent the boolean "checked" parameter of the signal. How is it getting into the parameter list of my method?!? The "slot" that receives the signal is an anonymous function defined as

lambda b=j : self.user_button_load(b)

which when it was evaluated was self.user_button_load(17) or such. That anonymous function gets a boolean as a parameter and ignores it, passing a literal when it calls user_button_load(). So howinahell does False get in there?

I've written this to the PyQt mailing list. We shall see.

Confused About Sizes

In this ongoing comedy of errors, I am still fumbling around trying to understand why a QPushButton takes up so much more vertical space than a QLineEdit, as described in the preceding post. This has brought me up against the Qt rules for layouts, which I have always found most confusing.

Go to the QWidget doc page and search for the string "QSize", that is, every method that either takes or returns a QSize or QSizePolicy. Start with the description passage on "Size Hints and Size Policies". Read the property descriptions for Size and SizeHint and SizePolicy, and follow some links. When you've read about all of those, try to explain in simple terms, what determines the vertical size of a default QPushButton and in particular, when you lay out pushbuttons vertically why do they take up more space than line edits do, as shown here:

On the left, three QPushButtons in a VBox. On the right, three QLineEdits in a VBox. Why are the pushbuttons spaced more widely than the line-edits? I sure can't say.

I put in code to display the QSizeHint from each after the layout was complete.

pb1 True 32 74
le1 True 21 142

Both return valid QSizeHints ("True" from isValid()), and vertical size hints that differ by 11 pixels, 21 versus 32. Well, alrighty then. I made a custom QPushButton that returned a size hint with a vertical of 21, same as the line-edit does, and the width of 74 as a stock QPushButton did in this layout. Here's what I got.

Two things of immediate note. One, the height looks right! The pushbutton lines up nicely with the line-edit. But, the look has drastically changed; it's no longer a Mac OS button shape! The rounded corners are gone and the shading is reversed. This is clearer if you click it to see the full resolution "retina" size dialog.

In an effort to better understand the size controls, I modified this custom class to report every size-related method call:

class SHPB(QPushButton):
    def __init__(self,text='',parent=None):
        super().__init__(text,parent)
        self.setText('Shpb')
    def sizeHint(self):
        qs = super().sizeHint()
        print('sizehint',qs.width(),qs.height())
        #return qs
        return QSize(74,21)
    def sizePolicy(self):
        sp = super().sizePolicy()
        print('sizepolicy',int(sp.horizontalPolicy()),int(sp.verticalPolicy()))
        return sp
    def baseSize(self):
        print('baseSize')
        return super().baseSize()
    def maximumSize(self):
        print('maximumSize')
        return super().maximumSize()
    def minimumSize(self):
        print('minimumSize')
        return super().minimumSize()
    def minimumSizeHint(self):
        qs = super().minimumSizeHint()
        print('minimumsizehint',qs.width(),qs.height())
        return qs
    def size(self):
        print('size')
        return super().size()

So which of these methods gets called? Just two: the only print statements come from sizeHint() (two calls) and minimumSizeHint() (one call). No other size-related method appears to be called. I strongly suspected that Qt is accessing the properties for base, minimum, and maximum sizes and the size policy directly, without going through a method call, or at least not going through a method that PyQt handled. So, wait a minute, suppose I set a property? I added this line to the __init__() code:

        self.setSizePolicy(self.sizePolicy())

Aha! On the next run, the sizePolicy method was called. It is returning (1,0), i.e. Minimum and Fixed. So apparently PyQt doesn't insert a Python hook in a C++ method, even when I provide an overriding method, unless there's some reason? Yup. I added these lines,

        self.setMinimumSize(self.minimumSize())
        self.setMaximumSize(self.maximumSize())

and then my minimumSize() and maximumSize() overrides were called when they were not called before.

Well, that's interesting, but it doesn't address the issue. Apparently I can, by returning a modified size hint, get a pushbutton to line up with a line-edit. (Later: I can also do self.setMaximumHeight(21) for the same result.) But this comes with two major drawbacks:

• I have to hard-code the height. 21 pixels looks right on Mac OS 10.9, but it might not be the best height for every platform. I'd much rather have Qt pick specific heights.
• I lose the native button shape, and get a Qt non-native button shape. Which means I would need to do it for every button, or else have inconsistent button shapes.

I think I would rather have uniform, native button shapes and Qt-assigned heights that are based on the platform, the active font, etc. The price of that is a "puffy" layout of the stack of Replace controls. I can live with that, I guess.

QIcon and Scaling, and Layout Spacing

One component of the Find/Replace panel (with whose layout I am still diddling around, which is pathetic) is the recall button. There are four QLineEdits, one for the string to find, the others replace strings, and for each line-edit there is a recall button that pops up a menu showing the last ten strings the user entered into that line-edit. This is a wonderfully useful feature of BBEdit's Find dialog, also appearing in Guiguts. You just have to compose a complicated search or replace string once, and then it will usually still be there, ready to use again, in that edit session and future edit sessions.

In V1 I created these rather clumsily, using a QComboBox. A combo box usually has a full-sized pushbutton with a text label in it. In order to get a small button I forced the width of the combo box to 32x32 pixels. It worked, although it looked a bit odd.

In working on V2 I discovered that Qt had a different widget, the QToolButton, which is specifically for this use of quick access to a popup menu. So I'm using that as the basis for the V2 RecallMenuButton class. And I wanted it to have an icon, some kind of clock-dial to mean "recent" (slavishly copying BBEdit here).

I burned a couple of hours around the internet looking for simple clock icons. I found a couple that would do, but when I tried them out they disappeared or turned into mush. The problem is the scaling that QIcon does. Inside the __init__ of a RecallMenuButton, it sets its icon with

        self.setIcon(QIcon(':/recent-icon.png'))

QIcon's constructor passes the path string to QPixMap. Then it scales the resulting pixmap to about 22x22 pixels, roughly. And the scaling algorithm is rather brutal. Here's the input file.

That's my own work, by the way. None of the free icons I could find online had graphic elements that were simple enough and meaty enough to survive the scale-down that QIcon does. They all turned into spatters of pixels or just nothing. I ended up making this with OpenOffice draw in about 1/5th the time I'd spent browsing for icons online.

Here's how the image ends up as an icon.

I continue to struggle with control of spacing of elements in a layout. I must stop obsessing about this and just take what Qt gives me. Here's what is bugging me. The Replace part of Find/Replace has three rows. Each row has a RecallMenuButton, a line-edit, and a QPushbutton "Replace". These three items are laid out in a QHBoxLayout. Then all three are stacked in a QVBoxLayout and the terror begins. For some reason the vbox wants to space the rows widely, thus:

That looks all puffy, I thought. Experimenting, I found that if I left out the pushbuttons, the spacing became more to my taste:

Well, I need the pushbuttons. I set them to have a vertical size policy of Fixed and tried various heights. The default height (in Mac OS) is about 32 pixels. Making them shorter makes them luck funny but doesn't change the line spacing. I'm at a loss and must get on to, you know, the actual functional code behind this UI? So I'll accept the puffy spacing for now.

Frames and Layouts

Today I spent about 4 hours futzing with the coding of the UI for findview. That is, creating the buttons and fields that make up the UI, and arranging them in layout objects. This was a tedious business for two reasons. One reason is that for every little tweak and change you make, you have to save the file and run the test driver, contemplate what you've achieved (usually nothing, or something inexplicably different from what you expect), quit and edit again.

Reason two was that I was fighting my own misconceptions and confusions about the relationship of a QFrame and a QLayout. About the middle of the session I had a satori. Suddenly I understood.

Instead of explaining my hours (indeed weeks) of confusion, I'll just explain how I understand it now. Here's the problem: you have a group of widgets that you'd like to display inside a frame, a nice sunken or raised panel, say. But you also need to arrange them, and your arrangements might need to be two or more levels deep. Here's an example.

For the Find panel, the controls related to actual searching are as follows:

• Four QCheckBox widgets that condition the interpretation of the search string: Respect Case, Whole Word, Regex, and In Selection. These are laid out in a row in a QHBoxLayout.
• A customized QLineEdit for entering the search string, and to its left, a QToolButton that pops up a menu of the last 10 search strings used. These two are laid out in a QHBoxLayout.
• Four QPushButtons that initiate searching named First, Next, Prior and Last. These are laid out as two pairs, First and Next, space, Prior and Last, in a row with a QHBoxLayout.

The three QHBoxLayouts are then stacked in a single QVBoxLayout to keep them together. Call that the_find_box.

I want to group all these ten widgets inside a styled frame. How to do this? OK, you are an experienced Qt hacker and you could do it in your sleep. But if you have not got the background of experience, the answer is frustratingly hard to find. I knew about the layout classes and how to use them to get widgets arranged as I want them horizontally and vertically. But doc pages for the layout objects never talk about frames. And the QFrame doc page doesn't talk about layouts.

The obvious thing would be to create a frame and style it using its methods, and then add it to the layout in much the way you add widgets, or stretch. Nope, doesn't happen. There is no explicit, documented way to connect a layout to a frame, nor vice versa. The word "frame" doesn't appear on the QLayout page, nor "layout" on QFrame.

Here's the answer. Set up this part of the UI in the following sequence:

• Create the individual buttons, line-edits, labels and checkboxes.
• Create the sub-layout objects, like the QHBoxLayout that holds the four QCheckBoxes mentioned above. Add the widgets to these.
• Create the QFrame and set its properties as desired.
• Create the outermost layout object, the one whose shape should be defined by the frame style, and make the frame its parent.

Using the example above,

    the_find_frame = QFrame()
    the_find_frame.setFrameStyle(QFrame.Sunken)
    the_find_box = QVBoxLayout(the_find_frame) # outermost layout is child of frame
    the_find_box.addLayout(the_checkbox_row)
    ...

That was the key realization: that you link a layout object to a visible frame by making the frame its parent. This makes no kind of software design sense to me. I don't see how a frame, which has only visible style properties, is an appropriate parent to a layout, which has powerful organizational properties. If anything, the relationship should run the other way, with the frame being a child, i.e. a subordinate, of a layout. If the way of connecting the two were to use a QLayout method "addFrame" it would make perfect sense to me.

But in fact, this the arbitrary way you assign a visible "look" to a collection of widgets organized by a layout: by making the QFrame the parent of the outermost QLayout.

Finding a Hitch Right Away

Today I mostly spent tidying up editview, and making its unit test and Sikuli tests work. They were back-level. When I wrote them there was no mainwindow or book, so they faked the features of those modules, but they faked them in ways other than I eventually implemented. So I had to strip out the fakery and put in real modules. But then stuff worked.

I spent quite a bit of time on a support method of editview called center_this(), which is given a QTextCursor with a selection of some unknown length (usually short, but I didn't want to limit it), and makes sure that the whole selection is visible, or if the selection is actually taller than the edit's viewport, its top and as much more as possible is visible. This is different from the existing ensureCursorVisible() method of QTextEdit, which only ensures that the first line of a selection is on the screen someplace.

It turned out to be complicated mostly because of the different units involved. You can get the location of a bit of text by the following means,

    def _top_pixel_of_pos(self, pos):
        tc = QTextCursor(self.document)
        tc.setPosition(pos)
        return self.Editor.cursorRect(tc).y()

Apply that to the cursor's selectionStart() and selectionEnd() positions, and add to the latter the height of a line from QTextEdit.fontMetrics().lineSpacing(), and you have the top and bottom of a selection in "viewport coordinates". You get the viewport size, and then can work out by how many pixels you need to scroll up or down to put the selection in the middle.

Now what? Well, the only reliable scrolling method is to modify the value of the vertical scroll bar. (QTextEdit inherits a scroll() method but when I tried to use it some very strange things happened. QTextEdit also inherits from QAbstractScrollArea, which gives it a scrollContentsBy() method that is explictly not to be used "programmatically".)

But the scrollbar operates in arbitrary units, so you have to get your move distance in pixels as a ratio of the pixel-height of the document, and multiply to get a fraction of the scroll bar's maximum value... bleagh. It works, finally.

That's not what I wanted to write about.

For the Find module I really want to provide the user with full PCRE-compatible regexes, and I discussed this in an early post here, where I concluded I was forced to use Python regexes, not Qt5's new PCRE-compatible QRegularExpression support.

Only problem with that is, it requires getting Python-code access to the document contents, which means calling QPlainTextEdit.plainText(). That copies the entire document contents into a Python3 string value. If I have to do that a lot, there could be considerable performance impact. Plus there's the issue of replacing a matched string. To do that, I'd have to

• Note the position and length of the matched string
• perform the regex replace in the Python string
• copy the replacement string
• form a QTextCursor to select the matched string by position and length in the editor's world
• use its insertText() method to replace the selection with the replacement string.

Doable, but not pretty. What would be nice would be, if the find() method of the Qt5 version of QTextDocument had been updated to accept a QRegularExpression. I spent a lot of time in V1 working around the restrictions of the document's find() method, but if I could use it, I could avoid trying to keep two massive texts in sync.

So I just tried passing a QRegularExpression to QTextDocument.find() and it was rejected, "arguments did not match any overloaded call". Oh sigh.

Just in case I'm missing something, I posted in the Qt general forum. But I don't have a lot of hope for this.

OK, I am embarrassed (again)

I try not to make a blithering idiot of myself in public too often, but in this matter of highlighting the current line, I certainly have. Proudly presenting code snippets showing an entirely wrong way of doing it... I blush.

OK the right way to highlight some line is to present it to the editor (QTextEdit or QPlainTextEdit) as an "extra selection." An extra selection is a ... what? It isn't a class on its own. It's an object that you acquire as follows:

        self.current_line_sel = QTextEdit.ExtraSelection()

You may have many of these; you tell the editor about them by passing a list of them to its setExtraSelections() method. In my case, a list of one item, a selection marking the current line.

An extra selection object is basically a tuple comprising a QTextCursor and a QTextCharFormat. These are assignable properties, there are no get-set methods for them. So here's the whole initialization.

        self.current_line_fmt = QTextCharFormat()
        self.current_line_fmt.setProperty(QTextFormat.FullWidthSelection, True)
        self.current_line_fmt.setBackground(colors.get_current_line_brush())
        self.current_line_sel = QTextEdit.ExtraSelection()
        self.current_line_sel.format = QTextCharFormat(self.current_line_fmt)

The bit about setting the FullWidthSelection property of the QTextCharFormat is key; it ensures that the new background color will be painted the width of the editor's viewport regardless of the length of the current line. I wouldn't have know that without seeing it in the Code Editor example.

When the cursor-move signal arrives, it is only necessary to update the position of the cursor in the selection, and to re-assign the list of selections. Here is the abbreviated cursor-move code now.

    def _cursor_moved(self):
        tc = QTextCursor(self.Editor.textCursor()) # copy of cursor
...several lines snipped...
        # Change the extra selection to the current line.
        tc.clearSelection()
        self.current_line_sel.cursor = tc
        self.Editor.setExtraSelections([self.current_line_sel])

It is necessary to clear the selection from the cursor before using it. Without that step, my current-line highlight disappears as soon as a non-empty selection is made. Double-click a word and it is highlighted, and the current-line highlight goes out.

It is also necessary to re-assign the list of extra selections every time. It is not enough to update the existing selection's cursor. You have to make the editor aware of the change. Which kind of makes sense.

Anyway that's all there was to it. The code is 50% less than the method I displayed in three previous blog posts. And it has no effect on the undo/redo stack or the document's modified status. So it's all good now, except for my ego.

Current Line, Again

Today I reviewed the editview module, the widget that contains the text editor and also a row of widgets to display the document name, the scanned image filename, the logical page number, cursor line# and cursor column#, all in a row at the bottom.

I'd made this widget using Qt Creator, which means that all the UI setup is in a separate file automatically generated, and merged into my widget using multiple inheritance. (I described the process in this post.)

The complexity of this setup was, I strongly suspected, causing an annoying problem in which the Notes panel, which is actually a second QPlainTextEditor, never got a proper highlight on its selection. A selection in Notes was always 50% gray, which should only be the case when it was visible but did not have the keyboard focus. In any case, the verbose and opaque code output by Creator and pyuic5 irked me. So, after reviewing the existing (and pretty much working) code, I tackled the job of bringing the UI initialization into my own module. It was a lengthy but basically clerical task of copying chunks from the generated module, pasting them into my own, and editing them to simplify them and throw out the redundant bits. (Example redundancy: every lineEdit got its text set to a null string, which is the default anyway.)

After fixing a few syntax errors and tweaking a couple of margin values, it worked. And: the annoying problem with selection highlighting was gone! Click in the Notes panel and make a selection, it's highlighted a nice green, while the selection in the main editor goes to gray. Click in the Edit panel, vice versa. Nice.

But now I had to face a bug that's been there right along (I was in denial). Remember a long time ago, well, May 6, when I wrote about setting a highlight on the current line? I did it by setting the Text Block Format of the current line. Bad Idea, it turns out.

Changing the text block format was causing two problems. First, as soon as I moved the cursor, the document status changed to "modified" (the document name in the lower corner got bold and red). This is because QTextEdit considers a change of text block format to be an undoable action. Anything that goes on the undo stack makes the document modified.

This wouldn't matter a great deal once the document actually was modified, but it is annoying that just hitting a down-arrow turns on the "save me!" indicator. I'm sure a user would complain. OK, I would complain.

But it had another problem also: it was effectively killing control-z undo. If I typed some characters and immediately hit ^z, the undo worked. But if I moved the cursor to another line first and then keyed ^z, the only effect was that the cursor moved down one line. Control-z had become down-arrow. What?!?

I figured it out with a little reading about undo. When QTextEdit performs Undo, it says it leaves the cursor at the end of the changed text. So here's what happens: move the cursor to another line by an arrow key or by clicking. That stacks two changes of TextBlockFormat, one on the old current line (to a white background) and one on the new current line (to a purty pale yellow one). Probably both go in as one undo action, I don't know.

Anyway, now key ^z. What gets undone? The last change of text block format. And the cursor is moved to the end of the restored area, which means, the start of the next line in the document. That's an edit cursor position change, which goes through my code to change the text block format of the new current line, stacking another undo action. Another ^z undoes that and moves down a line. Etc.

I now have to go back to trying to highlight the current line using "extra selections". I tried that initially and something didn't work, I forget what. Well... mañana.

Annotations and type-declarations

Today I finished reviewing and retesting book, mainwindow and editdata. Just editview to go, and on the casual schedule I laid out tuesday.

Also today I learned about Guido's proposal for more detailed function annotations based on mypy. In the Reddit comments for that I learned about the Obiwan package for function annotation by Will Edwards (corrected link). And from both, learned that Python 3 already has support for type-annotating at least function arguments and return values.

All really interesting in an academic way. I like the idea of type-annotating functions and return values, but not if it is merely a formal type of commentary—which, it appears, it is and would remain under all these proposals.

My antique coding style, evolved from years of writing code professionally in both strictly-typed languages like C and Pascal and in completely un-typed assembly languages, is to make the types of inputs and outputs of every function, as well as the type of every variable, crystal clear, at least in my own mind. I don't fudge things and I never overload the meanings of a functions. It would be nice to have a way of documenting this in Python syntax if that would get me some additional function from the language—like compile-time error checking, or code optimization. So I did a quick trial of the Python 3 annotations.

def check(fname:str):
  enc = 'UTF-8'
  if '-l.' in fname or '-ltn.' in fname or fname.endswith('.ltn'):
    enc = 'LATIN-1'
  return enc

check('asd')
'UTF-8'
check('x.ltn')
'LATIN-1'
check(5)
Traceback (most recent call last):
  File "/Applications/WingPersonal.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 1, in 
    # Used internally for debug sandbox under external interpreter
  File "/Applications/WingPersonal.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 3, in check
    pass
builtins.TypeError: argument of type 'int' is not iterable

No error checking. The clearly incorrect call check(5) slides right by the compile phase and traps at just where it would without an annotation.

So what's the point of annotating? To the human reader/maintainer, it is blindingly obvious (from comparisons to string literals and the use of .endswith()) that fname is expected to be a string and only ever a string. The human doesn't need the annotation (at least for this simple application) and the Python interpreter effectively ignores it. It's just a pointless decoration.

I can see that you could write a lint-like program that would use annotations to check for obvious type transgressions. Maybe pylint or some other tool does that now. But there are two major shortcomings to such a pre-processor. First, you have to incorporate it into your workflow somehow. Maybe it could be an automatic part of a test suite? But it will always be an extra step that complicates the programmer's work flow.

Second and more seriously, a static lint-like tool cannot catch the great majority of the type errors that arise. In fact, it can probably only catch errors in (1) literal argument values, where the type error is manifest in the code and is also (2) in the same source module as the annotated function.

If the annotated function is in an imported module (from utilities import check; check(5)) the lint has to examine the imported module to even be aware of the annotation. And if the function is defined in the present module but the argument comes from another module (import constants as C; check(C.INT_VALUE)) again the lint has to have executed the import to know anything about the referenced literal.

And a static type-checker is helpless if the argument value is not a literal—a function return, or simply the result of an expression. What can it do with check(foo(bar(baz())))? Type errors in this case can only be caught by the Python interpreter at the time it is compiling the expression. Then the dynamic AST could in principle provide enough information to predict an impending type error.

So until the interpreter itself implements and checks type annotations, I don't see any value to using them. That interpreter doesn't have to be CPython, note; in fact I would think that the PyPy or Cython teams would see annotations as a major asset for optimization, with checking falling out of an optimized execution.

How wide is a button?

Reviewing imageview.py I discovered a bit of code commented-out. In the setup of the UI, imageview creates two QPushButtons for instant zooms, labelled (in English) "to Height" and "to Width". Obviously these label strings are not the same width. And, if they are translated to some other language, they will have different widths still.

(Well, in German they would be zu Breite and zu Höhe, thank you Google Translate, the Width one still the shorter, but in French à Largeur and à Hauteur, almost equal. Can't quickly find a language in which the Height one is the shorter, but I'm sure it exists.)

Point is, I would like these buttons to always have identical widths, and not go from o_O to O_o as the translated UI changes. So, get the max of the two widths and assign it as the minimum width to both. Right? Right?

Seems not. The original code, which I'd commented out and then forgotten about, read

        w = max(self.zoom_to_height.width(),self.zoom_to_width.width())
        self.zoom_to_height.setMinimumWidth(w)
        self.zoom_to_width.setMinimumWidth(w)

That was a disaster, I don't know what Qt was returning for a width but the buttons ended up about 500 pixels wide each. No wonder I'd commented it out. But how to do it right? I just want to know the width of the label-text of the button, at run-time, after the locale has selected for the language.

Cutting short an hour's browsing around the Qt forums and the Qt Assistant, I find the answer is quite simple, really. One gets the fontMetrics from the widget. Then you can ask the fontMetrics object for the width() of any string of text, or specifically of the widget's current text. So I end up with an inner subroutine:

        # Function to return the actual width of the label text
        # of a widget. Get the fontMetrics and ask it for the width.
        def _label_width(widget):
            fm = widget.fontMetrics()
            return fm.width(widget.text())

And then apply it like so:

        w = 20 + max(_label_width(self.zoom_to_height),_label_width(self.zoom_to_width))
        self.zoom_to_height.setMinimumWidth(w)
        self.zoom_to_width.setMinimumWidth(w)

The 20-pixel extra is to make sure that even at a minimum squeeze, there will be some margin around the label text. The labels should be pushed to their minimum because they are in an HBoxLayout with stretch items on both sides.

Other than this, and the usual obsessive nit-picking of comment wording, all I had to change was some out-of-date code in the unit test driver. One more down.

Reviewing, and timeline guessing

Today I reviewed worddata.py and pagedata.py, the data models for the vocabulary and pages tables. Didn't find any bugs; made some minor edits to commentary and message texts as usual, and updated the internal doc of methods.

Remaining to review:

• imageview.py
• mainwindow.py
• book.py
• noteview.py
• editdata.py
• editview.py

I expect to finish these this week. editview will take the longest because I also plan significant code changes. In this module, the UI for editing was created using the Qt Designer, and the incorporation of its glob of UI setup makes for awkward code, and is either causing an odd interaction with noteview, or is at least getting in the way of diagnosing it. So I'm going to tear out the generated UI setup and do a manual one.

What happens next? Next is to write the very important, complex, and performance-critical findview module. That will take at least a week. Starting there the sequence is:

• find; 1 week
• wordview, the Vocabulary panel; another week, as I need to work out how to handle the UI to support drag-and-drop moving of words between the vocabulary list and the good-words list.
• chardata and charview, the Character panel, and
• pageview, the Page table panel; about a week to do both.
• fnotedata and fnoteview, the Footnotes panel, a week.
• loupeview, integration with bookloupe: two weeks here because there are many unknowns about how to integrate this hunk of somebody else's C code into Python and to display its output in a useful way.
• translate, the code to parse a DP-formatted book and use it to drive the extremely clever API that I have devised for writing format-translation modules. A week to code and test.
• the etext translator and the HTML translator, example clients for that API, two more weeks. (I may sneak a Markdown translator in at the same time.)

So that's a commitment for (quickly counts on fingers) nine weeks of work, or say to late October. At this point I would have a pretty functional product and would make it available for early adopters who can run from a clone off github, i.e., who have Python 3 and Qt 5.3 (or 5.4, by that time) installed.

Also at this point I would want to download a PP book and actually process it using the new code. Something with footnotes and sidenotes, so probably a month's worth of work to finish. Let's say that takes through November into mid-December.

Then, or possibly before completing a test PP job, I need to spend some days learning and experimenting with pyqtdeploy, which I expect to use to bundle the app (in place of pyinstaller). If I get through this in December, I can put up a Beta version, bundled for OSX 10.9, Ubuntu 14.04, and Windows 8 (the only supported platforms for this version), sometime in January.

Beyond that there will some functional catchup: doing the documentation (2 weeks); doing a new "suggested workflow" doc (another week); actually implementing dragging a panel out of the app to be an independent window (couple weeks probably to get it right); and to bring along the cute little Greek palette from version 1.

Wow. This whole thing could actually get finished in maybe 7 or 8 months. What will I do with all the free time I'll have next spring?

Worry about Lion?

Today's code review was utilities.py, a whole collection of things related to QFile/QTextStream that kind of grew like a fungus while I was coding mainwindow and dealing with opening books. All I did to it was to clarify some commentary and add one minor feature to my "MemoryStream" class. Then I went to run its test driver.

When I'm coding, I usually use my macbook. But for this review phase, I wanted to take advantage of the larger screen on my desktop system. All the files are in dropbox and accessible from both systems. And both systems are set up with Qt/PyQt 5.3, and Wing IDE. So usually it is just a matter of which screen I feel like sitting at. Obviously I can't tote the big machine to the coffee shop, or curl up with it on a lawn chair in the back yard. But other than location, it doesn't usually matter which I use.

Usually. Today I ran the driver for utilities, and when it came to exercising ask_existing_file(), things got interesting.

def ask_existing_file(caption, parent=None, starting_path='', filter_string=''):
    # Ask the user to select a file
    (chosen_path, _) = QFileDialog.getOpenFileName(
            parent,
            caption,
            starting_path, filter_string
        )
    if len(chosen_path) == 0 : # user pressed Cancel
        return None
    return path_to_stream(chosen_path)

It's just a wrapper on getOpenFileName, covering up the awkward API that returns both a path and a filter string, and when a path is chosen, converts it into a QTextStream. Very straightforward and no place where my code could mess up the OS. Right?

So the test driver first calls it with the caption "Press CANCEL", and looks for a return of None. Then it calls it with a caption "Choose unreadable.txt" and I am supposed to use the file dialog to navigate to a file with 000 permissions, which will be detected as unreadable (in path_to_stream) and again return None. Finally the driver calls it with a caption directing me to choose a certain test input file that exists, and it will check that it got a text stream. Very plain-vanilla.

Except it crashed Python. The first, or sometimes the second, or definitely the third entry into getOpenFileName would result in a segfault at location 0000000000000120x, as the crash report helpfully noted the 64-bit address. Most of the time. Sometimes it would die with a "pure virtual method called" abort. But either way, it would segfault deep in the OS X Finder, called from getOpenFileName.

I was not fazed. I knew what to do. I just opened my laptop and ran the identical test, which worked perfectly, no crashes.

The difference? My desktop machine is an old Mac Pro with a 32-bit BIOS that cannot be upgraded past OS X "Lion" (10.7). While the laptop is proudly 10.9.4 "Mavericks". Other than that, the software environment is the same: same Python (3.3) and Qt/PyQt 5.3.

OK, so it looks as if anyone who runs PPQT on Lion is going to be disappointed to say the least, because it will segfault the first time they try to open a file. Should I worry? I think not. I will have to specify it is for Mavericks and up. Too bad about those with an old system that can't upgrade.

I would have a later desktop system but I am determined that my next desktop will have a 4K monitor. And Apple is inexplicably dragging their feet on producing a 4K iMac (which I'd buy in a heartbeat) or a 4K Thunderbolt display, to which I could dock my macbook.

Unreliable paths

Finished coding paths.py which (big surprise) turned out to be more complicated than planned. Basically this is just a module that has a pair of static globals _EXTRAS and _DICTS that contain path strings, and offers get- and set- functions for them. Simple.

The complications come in during initialization. Like its Preference-related buddies colors.py, fonts.py, and dictionaries.py, it has an initialize(settings) function that receives a QSettings, and is supposed to recover from it, the last user-set values for what it manages, namely those two paths. Still simple, like this?

    global _EXTRAS
    _EXTRAS = settings.value('paths/extras_path','some default here')

Not simple. First, what is the default if there is no entry in the settings? Presumably that would indicate a brand-new installation, one where PPQT has never run and shut-down to write some settings. Or the settings were erased. (Or PPQT is on a thumb-drive and was just plugged into a different computer.)

Then, supposing that there is a value in the settings, there is no guarantee that it still represents a valid path. Maybe the path has been removed, or renamed, since the settings were written. It can't be trusted.

Plus, the path to extras and the path to dicts need different default assumptions. It's OK for the dicts path to be a null string, the spellcheck dicts for a book might be in the book's folder.

And the more I fiddled with this initialize(settings) function, the more scenarios came to mind. Finally I settled on this set of policies.

First, there is a function paths.check_path(path) that reads like this:

def check_path(path):
    return os.access( path ,os.F_OK ) and os.access( path, os.R_OK )

It returns False for anything not a valid path to a readable end-point. In particular, it returns False for a null string. So the null string is the appropriate default for the settings.value() calls.

If the "extras" path value from settings passes check_path(), use it. This is the usual and expected case.

If it fails check_path() (either because it wasn't set or is not valid in the present system), look for a folder "extras" in the same location as the app executable, and set that as extras.

If that doesn't exist, set the CWD as the extras path.

If the "dicts" path from settings passes check_path(), as normal and expected, use it.

If it fails check_path(), look for a "dicts" folder on the above-set "extras" path and use that.

Otherwise, leave the dicts path as a null string.

With that all coded up and a test driver written and executing, I could then modify dictionaries.py to use it, and modify its test driver accordingly and run it. That was it for today.

Tomorrow, modify mainwindow.py to use the paths module, and finally check and update the spreadsheet of functions-by-module for all the above. Then I can continue with my easy code review.

Just a quick code review

So, home from a refreshing month in Scandinavia and back to work. I decided I would start by reviewing what I'd done so far, a few thousand LOC in a couple dozen modules, just so it would be fresh in mind. Also, I have some internal documentation, mainly a spreadsheet listing all the public methods by module, and I knew it was out of date, so bring that current as I go.

But basically a quick code review, barely more than a read-through.

Yeah, no.

I started with fonts.py and colors.py, two simple modules that store global choices about, duh, font and color choices. Someday there'll be a Preferences dialog that will use these; for now they only ever set the programmed defaults. And it took over an hour each. Two, 250-line modules. But the prologs were out of date with the code, and the code wasn't in a logical sequence, and some comments weren't clear, and some things weren't being logged that should be...

I actually found one small bug, a typo, in colors.py. But mostly it was tidy tidy tidy format format format.

Then after lunch I tackled another simple module, dictionaries.py, which is the basic interface to spellcheck. It knows how to find the *.dic and *.aff files for Hunspell, how to make a list of available tags, and set up a spellcheck object given a tag.

Except that it needs to know where the user wants to look for those dictionaries. Back in version 1, there was only one place, extras/dicts, a folder of dictionaries I collected and distributed with the app. But that makes it hard for the user to introduce new dicts or custom dicts, because PPQT only looks in extras/dicts and that folder gets replaced if the user downloads a new version. So I've known all along that arrangement had to be broken for V2. There will be a Preference for where to look for extras itself, so the user can copy that to some other location where a new version won't clobber it. And a Preference for what folder to search for dict files, defaulting to extras/dicts.

Actually if the user wants to change dicts while editing a document, there's a two-level search, first in the document's folder, then in the chosen dicts folder. So you can drop a custom dic/aff pair in alongside the book file and we will use that, over another with the same tag in the dicts folder. Anyway, where was I?

Oh yeah, reviewing dictionaries.py. Which had a half-assed version of this logic, and which depended on the mainwindow module telling it where the extras folder was. That shouldn't be mainwindow's job. In fact, I realized, I should have a module that "knows about" the extras and dicts folders, the same way I have one that "knows about" colors and one for fonts. It's just another sub-section of Preference recording. So the next two hours were spent in creating this new module and its test driver, which is about half-done, and then I will be modifying mainwindow and dictionaries to use it.

Just a simple code review.