Plugging along

In two marathon sessions — well, marathon for me, about four consecutive hours each — I completed functional coding of the mainwindow, all the elements of the File menu (Open, Recent->list of recent, Save, Save As, Close). This entailed major and frequent revisions and additions to the "utilities" module, where I have corralled all uses of QFile, QFileDialog, QMessageBox and the like. And some changes to the Book object, trying to get its relationship to the main window just right, efficient and also clear.

But that's pretty well done now. Next big task is to write unit test drivers for both utilities.py and mainwindow.py to ensure every branch is exercised. For the main window that will mean using Sikuli to do visual testing of the GUI, and that's time-consuming (but mega-fun to watch it run when finished). This will take much of next week.

Edit Menu(s)

Meanwhile I've been mulling how to handle the Edit menu. It was a problem with V.1; I never could, for example, get the Edit menu to work right when one of the panels, like the Word panel, had the focus.

Working on the File menu I realized that the menu actions are key. Each menu consists of a list of QActions; and each QAction has a "triggered" signal that is bound to a slot in some QWidget derivative. Well, what happens when the widget to which, say, the Edit>Cut action is bound, is not the focused widget?

This is particularly important when contrasting the Edit menu when a document (QPlainTextEdit) is focused, versus when, say, the Word table (QAbstractTableModel) has the focus. Edit>Copy in the first case means, the current selection of text to the clipboard. In the second, the current selection is some number of cells available as a list, and there may well be processing needed to format them before their value(s) go to the clipboard.

For the editor, there are Edit menu actions I want to support like to-lowercase, to-uppercase that are not appropriate for other panels, and shouldn't even appear (or at least, not be enabled).

And then there's the issue of having multiple books open (which changes everything, as I've lamented frequently before). So let us say that the Edit>To &Uppercase menu action is bound to a slot in the editview object for Book 1. And now the user clicks in the edit tab bar to make Book B the focus of his typing. Different editor contents, different selection, all handled in a different goddam object. Choice of Edit>To &Uppercase now should affect the current selection in Book B, but how is Qt to know that? How to keep it from sending the signal to the editview object representing Book 1?

So I've about concluded that every widget that supports Edit actions needs its own unique Edit menu. And somehow (!) when any such widget gets the focus (focus-in event), it puts its own Edit menu into the application's menu bar; and when it loses the focus, it removes it again.

This takes care of the problem of signal-binding. Each widget that wants an Edit menu, creates its own Edit menu and populates it with such actions as it supports, binding each to the slots in its own code where it does things its way, upon its own data.

But how to swap Edit menus in the menu bar, quickly and simply? I note that what the QMenuBar supports is not menus per se but menu Actions. And I note that QAction supports a property "visible" with a method "setVisible(bool)". So tentatively I am thinking I will have any Edit-supporting widget, upon creation, add its own Edit menu to the app's menu bar. There might be a dozen Edit menus in the menu bar! But it adds it with visible False, and sets visible True on focus-in and False on focus-out. So hopefully only one (or possibly zero) Edit menus will actually be visible in the menu bar.

Does QMenuBar actually support such shenanigans? Damn if I know! If it doesn't, plan B is to have each widget add its menu on focus-in, and remove it again on focus-out, which seems uglier.

Stay tuned, it could be a rocky night...

Default value of f_of_x

Whoof! What started as a modest revamp of my first-draft file handling code has turned into a major exercise of revising and refactoring. But I also found a cool thing to share!

The main window manages the File menu. An important feature of the File menu, as I mentioned yesterday, is the sub-menu of Recent documents. This has to be built and populated dynamically, upon the signal aboutToOpen from the File menu's action. Which means that the program should not spend much time formatting that menu.

Input to the Recent menu is a list of previously-opened files, held as a list of path-strings. The list is kept in sequence by use-time, with the most-recently-used files first. As I mentioned in the prior post, this list can't be sorted by filename, because it might contain two files with the same name but different folder paths.

When populating the menu, the code cannot assume that these files exist. One might have existed yesterday, but it was on a USB stick that is not mounted at this time. So that file should not be shown in the menu. However, a file should not be discarded from the list just because it is not available one time. Before the next time the File menu is shown, the USB stick could be inserted. Then the file should be shown as available.

So there's the situation: at the instant the user clicks on File in the menu bar, the code must run down a list of as many as 10 filepath strings and determine for each,

• Does it exist?
• If so, make a menu action for it

Of what should the menu action text string consist? In BBEdit, the string is "Filename emdash folderpath". In the Wing IDE that I use, it's "Filename (folderpath)". Mac default apps like TextEdit and Numbers don't bother with paths; they just show filenames. OpenOffice has "n: fullpathstring" for n from 1 to 9. PPQT V1 also displays an index number, "n filename" (no colon). The index number is also set as the accelerator key for Windows, so a really hot Windows user could key alt-F alt-R alt-3 to open the 3rd most recent file. A pretty silly feature IMO. I believe for V2 it will be "n filename (folderpath)".

This means that the logic of populating the sub-menu is, for each pathstring in the list:

• if the file isn't accessible, continue
• split the path into (filename, folderpath)
• make the string "{0} {1} ({2}).format(index,file,folder)"
• make a QAction and add it to the menu

Caching

ALERT! The following had a major although un-obvious error, which is corrected below.

Checking for accessibility means asking, does the file exist, is it really a file (not a directory), and is it readable? These are methods of the QFileInfo object, as is the ability to fetch the filename and folderpath separately. But that means instantiating a QFileInfo, a moderately expensive process. Since the same set, or an overlapping set, of path strings will be checked again and again, it makes sense to cache the QFileInfo objects. The first time a path is checked, make a QFileInfo. If (when) it is checked again, re-use that QFileInfo. Good concept; how to implement?

There exist nice "memoization" solutions for Python, generic code that lets you put a decorator @memoize on a function to automatically cache the result that corresponds to each unique argument. However in this case, I don't want to cache the result for a given path string, I want to cache an intermediate value, the QFileInfo for that string, so I can use it in a different function instead of recreating it. So memoization is out.

What I need is something like collections.defaultdict: a dictionary whose keys are pathstrings, and whose values are QFileInfo objects upon those pathstrings. When the dictionary is queried for a given key, and key is not present, it should set a default value of QFileInfo(key).

Unfortunately, this is not what collections.defaultdict provides. It allows you to provide a "default factory" that is a fixed value, or a classname such as list, in other words the default factory is a constant function f(k). What I want is a defaultdict that provides a default factory f(key), where the default value is a function of the missing key. And indeed, defaultdict provides a way to do that, by overriding its __missing__() method. Update: When I first wrote this, I thought that __missing__ did the whole thing: provided a default value for the missing key, and stored that value as the value of the key. This was not so! It is necessary for __missing__ to save the default value explicitly, as shown below.

Without further ado, here is generic default dictionary that does this:

from collections import defaultdict
class key_dependent_default(defaultdict):
    def __init__(self,f_of_x):
        super().__init__(None) # base class doesn't get a factory
        self.f_of_x = f_of_x # save f(x)
    def __missing__(self, key): # called when key is not defined
        ret = self.f_of_x(key)  # calculate the default value of key
        self[key] = ret         # save for future uses
        return ret

To use this, create an object of this class passing the name of a function of one argument that returns an appropriate default value. In my case I used it so:

_FI_DICT = key_dependent_default( QFileInfo )

def file_is_accessible(path):
    global _FI_DICT
    qfi = _FI_DICT[path]
    return qfi.exists() and qfi.isFile() and qfi.isReadable()

# Split a full path into a tuple (filename, folderpath)
def file_split(path):
    global _FI_DICT
    qfi = _FI_DICT[path]
    return ( qfi.filename(), qfi.canonicalPath() )

The mainwindow will call file_is_accessible for some path, and soon after, will call file_split for the same path. The first time, a QFileInfo is created. On every subsequent call, the same QFileInfo is interrogated.

The code for key_dependent_default imposes no limit on the size of the dictionary. In my case, there will be at most 10 recent files in the settings when the app starts up, plus as many unique files as the user opens during the session. So I am not concerned about the size of the dictionary. But it would be possible to code the class so it limited its own size. When it reached the limit, it could simply stop caching new items, or it could delete one of its own members at random, or with more work, it could delete the least recently used member.

Nesting in Parentheses

Today I meant to tear into both the Book and Mainwindow modules for major changes. As mentioned in Friday's post, mainwindow needs to stop keeping track of files as a dictionary {filename:path-to-file}. Instead, I worked out over the weekend, it must keep lists strictly of entire, absolute (non-relative) path-strings. Nevertheless, there are often times when the code, given a filepath, needs to get the basename from it and similar tasks handled, in Python, by members of os.path. There is an interesting comparison to be drawn between the facilities of os.path and Qt's QFile, QDir, and QFileInfo. Perhaps I'll find time to get into that tomorrow.

Because today I sat down at my desktop machine with its Cinema Display to do some development there. Most of the coding I've done so far this year has been performed slouched on my spine with my laptop on my tummy. But for making major changes spread over two biggish files, I wanted the big screen, better keyboard, and trackball of the desktop machine.

Almost immediately this ran into problems. Running any kind of test produced, first, an error on import regex, the extended regular expression module. Oh. Hadn't installed that on the big machine. Ok, do that: and then followed a half hour's digression attempting to get easy_install (misnomer!) and pip (it's not a pip) to work, and finally downloading the module from pypi and manually running its setup.

Restart and immediately... No module named hunspell. Oh, right. I never installed the hunspell interface on this machine. So do that. Almost immediately after, No module named blist. Oh. Right. So back to pypi and get the blist package and install it, and it produces a baffling error message. Its __init__.py imports blist._blist which is right there next to blist itself but Python can't seem to find it. In trying to figure that one out, I happened to notice that in /Library/Frameworks/Python/Versions, the link Current pointed to 2.7, not to 3.3, although 3.3 is being used. Hmm. sudo rm Current and suddenly blist imports like a charm.

Well, all righty then. Back to trying to run book_test. Oops, an error from code that's been working for weeks,

self.scroll_area.setSizeAdjustPolicy(QAbstractScrollArea.AdjustToContents)

...produces an error, "QScrollArea object has no member setSizeAdjustPolicy. Well, it certainly does, or I wouldn'a coded it. Verify that by looking in the Qt Assistant. Yup, there it is, oh wait, "added in Qt 5.2". Ooooohhhh. Hastily enter in the Wing IDE Python window,

from PyQt.Qt import PYQT_VERSION_STR
PYQT_VERSION_STR
'5.1.1'

So although this machine has Qt 5.2, it has a back-level PyQt. Which means, it's time to upgrade the machine to PyQt/Qt 5.3. Download the latest Qt, Sip, PyQt, run their configure, make, and install steps. And there goes most of the rest of the afternoon. Sure glad I don't have an anxious manager wanting to know if I'm going to meet my delivery goal...

What a tangled web we weave...

The mainwindow class defines everything to do with the File menu: new, open, close, save-as, and a submenu of "recent" files. The latter is a feature I often find useful. I don't have to remember anything about a file except that not long ago I edited it. Just go to the end of the File menu and there it is, without having to navigate the file system to find it. However this is a feature that has some subtle implementation details, some of which I am only just now remembering as I start to code the version 2 of it. I should have reviewed V1 more carefully.

When coding the initial form of this I decided rather hastily that I would keep the list of "recent" files as a dict with filenames as keys and their paths as values: {filename.typ:path-to-file}. Of course you see immediately the problem with this, don't you?

Duplicate filenames! What if the user opens ~/documents/fred.txt and (then or another day) opens /pgdp/history/fred.txt? They are (presumably) different files but a Python dict allows only one value for the key fred.txt. Looking back at V1 code I see that I kept a simple list of full pathnames, with the most recent at the front. Much better.

Unfortunately the {filename:path} dictionary idea is in several places in the code and in the unit test driver. So now I am looking at changing 50 lines of code or more scattered in two or three modules. Because I didn't take the time to look at the working V1 code. Oh, bleagh.

Another subtlety of the recent-files list is, what if the file no longer exists? Or, much the same thing, what if it was opened from a mountable drive (e.g. a USB dongle) that isn't mounted just now? The file could appear or disappear between uses of the File menu.

The answer to that is, that the Recent sub-menu has to be (re)built dynamically, on the fly, when the File menu is about to be displayed. There is an aboutToShow signal emitted by the File menu action itself. The slot for that signal must have the code to clear and repopulate the Recent sub-menu with an action for each file in its list that: (a) is not already open but (b) still exists and is accessible. (Note that if a file is not now accessible, it doesn't go in the menu, but does stay in the list, so if it comes back at some future time, we'll show it.) I modeled the V1 code after an example in Summerfield's book. I remember thinking at the time, really? All this code gets executed between the mouse's click on the word File, and the painting of the menu? And nobody notices a delay? I still think it's remarkable.

A related issue is one that did not arise in V1. When the user selects File>Open and chooses a file, what to do if that's a file that's already open? We do not want to open a second copy for sure. But that means it needs to be easy to detect when a chosen filepath is identical to one of the possibly-several files already open. If it is, don't proceed with the open, but do "focus" that file: make it the currently selected tab in the edit tabset, so it's visible.

Yet another difference from V1: when the close signal is received, V1 looked to see if its one-and-only document was modified, and if so, gave the user the choice Yes to save, No to not save and continue with the Quit, or Cancel to not Quit. But with V2, it is possible there are multiple modified documents at Quit time. So the warning message at least needs to say how many modified documents there are. But should a Yes reply mean, save everything? That is, do a File>Save for each modified document? A problem with that is, some of them might be modified New documents with filenames of "Untitled-n" and no related path string. They need to be treated as Save-As, with a file-save dialog so a proper name and folder can be chosen.

Or, should the close event just loop through all open documents, and present a separate modal dialog for each one that's modified: "File filename.typ is modified. Save it now?" Yes/No/Cancel. Or better, Save/Don't Save/Cancel Quit. For normal files, clicking Yes gets an instant save. For "Untitled-n" documents, a Yes is immediately followed by presentation of the standard Save-As dialog.

That makes everything clear, but it could mean that the user who tried to Quit or clicked the [x] button in the window border is presented with a sequence of modal dialogs one after another. If the user is under stress (told to hurry up and get off that machine for some reason) this could be quite annoying. But what alternative is there?

BBEdit's way

Well, here's one alternative. When I tell BBEdit to quit with two open, modified documents, it... quits! No dialogs, no warnings. And the files on disk do not reflect the changes, so there was no secret saving going on.

When BBEdit is restarted, it shows a small progress message "Restoring BBEdit State" and then it opens the two modified documents and shows them with the modified text and a state of modified. How does it do that?!? It must save a secret copy of any modified file so it can recover that state on startup. But where?

Dude, where's my tab?

Well, rolling along here at ppqt central, we have the basics of a main window, now it is time to add the File menu and the actions it commands. First up is File>New. Easy-peasy one would think, since the _new() method is already in place and being used as part of initialization.

        self.file_menu = self.menu_bar.addMenu(_TR('Menu bar', '&File'))
        # Populate the File menu with actions.
        #  New -> _new()
        work = self.file_menu.addAction( _TR('File menu','&New') )
        work.setShortcut(C.CTL_N)
        work.setToolTip( _TR('File:New tooltip','Create a new, empty document') )
        work.triggered.connect(self._new)

Add a QMenu('File') (appropriately translated) to the application QMenuBar. To it, add an action 'New' and connect its signal to the proper slot. And this should work. So test it. Up comes the app and oooh! It has a File menu with only one item, New. And selecting that executes the code of the _new() method. I have verified this and also that each time File>New is selected, the app creates an empty Book with the names Untitled-0, Untitled-1, Untitled-2 and so on, and adds them to the edit tabset on the left side. And the tabset.addTab() method returns a tab index of 0, 1, 2 as expected.

Except ... the tabs don't appear. The added tabs are not shown; the only visible tab is the first one, for "Untitled-0". Wut?

The code for adding a New document is exactly the same as for adding an existing opened file, and that works. The unit test driver fakes up settings showing two previously opened files and the app opens them and they appear in separate tabs (as shown two posts back).

OK run the unit test and it opens two existing books. Now do New. Hmm. The debug printout reports "added Untitled-1 at index 1" even though there are two open books, so the name should be "Untitled-2" and the index, 2. Add the printout to the _open() method as well.

added realbook.txt at index 0
added small_book.txt at index 1
added Untitled-1 at index 1

No, that's not right. Should be Untitled-2 at index 2. It's like the _new() was happening in a completely different Mainwindow object, operating on a different book sequence number and different tabset. Could that be?

Yes it could be! The unit test driver created three MainWindow objects in sequence. The first is used to check that various settings are properly saved, like this:

settings.clear()
mw = mainwindow.MainWindow(settings)
...do things like mw.move(new position)...
mw.close() # force writing settings
...read values out of settings and assert things about them...
mw = None # destroy main window

And then another round of that: load the QSettings with fake info; create a new MainWindow object in mw; close it; look at settings values; assign mw=None.

Then finally do that one more time: load the QSettings object this time with two valid "previously-open" files; create mw=mainwindow.MainWindow(settings); call app._exec() and interact with the window.

If the first two rounds are commented out, so only the third MainWindow is created, it works: File>New makes a new file that appears in the edit tab bar.

Conclusion: somehow the QApplication is holding onto one or both of the first-created MainWindow objects. And it remembers how that (or those) objects created File menus. And somehow the File>New action is being directed to the local data owned by one of those zombie main windows, even though both of them had executed their .close() methods and all references to both had been overwritten in the Python code.

So: split the unit-test into two or three separate programs... boring.

Edit: On later consideration, I think the link is in the File menu QActions. Each QAction has a reference to the "slot" to handle its the triggered() signal. Each slot is specified using the main window's "self" pointer, e.g.

    act.triggered.connect(self._new)

That "self" value is a reference to the current instance of the MainWindow class, a QWidget derivative. Each time I created a MainWindow instance, it initialized by creating a QMenuBar and putting File>New and File>Open actions in it. Somehow the first of these "stuck" in the app's memory, and kept that object alive, not garbage-collected, even though I'd deleted every reference I had to it. Any use of the Mac OS menu bar's File>New, regardless of the fact that MainWindow #3 was running and active, still invoked the code in the zombie MainWindow #1 (or #2?).

Don't think of it as a bug...

...think of it as material for a blog post!

OK so there is an annoying bug in the edit display. I am putting a pale highlight on the current line (the line with the cursor) -- see the image on this post. This fixes an annoyance I had with version 1, that on a large screen it was easy to lose track of the cursor, especially after paging up or down. I'd often have to rattle the left and right arrow keys so I could spot the cursor moving.

Here's the basic code, cut down:

Note: The following is not the correct way to set a current-line highlight. Do not emulate this code. See this post for the problem with it and a later post for the correct approach.

    def _cursor_moved(self, force=False):
        tc = self.Editor.textCursor()
        self.ColNumber.setText( str( tc.positionInBlock() ) )
        tb = tc.block()
        if tb == self.last_text_block and not force :
            return # still on same line, nothing more to do
        # ...here fill in line-number, scan filename, and folio widgets...
        # Clear any highlight on the previous current line
        bc = QTextCursor(self.last_text_block)
        bc.setBlockFormat(self.normal_line_fmt)
        # remember this new current line
        self.last_text_block = tb
        # and set its highlight
        tc.setBlockFormat(self.current_line_fmt)

That's pretty simple: if the cursor is on the same line as last time, set the column number widget and bug out quick. (The force argument is so this same method can be used in certain rare cases by other routines to repaint the widgets even if the cursor hasn't moved.)

If the cursor is on a different line than the last time it moved, which we will know because the QTextBlock returned by the current cursor is not the same as before, then clear out the highlight on the previous line/block, and set a highlight on the new line/block. It never sets a highlight without also clearing one.

And it mostly works. Move the cursor with the arrow keys or by mouse-clicking, and the pale highlight moves around with it. Great!

Except in one case: if I select multiple lines—by dragging or by shift-clicking or by shift-down-arrowing—then all selected lines get the current-line highlight and it does not clear if I click somewhere else. It only clears if I run the cursor back over those lines. And here's the kicker: as I drag or shift-down-arrow to extend the selection, the line-number widget is updating properly. So the code above is being executed, recognizing that it is on a different line (or it wouldn't update the line-number widget), which means it must be clearing the highlight. But it doesn't. WUT?

Time to put in the print statements.

        dbg1 = tb.blockNumber()
        dbg2 = self.last_text_block.blockNumber()
        print('clearing block {0} setting block {1}'.format(dbg2,dbg1))

And it reports exactly what it should: Put the cursor on line 0, key shift-down-arrow to make a selection, it reports "cleaning block 0 setting block 1, clearing block 1 setting block 2" and so on, but the highlights on line 0 and 1 are not clearing. They are partly hidden by the brighter selection highlight, but there. Here's a picture.

If I click on line 4, the highlight on line 2, the last current line, is cleared, as is the selection highlight, but the highlights on lines 0 and 1 remain.

So there's no logic error, but some issue with setting a block format when the block is under another format, the selection highlight. I note that setting the current_line_fmt works, even when selection is in progress. You can see it sticking out from under the selection on line 2 above. But setting the normal_line_fmt doesn't take. What's the difference? It is initialized so:

        self.normal_line_fmt = QTextBlockFormat()

It's an empty, default format object. Whereas the current line format is set up:

        self.current_line_fmt.setBackground(colors.get_current_line_brush())

It is a format object with an explicit background brush. (The colors module returns basically QBrush(QColor('#FAFAE0')); eventually the user will be able to choose this color as a Preference.) Anyway the current line format has an explicit background brush and the normal format does not. Should I give it one, and if so, what color? Back to the Qt Assistant...

Answer: No. I changed the initialization of the normal format:

        self.normal_line_fmt = QTextBlockFormat()
        self.normal_line_fmt.setBackground(QBrush(Qt.white))

I also tried Qt.color0, the transparent color. The bug continues. So it is not the lack of an explicit background brush that prevents the clearing. Back to the Assistant...

Ah. Re-reading the doc for QTextCursor.setBlockFormat() I note it reads "Sets the block format of the current block (or all blocks that are contained in the selection) to format." Oops.

The problem, I betcha, is in the line

tc.setBlockFormat(self.current_line_fmt)

Because tc is the actual cursor of the current document, the cursor that represents the current selection. And when it is used to set the block format of the "current" block, it sets a block format for "all blocks in the selection" which includes the prior block whose format was just cleared!

And that's it. If I set the current line format using a new cursor with no selection, it all works. So the final lines of code now read:

        # clear any highlight on the previous current line
        temp_cursor = QTextCursor(self.last_text_block)
        temp_cursor.setBlockFormat(self.normal_line_fmt)
        # remember this new current line
        self.last_text_block = tb
        # and set its highlight
        temp_cursor = QTextCursor(tb)
        temp_cursor.setBlockFormat(self.current_line_fmt)

Make a new text cursor based on the current block, one which has (by default) no selection. Use that to set the format for the current line. It affects only the actual current line, not any other lines that might be in the current user selection. Bug fixed! Do a commit and break for lunch!

Mainwindow debut

The main window and the Book classes, and the relationships between them, are now pretty well laid out and functional, as proof of which see:

Click to see the whole retina-scale megillah

Note there are two documents open (two tabs on the left side). This is the first actual user-visible enhancement beyond version 1.

There are a few minor issues immediately apparent when trying it out, but they should be easy to sort out. (The editview seems to have regressed a bit, for one thing.) After fixing those, the next piece of code to write is the File menu and a few actions for it.

So, whee! And a pleasant Memorial Day Weekend to those in the USA, and a pleasant ordinary weekend to anyone else!