byteplay ok on linux

Oh my but this was a strenuous afternoon. I believe I shall now drone on about my adventures, just because the story ends well.

First, I installed Ubuntu 14.04LTS on my 64-bit dev virtual machine. I kind of hated to do this because the install ISO does not offer what to a Mac user is the sensible choice, of reinstalling the OS without changing the user files. Nope. Either it will install "beside" the existing system, meaning in a different disk partition, or it installs "over" the existing system which wipes all your user files and settings. So I did the latter, knowing full well I would have to reinstall all my dev tools and dependencies.

In a month or two I will need to install Py/Qt5.6 and some Python modules, so I can upgrade PPQT and Cobro. But for the time being, I only needed to get a working Wing IDE and Python 3.5. The Wing IDE is a simple install, then activate with my license key.

Python.org does not offer an installable package for Linux. There are Ubuntu/Debian packages for Python 2.7 and for Python 3.4, but I needed 3.5. The only way to get that is by downloading the source package and making it. Well, I've done that often enough. Download, unzip, make, make test, sudo make install. All seems to go well until I try to start Wing, and it can't start Python. "Missing module _struct".

DuckDuck that phrase and you will find that hundreds of people have encountered it. What does it mean? It can be properly translated as, "you suck at installing software". Something didn't go well in the make-install step. More missing things show up when I find out that neither pip nor easy-install were installed. When I try to download and run get-pip.py, it fails with an error because the SSL lib wasn't installed, and also the zlib was not installed.

Scrolling back in the make output I find a list of modules that it couldn't locate. Did it put the list at the end where a person would notice it? Of course not! It buried it between a couple of hundred lines of output before and after. Did it tell you what to do? Of course not! Well, it did; it helpfully advised that you "look for the name of the missing module in setup.py".

Which I did, and saw where it built a list of libraries to search for things. Then I got a terminal window and used the find command to find the things it was not finding. Oh, there they are, in a special 64-bit library. Add the path to that library to the code of setup.py and rerun the make install. OK then! Now I've got pip, and Wing is working, whee.

So now I can attempt what I set out to do a couple hours earlier, pip install byteplay3. Which of course fails with an obscure message, "Could not find a version that satisfies the requirement". What requirement? Back to DuckDuckGo. Oh yeah, plenty of people were having this problem a couple years ago, when pip began to require version numbers that complied with PEP400. I read PEP400. No, my version number fits the pattern. But in one of these postings I see a remark that, oh, also, pip does not support modules that are not hosted directly at PyPi. A module hosted at github, for example. Oh. Which I am doing.

So I find out about setup.py's build and sdist and upload commands. And find out the hard way, that you can't do

python setup.py sdist
python setup.py upload

oh no no no! You have to do

python setup.py sdist upload

all one line. Because...? Who knows.

Anyway, did that, and then in my Linux system and my Mac OS system both I could do pip install byteplay3 and it did it and I could start Python and run the example from the readme, and it worked.

Tomorrow, Windows. I am so looking forward to that.

Function signatures; bytecode lacks "computed go-to"

Well, sure, there was a bug. A good thing I blogged about it, or who knows it would have gone untested and unfound?

The Code class represents a code object, but in a form that can be manipulated. It has two crucial methods. from_code() accepts a Python code object and captures all its contents, returning a new instance of Code class. I had properly updated that method to notice the Python 3 features of varkwargs and the kwonlyargcount.

I had not completely dealt with these in the other method, to_code(). Calling to_code() of a Code object returns a code object that is supposedly equivalent. But I had not included the kwonlyargcount in the calculation of the code.co_argcount. So it was off by 1, causing a TypeError exception, wrong number of arguments, when you called the code.

Testing for that also revealed a bug in my unit-test scaffold code. But it's all good now.

Tiny Basic and the computed go-to

In the first post in this series, I mentioned that one use of byteplay would be to implement domain-specific languages using Python bytecode as a the implementation language. And I suggested I might try to implement a Tiny Basic compiler in this manner. Turns out? Not so much.

Historic background: Tiny BASIC was the term for several, minimal BASIC interpreters for early microcomputers. The first was written by Tom Pittman of Itty Bitty Computers. The more influential version—because it was for the 8080 where Pittman's was for the 1802—was by Li-Chen Wang. I was aware of Tiny BASIC although I never used it. (I programmed my Z80-based CP/M system in assembler, thank you.)

Anyway, Tiny BASIC is such a minimal language that its interpreter, including an adequate editor, can be implemented in a few kilobytes. On a walk yesterday I thought about it and at first got rather excited about the possibility. I quickly arrived at a program structure (a dict keyed by the line number with the text of the line as value) and visualized how I could use the built-in compile() function to reduce expressions to bytecode, and glue those bytecode bits together with more bytecode and thus produce a whole Python function from a BASIC program.

Then, sitting in a coffee shop, I used my phone to look up the syntax of the language...

...and is it not a fabulous age we are living in, where one can be sitting over a capuccino and have the passing notion, "I wonder what was the syntax of a programming language last used over thirty years ago?" and be reading the answer in less time than it takes to describe it? Seriously, people, why are we not all happy as kings?

So, here's the manual. Less than five minutes of skimming on the little screen of the phone revealed a major, major problem.

The problem is that Python bytecode has no real "jump" instruction as found in an assembly language. It has several jump opcodes, for example POP_JUMP_IF_TRUE and JUMP_ABSOLUTE, but the argument to all of these is a fixed integer offset in the bytecode string. The amount to jump forward or backward, or the absolute offset to jump to, is hard-coded in the instruction. There is no way in bytecode language to say, jump to the offset encoded in the top-of-stack item.

Without such a "computed go-to" it becomes much more difficult to implement Tiny BASIC. Because Tiny BASIC has both a computed GOTO and a GOSUB. Of the GOTO, Pittman's manual explicitly says, "the next statement to be executed after a GOTO has the line number derived by the evaluation of the expression in the GOTO statement. Note that this permits you to compute the line number of the next statement on the basis of program parameters during program execution. (my emphasis)". The GOSUB presents the same problem in two ways. First, it takes an expression, so the actual target is determined at execution time. Second, it stores the return location on a stack for use by the RETURN statement. RETURN is effectively a GOTO where the destination is fetched from the call stack.

None of these things can be implemented using a bytecode jump. So that pretty well ends any thought of compiling a Tiny BASIC source file into a single Python function.

I did think of a complicated mechanism, basically each line of the BASIC program would compile into a separate function that could operate on globals (the BASIC variables) and must return the number of the next line to be executed, with None meaning, "next sequential". But it would be kind of ugly and clumsy and not a good demo of byteplay3.

Byte-playing, a new project to doodle on

So. Been awhile.

Going to introduce a different project I've been messing with for a few weeks: byteplay3.

More on that in a minute. First the status of old projects.

PPQT2 continues to have one or two regular users, and a minor UI issue was posted a couple of weeks ago. No real bugs found; is that good news? It says something about the code quality; but I suspect it says more about how nobody is using it. I would be a regular user, if I were still doing PGDP Post-Processing. Unfortunately, the switch to EPUB with its unavoidable compromises on book quality has killed my interest in that. So I'm not PP'ing any more.

Anyway, I want to fix that UI issue and bring it up to the latest levels of its dependencies. Qt 5.6 is due out shortly, with PyQt5.6 to follow soon after. So when I can install those levels, I'll do the code update to the Find panel and rebuild on all platforms. That will probably be the end of PPQT2. Well, it was a most satisfying hobby project to design and build.

I remain a daily user of CoBro, and it definitely needs to be refreshed. It embeds the Qt WebEngine. Recently a couple of the comics I read have stopped loading, I think because they are insisting on a higher or different level of https encryption than this old WebEngine module supports. So I will also rebuild CoBro with Py/Qt5.6 and, one hopes, it will stop giving an obscure error when it tries to load Penny Arcade.

But that's all to do in a couple of months, whenever the Py/Qt upgrade happens.

The new project, byteplay3, needs to be at Python 3.5. I've been making do with 3.4 for a while, and there's no function in 3.5 that would benefit PPQT, but for byteplay I need to test against async coroutines and such.

Byteplay

The original byteplay, written by Noam Raph, was uploaded to PyPi in 2010. Briefly, the point of byteplay is to make it easier to diddle with the bytecodes generated by the Python compiler. You, gentle reader, are a highly competent Python user, so I will only show the meat of the example code. I'm sure you can figure out what's going on.

>>> def f(a, b):
...   print(a, b)
...
>>> f(3, 5)
    3 5
>>> from byteplay3 import *
>>> # convert code object of function f to a Code object
>>> c = Code.from_code(f.__code__)
>>> c
    <byteplay3.Code object at 0x1030da3c8>
>>> print(c.code)
2        1 LOAD_GLOBAL          print
         2 LOAD_FAST            a
         3 LOAD_FAST            b
         4 CALL_FUNCTION        2
         5 POP_TOP              
         6 LOAD_CONST           None
         7 RETURN_VALUE         
>>> c.code[4:4] = [(ROT_TWO,None)]
>>> f.__code__ = c.to_code()
>>> f(3,5)
    5 3

Short intro follows. If you understood perfectly what happened in that demo, skip ahead. Or for a longer explanation see the detailed "about" page that I've made (starting from Raph's original).

When Python processes a def statement or lambda or compile() expression, it compiles the source text into an internal form. The heart (though by no means all) of the internal form is a byte array, a string of "bytecodes" which represent machine instructions for a simple stack machine. The bytecode representation is binary and designed first for speed of execution and second for compact storage. Readability and post-compile editing were not goals of that design.

The standard module dis will display the bytecode of a function or compiled expression in much the same format as shown in the example. In fact, dis of Python 3.5 has a nice facility that lets you read a bytecode stream one instruction at a time from an iterator. (I think this preempts a couple of the usual uses of byteplay. But it still has some value.)

That takes care of displaying the bytecode of a function. But what if you want to modify it? In the preceding example, a ROT_TWO instruction is inserted in the sequence and that changes the function's behavior. More realistically, there are many opportunities for peephole optimizations, where you look for inefficient code sequences and shorten them. Ryan Kelly made the promise package based on the original byteplay. It provides decorators that optimize certain code sequences of your functions.

Or, I can imagine wanting to generate a bytecode sequence starting with some other notation. You could design some little Domain-Specific Language, and compile it down to bytecodes, and call it from a Python function. I'm considering doing a demo of byteplay3 in which I implement, say, Tiny Basic by compiling it into bytecodes. Python bytecode: the poor man's LLVM!

Kelly's promise module, and the original byteplay, are firmly dependent on Python 2. I thought it would be fun to bring byteplay, and maybe promise, into the world of Python 3. And that's what I've been doing in the odd spare hour for the past month or so.

This is long enough; I'll delve into some of what I've done next time.

Meanwhile ... if Noam Raph is out there? I'd love to talk! You aren't on Facebook or LinkedIn nor a user on github...