eating cucumbers without permission...

Oh, that’s right, I have a blog! I’ve got a bunch of old projects to document, but first, a minor diversion of this past week.

Eliazar Parra published a fabulous user-script to add “backbars” to social link sites like Digg and Reddit, where content is voted on and scored. The idea of a backbar is that each item is unobtrusively shaded with a background that corresponds with its score.

The backbars effectively transform the page into a bar chart, and in some cases it’s a spectacular improvement. His Stack Overflow example highlights the shortcomings of the original design, which disguises entire orders of magnitude by turning “39,9000″ into “39.9k”, which has the same visual weight as “3,996″.

Backbars make all the difference:

Love it!

Generating Images

However, the original implementation has external dependencies, most vexing being coloured rectangles hosted on the author’s site. As a result, you can’t even change backbar colours.

This led down the rabbit hole into dynamically generating the images from within the script. Most formats are frustratingly complex to implement (I am not porting zlib to JavaScript), so it had to be simple bitmaps.

Luckily, they don’t only come uncompressed; there are several flavours of run-length compression, a simple scheme whereby you replace a series like redpixel + redpixel + redpixel... with a single redpixel * 100. In the case of these big blocks of colour, it takes 2 bytes to encode 255 pixels, quite a staggering improvement over the 1020 bytes of the uncompressed 24-bit version.

The end result: jsbmp, a small library for generating bitmaps in JavaScript.

Pretty useless, but there are a few intriguing possibilities. Sparklines, maybe?

Not Invented Here

And then of course I had to do something about jQuery, so I ended up reimplementing the script. Enter Admiral Backbar. (It’s a trap!)

Installing Roundup 1.4.6 at Dreamhost stretched from the “15-30 minutes” specified in the installation docs to something more like four hours. This is a collection of notes for next time.

Dramatis Personae

Roundup is the software. It’s billed as “a simple-to-use and -install issue-tracking system”, but it’s so configurable that it’s probably better described as a lightweight tracker-oriented framework.

The Roundup installatation includes the Roundup module — i.e., what you get when you import roundup — and a set of administration scripts: roundup-admin, roundup-server etc.

You can use the roundup-admin script to create a tracker. If Roundup is a framework, the tracker is the application. It includes a data schema, page templates, extensions, custom behaviours and so on.

To access the tracker you will need to set up the web interface.

1. Installation

If you get the source distribution, the instructions suggest that you install it. Don’t.

Which is to say, the docs assume that you are a server administrator attempting to add the Roundup module and associated scripts into your Python installation directory. If you run this:

`python setup.py install --prefix test_directory`

The result is:

• test_directory/Lib/site-packages/roundup/ — Roundup module
• test_directory/Scripts/ — platform-specific administration scripts
• test_directory/share/roundup/templates/ — included templates
• test_directory/share/roundup/cgi-bin/ — included web interface
• test_directory/share/locale/ — translation files

i.e., files that slot neatly into a Python install. Because I have not installed a custom version and obviously can’t touch the server-wide installation, this structure is not ideal.

2. Be Careful About Changing The Directory Structure

I want the Roundup module in my directory for Python libraries, /home/me/pylib. I don’t care about locales and the scripts are just thin wrappers around Python files in roundup/scripts.

It’s the templates that are the problem. The comment on the listTemplates function in roundup/admin.py reveals the 5-step process Roundup uses to find them:

Look in the following places, where the later rules take precedence:

1. <roundup.admin.__file__>/../../share/roundup/templates/*
   this is where they will be if we installed an egg via easy_install
2. <prefix>/share/roundup/templates/*
   this should be the standard place to find them when Roundup is installed
3. <roundup.admin.__file__>/../templates/*
   this will be used if Roundup’s run in the distro (aka. source) directory
4. <current working dir>/*
   this is for when someone unpacks a 3rd-party template
5. <current working dir>
   this is for someone who “cd”s to the 3rd-party template dir

Either throw the templates directory into /home/me/pylib too or only ever run the roundup-admin script from a template directory. Or, if you don’t already have a directory for libraries, just run it from the source distribution.

3. Create a Tracker

Run roundup-admin if you’ve installed Roundup and the script is on your PATH, otherwise python roundup_admin.py.

Type install and follow the prompts:

Enter tracker home: /home/me/tracker/
Templates: classic, minimal
Select template [classic]: classic
Back ends: anydbm, mysql, sqlite
Select backend [anydbm]: mysql

If the template or backend files can’t be found, it won’t be possible to select them, so exit and fix the problem.

You’ll have to come back to roundup-admin later, but now it’s time to configure the tracker.

4. Configure Your Tracker

Head to /home/me/tracker/ and edit config.ini. Search for “NO DEFAULT” to find the items that need to be set. The tracker won’t run at all if they’re not.

In the Dreamhost context, consider a subdomain — http://tracker.example.com/ — because it will make deployment much easier.

Return to roundup-admin and initialise the tracker to set up the default user accounts and roles.

5. Creating The Web Interface

I spent several hours failing to track down mysterious bugs apparently caused by some combination of my tracker settings and the bundled CGI interface. Try using the WSGI interface instead:

#!/usr/bin/env python2.4

# Enable HTML tracebacks
import cgitb
cgitb.enable()

# obtain the WSGI request dispatcher
from roundup.cgi.wsgi_handler import RequestDispatcher
tracker_home = '/home/me/tracker/'
app = RequestDispatcher(tracker_home)

from wsgiref.handlers import CGIHandler
CGIHandler().run(app)

Note that wsgiref isn’t included in Python 2.4. Dreamhost isn’t providing 2.5 yet, so just download the package and put it somewhere handy.

6. Deploying With Passenger

You can use Phusion Passenger to deply WSGI applications; the wiki has more details. Just set it up in the panel and modify the web interface a little:

#!/usr/bin/env python2.4

import sys, cgitb

# Enable HTML tracebacks
cgitb.enable()

# Even if you've got your paths set up to find
# your python libraries automatically, Passenger's
# interpreter won't.
sys.path.append("/home/me/pylib")

from roundup.cgi.wsgi_handler import RequestDispatcher

# The WSGI app has to be called "application"
application = RequestDispatcher("/home/me/tracker/")

# That's it.

Save it as passenger_wsgi.py and you’re good to go.

Mark Helprin’s Winter’s Tale is a delight, even if its apparent themes are little more than an excuse for the wordplay window-dressing. I might not have found perfect justice, but a hall of light and mirrors built from language is quite enough for me.

One thing that needled, not being the sort to read with dictionary in hand, was this:

Though Mrs. Gamely was by all measures prescientific and illiterate, she did know words. Where she got them was anyone’s guess, but she certainly had them. Virginia speculated that the people on the north side of the lake, steeped in variations of English both tender and precise, had made with their language a tool with which to garden a perfect landscape. Those who are isolated in small settlements may not know of the complexities common to great cities, but their hearts are rich, and so words are generated and retained. Mrs. Gamely’s vocabulary was enormous. She knew words no one had ever heard of, and she used words every day that had been mainly dead or sleeping for hundreds of years. Virginia checked them in the Oxford dictionary, and found that (almost without exception) Mrs. Gamely’s usage was flawlessly accurate. For instance, she spoke of certain kinds of dogs as Leviners. She called the areas near Quebec march-lands. She referred to diclesiums, liripoops, rapparees, dagswains, bronstrops, caroteels, opuntias, and soughs. She might describe something as patibulary, fremescent, pharisaic, Roxburghe, or glockamoid, and words like mormal, jeropigia, endosmic, mage, palmerin, thos, vituline, Turonian, galingale, comprodor, nox, gaskin, secotine, ogdoad, and pintulary fled from her lips in Pierian saltarellos. Their dictionary looked like a sow’s ear, because Virginia spent inordinate proportions of her days racing through it, though when Mrs. Gamely was angry a staff of ten could not have kept pace with her, and half a dozen linguaphologists would have collapsed from hypercardia.

— Winter’s Tale (New York: Harvest, 2005), 225-226

For reference (thanks, Oxford English Dictionary!):

Leviner

? (presumably not from “levin”, to emit flashes of lightning)

march-land

a border territory

diclesium

botanical term for a kind of dry, seed-retaining fruit

liripoop

part of a graduate’s hood in early academic costume; later (presumably by derivation) “to have [one’s] liripoop” was to have learned a lesson or part

rapparee

a 17th-century Irish pikeman; later an Irish bandit

dagswain

a kind of rough bed-cover

bronstrops (singular)

a female procuress [of sexual services]

caroteel

an old commercial measure of quantity (”a caroteel of cloves”)

opuntia

originally a Greek herb; later an American cactus

sough

a whisper or murmur or breath; or, a drain or swampy place; or, a ploughshare

patibulary

of or relating to a gallows (patibulum: fork-shaped gibbet)

fremescent

growing noisy

pharisaic

of the Pharisees, hence legalistic, self-righteous, devoted to the letter and not the spirit

Roxburghe

a style of bookbinding

glockamoid

shaped like an arrow-head (note: not in OED)

mormal

a kind of scab or sore

jeropigia

from Portuguese “geropiga”, a mixture of grape juice, brandy and sugar used to adulterate wines

endosmic

relating to endosmosis, the flow of a fluid from an area of lesser concentration to one of greater

mage

a magician, or more generally a wise man

Palmerin

16th Century Spanish romantic hero, hence any knightly champion

thos

old Greek and Latin name for some kind of canine animal not definitively identified by subsequent historians

vituline

of or like a calf (vitulus: calf)

Turonian

part of the Cretaceous period

galingale

a gingery spice; better known as galangal

comprodor

possibly a misspelling of “comprodor”, a native steward or head servant, intermediary with the locals

Nox

personification of the night, from nox, night

gaskin

a kind of breeches; or, formation from “gasket”

secotine

possibly a misspelling of “Seccotine”, a brand of glue originating in the 19th century

ogdoad

the number eight, a set of eight; or specifically the Ogdoad, eight divine beings of ancient Egypt

pintulary

?

Pierian

relating to Pieria, home of the Muses; hence, poetic

saltarello

an animated Italian and Spanish dance

Trawling the book for the rest of Helprin’s vocabulary I leave for someone else, but special mention is due to “amphibological” — of amphibology/amphiboly, ambiguity of speech, especially deriving from grammatical construction — for appearing in context in the title Amphibological Whimsey Dances. It’s a better name for wordplay than wordplay.

Microsoft hardware is fabulous, but — like everyone else ever to buy one — I was disgruntled to find that the Natural Ergonomic Keyboard 4000’s centre-keyboard lever is set to control zoom, and only zoom, rather than something like vertical scroll. Worse, IntelliType won’t let you change it, unlike almost every other special key on the board.

Luckily, it turns out that it’s a limitation of the interface, not the software; you can get a lot more done with a little registry tweaking and two XML configuration files in the installation directory:

• C:\Program Files\Microsoft IntelliType Pro\commands.xml
• C:\Program Files\Microsoft IntelliType Pro\mscmdkey.xml

There are two different kinds of customisation possible:

1. Use the registry to map keys to commands specified in mscmdkey.xml
2. Change the behaviour of those commands in specific contexts with commands.xml

First, we want to look at mscmdkey.xml.

mscmdkey.xml

mscmdkey.xml lists commands sent out by the keyboard to IntelliType. It should probably be treated as read-only, but it’s informative; commands.xml can’t easily be edited without it. The commands look like this:

<Command name= 'VOLUME_UP_COMMAND' id='700' isUI='false' default='true' >
    <ResourceIDs displayName='3809' description='4009' descriptionPlusAccel='0' osdText='4270'/>
</Command>

The magic numbers are inscrutable (a displayName of 3809, an osdText of 4270?), but the name and id are simple enough.

The name is a human-readable label. VOLUME_UP_COMMAND is the default action of the volume-up button. SAVE_COMMAND corresponds to the “Save” button above F11. Some of the commands don’t have buttons on all keyboards: the Natural 4000 has no “next track” button (MEDIA_NEXT_TRACK_COMMAND), but the Natural MultiMedia does. Some of the commands never have buttons: BATTERY_LOW_COMMAND is an automatic feature of the battery-powered wireless models.

We’ll need the id when we get to commands.xml. To change the Zoom lever we want:

• <Command name= 'ZOOM_IN_COMMAND' id='319' default='true' >
• <Command name= 'ZOOM_OUT_COMMAND' id='320' default='true' >

319 and 320, respectively.

Some commands have an MSReserved sub-element. The appCommand attribute corresponds to a WM_APPCOMMAND parameter (e.g., for NEW_COMMAND it’s 29, the defined value of APPCOMMAND_NEW).

I’ve had no success trying to add my own commands under high, unused ids.

Remapping keys

The IntelliType software allows you to remap some, but not all, of the keyboard’s keys. When you do, the changes are saved to the registry at HKEY_CURRENT_USER\Software\Microsoft\IntelliType Pro\EventMapping.

The easiest way to find the key-code of a key you want to remap is to use IntelliType to assign it, then modify the values. The entry for my second favourites button looks like this:

[HKEY_CURRENT_USER\Software\Microsoft\Intellitype Pro\EventMapping\79]
"ShellExecute"="cmd.exe"
"Friendly"="Python"
"Arguments"="/K python"
"Command"=dword:00000320

The important one is Command. dword:00000320 is 0×320 hex, or 800 in decimal. Search mscmdkey.xml for a command with an id of 800 and you’ll find SHELL_EXECUTE_COMMAND — that sounds a lot like what we’re doing. You can use any of the commands from mscmdkey.xml here, though not all of them work.

Friendly is just a human-readable name (this entry displays in the favourites menu as “start Python”). ShellExecute and Arguments are specific to SHELL_EXECUTE_COMMAND.

IntelliType can only set the favorites buttons to applications and URLs, but I miss the next and previous track buttons from my Natural MultiMedia. I’ve put them as faves 4 and 5:

[HKEY_CURRENT_USER\Software\Microsoft\Intellitype Pro\EventMapping\81]
"Command"=dword:000002c0

[HKEY_CURRENT_USER\Software\Microsoft\Intellitype Pro\EventMapping\82]
"Command"=dword:000002bf

0x2c0 is 704, MEDIA_PREVIOUS_TRACK_COMMAND; 0x2bf is 703, MEDIA_NEXT_TRACK_COMMAND.

Some keys can’t be remapped. The “My Favorites” key, for example, can be disabled with DISABLE_COMMAND (400) but operates as normal with any other value.

Editing commands.xml

commands.xml lets us redefine what those commands actually do in any given context. It makes sense: not all software is alike, so triggering (e.g.) a spell check will require something different in OpenOffice.Org Writer than in Microsoft Word. Many of the mappings are simpler than you might think: SAVE_COMMAND (311), for example, is just a macro that performs the “Ctrl + s” keyboard shortcut!

commands.xml looks something like this, trimmed for brevity:

<DPGCmd>
    <ENG>
        <Application UniqueName="StandardSupport">
            <C311 Type="5" KeySeq="ctrl s" />
            <C401 Type="5" KeySeq="F7" />
        </Application>
    </ENG>
    <ALL>
        <Application UniqueName="Notepad" AppName="Notepad">
            <C311 Type="1" wParam="0x10001" />
            <C401 Type="0" />
            <C309 Type="5" KeySeq="alt F4" />
        </Application>
    </ALL>
</DPGCmd>

We’re not interested in a lot of this. Each installed language has an element under the root DPGCmd node defining the function of a command in that locale. ENG means English. The special element ALL applies to all languages; unless you’re a hardcore polyglot then it’s all you’ll need.

Commands are mapped on a per-application basis, but AppName doesn’t seem to actually do anything: change “Notepad” to “Potato” and the keys will work the same.

UniqueName is the important one, and refers to the window class name passed to the relevant Windows API functions. If you want to do application-specific customisation, there’s third-party software around (e.g. The Customiser) that can get a class name from any active window. If not, the special UniqueName value “StandardSupport” applies to all window classes.

Be careful not to have conflicting rules. Commands defined under specific UniqueNames will override commands defined under StandardSupport, but they can’t be defined under specific languages if they’re also in ALL.

Commands

Every Application contains elements with names like C319, where 319 is the id of a command in mscmdkey.xml. All of these command-elements have a type attribute:

• Types 1-4 take different kinds of undocumented magic numbers. Type 2 commands seem to be handled by Windows, rather than the active application, as they execute shell functions (open default browser, search window etc.), but that’s all I can figure out.
• Type 7 commands take another type as a subtype. By default it’s only used for OFFICE_TASK_PANE_COMMAND, so I’m assuming it’s a hack just for that.

The others are easier:

• Type 0 disables the key.

• Type 6 takes an Activator which is passed to the window. Some of them are evidently application-specific — IllustratorZoomin, IE7Save — but others seem more general. An incomplete listing:

  • ZoomIn
  • ZoomOut
  • ScrollUp
  • ScrollDown

Type 5 is the fun one, which takes a simple keyboard macro in its KeySeq attribute. As mentioned above, “Save” is implemented like this:

<C311 Type="5" KeySeq="ctrl s" />

i.e., it’s exactly the same as pressing “Ctrl” and “s”. Multiple chords can be separated with a | pipe character, so if you want a “Hello world!” button:

<C203 Type="5" KeySeq="shift h | e | l | l | o | 
    space | w | o | r | l | d | shift 1" />

Macros seem to be deliberately limited, possibly as a security feature. You can’t alt tab out, for example, or navigate menus with alt | f | downarrow | downarrow | enter. You can implement macros that reach into “Save As” or “Open” file-pickers, though, and presumably other kinds of dialog.

See Also

I wouldn’t even have considered looking for baroque XML configuration files if the internet hadn’t said it was possible. Joel Bennett’s guide has the most detail by far, plus a handy bit of XSL to remove the F-lock annoyance.