eating cucumbers without permission...

And, while with silent lifting mind I’ve trod
The high, untrespassed sanctity of space,
Put out my hand and touched the face of God.

– J.G. Magee, High Flight

It’s a terrible shame that the second half of Event Horizon belonged to one of the worst science fiction films of the 1990s, because the first half promised one of the best.

Dr. Weir (Sam Neill) and his team head into space to investigate an experimental spacecraft, thought lost seven years before when a test of its faster-than-light engine went awry. There’s a whole universe of possible explanations for its reappearance, from aliens to temporal anomalies to quantum somethings-or-others, until Weir speaks those ludicrous, devastating words: the ship has returned from “a dimension of pure evil”. It’s not just a wretched turn of phrase; it’s not just that the anticipated twist is the equivalent of “Christine in space” or “HAL666″. It’s a betrayal.

Science fiction is science fiction, and scientific endeavour is founded on one basic principle: given time and study, given logic, curiosity and empirical investigation, we can figure it out. We can find out what makes it go. We can reverse-engineer the secrets of the universe.

By contrast, Weir gloats:

Did you really think you could destroy this ship? She’s defied space and time. She’s been to a place you couldn’t possibly imagine.

Another character describes the ship’s destination as somewhere “beyond scientific reality”, “hell”. These are not the words of a scientist; they are the words of a priest. Event Horizon invokes the inexplicable supernatural and thereby becomes fantasy. Its moral is the moral of Babel or Icarus: who are we, mere mortals, to touch the sky?

Sunshine, written by Alex Garland (28 Days Later) and directed by Danny Boyle (Trainspotting), is a significantly better film, though its science is almost as dubious and it echoes its predecessor’s painful tumble into creature-feature horror. A team of astronauts are sent aboard the ship Icarus II to reignite our dying sun with a nuclear device, but events take a turn for the Horizon when they discover the Icarus I, thought lost seven years earlier when it failed in the same mission, intact but apparently abandoned.

When the crew have their corresponding encounter with an entity beyond easy understanding, it’s with nothing so abstruse as extradimensional evil: they find the sun. Even for the audience it approaches the spiritual. Like 2001: A Space Odyssey, some of the most memorable moments are long, slow shots of objects in space; and though the discordant post-rock score couldn’t be further from 2001’s classical, the effect is the same. At once beautiful, frightening and inspiring, the images hint at the sublime. In Sunshine, they are images of the sun, so massive, so bright, so fundamentally alien to human experience that an emotional response is inevitable.

For the characters the experience is palpably religious. They dream about it at night; Searle (Cliff Curtis) sits in the observation chamber and bathes in light until he burns. Pinbacker (Mark Strong) makes it explicit: he looks into the sun and finds God there. The Icarus myth had already been registered in the names of the ships, but he invokes it again in his attempt to sabotage the mission. He is the instrument of God’s will: in accordance with the divine plan, humanity must be allowed to die.

Pinbacker is a murderous psychopath; the crew sensibly refuse to believe him. Humanity wins, and God’s plot (or Pinbacker’s interpretation of it) is foiled. That is not to say that the spiritual element of the film is diminished, but for the remaining characters it is a purely secular matter. Capa (Cillian Murphy) at one point comments that their computer simulations are inedaquate, because the high heat and the high gravity are enough to bend time and space. The phenomenon is graphically illustrated in the penultimate sequence, and though obviously unscientific — Boyle has quipped the warning, “Kids! Don’t stick your arm in the sun!” — it’s a figurative expression of the same sense of wonder that most of us find in our interactions with the natural world. A flower is no less perfect as the culmination of millions upon millions of evolutionary generations than it is as the work of a divine craftsman. The sun is no less magnificent as the product of purely natural energies. Even the irreligious can appreciate sublimity.

In Event Horizon’s defense, it’s not necessarily unscientific to claim something as unimaginable: human brains have limits. We evolved that way, coded to survive as hunter-gatherers. We were not coded to imagine the ten or eleven dimensions postulated by theories of quantum mechanics; it’s hard enough to accept that time is a fourth. (How do you visualize four dimensions on a three-dimensional plane?) Less esoteric but no less true, most of us couldn’t easily distinguish between a hot oven at 200°C and one at 250°C. To a primitive brain it hardly matters: both are hot enough to burn. How then can we deal with the surface of the sun, closer to 5000°C, or the core, over 12000000°C? Death occurs too soon for nerve impulses to even reach the brain. “Unimaginable” is an appropriate label.

But imagination isn’t the end-all, and even if it fails we’re still capable of comprehending the difference between “two hundred” and “twelve million”. One of the many glories of science is that “unimaginable” is not the same as “indescribable”.

I am a man with an mplayer. Luckily, it’s less of a hammer and more of an absurd swiss army knife: there’s nothing it can’t do! The documentation is excellent but overly complicated for a beginner, hence this quick guide to some recipes I’ve found useful. I don’t want to have to work them out from scratch again.

Demux the audio from a video file

A few days ago I used this to extract the audio from Neil Gaiman’s Authors@Google talk. It ignores the video and dumps the audio straight to a file.

mplayer <input_file> -dumpaudio -dumpfile <output_file>

The downside is that it’s only useful if the audio is playable on its own (in this case it was already an mp3). If not, mencoder can be used to convert arbitrary video files:

mencoder <input_file> -ovc frameno -oac mp3lame -lameopts cbr:br=128 -of rawaudio -o <output_file.mp3>

The -oac option determines the output audio codec; here it’s the LAME mp3 encoder, set by -lameopts to encode as 128kbps CBR.

Convert audio to mp3

The method I’ve seen recommended elsewhere is to convert the file into raw PCM audio, typically a .wav file.

mplayer <input_file> -novideo -ao pcm:file=<output_file.wav>

(This also works to extract the audio from a video file, incidentally.)

The downside is that .wav files are huge and useless, so we have to have plenty of spare disk space — 10MB per minute of audio! — and use a separate encoder, like LAME, for compression:

lame --preset medium <input.wav> <output.mp3>

mencoder only works on video files, but there is a dodgy method for forcing it to process audio. The trick is that it’s possible to specify an external audio file — the file we want to convert — to use in place of the video’s real audio stream. We just need a dummy video to pacify it:

mencoder -ovc frameno -oac mp3lame -lameopts cbr:br=128 -of rawaudio -o <output_file.mp3> -audiofile <input_file> <dummy_video_file>

It’s a hack, but it does seem to work. If mplayer can play it, you can convert it, and mplayer can handle just about anything: RealMedia, FLV, AAC…

Extract part of a video file

mencoder <input_file> -o <output_file> -oac copy -ovc copy -ss <start> -endpos <time_since_start>

Setting -oac and -ovc to “copy” — that is, to copy from the streams without encoding them — is fastest, but won’t always work. Try -ovc lavc and -oac mp3lame for decent defaults, or consult the “Encoding with MEncoder” guide for the (extraordinarily complicated) details.

The time format is [[hh:]mm:]ss[.ms]. “15″ is fifteen seconds; “10:05″ is ten minutes and five seconds; “09:10:05″ is nine hours, ten minutes and five seconds.

Convert a video to a series of images

mplayer <input_file> -noaudio -vo jpeg

-vo gif89a supposedly generates animated GIFs, but I can’t get it to work. Images can be resized or filtered like any other video stream, so that, for example, it’s possible to generate 100×100 thumbnails:

mplayer <input_file> -noaudio -vo jpeg -vf-add scale=100:100

Rip streams

mplayer can play streaming audio and video, so it can rip streaming audio and video. The input file in any of the above examples can be replaced by a URL.

PyParsing is a Python library for building recursive descent parsers. An example speaks a thousand words:

>>> from pyparsing import Word, OneOrMore
>>> words = Word("abcdefghijklmnopqrstuvwxyz")
>>> sentence = OneOrMore(words) + "."
>>>
>>> print sentence.parseString("i am a simple sentence.")
['i', 'am', 'a', 'simple', 'sentence', '.']

This document is a record of my first attempt at using PyParsing: we’re going to be building a parser for Google-style search queries. It’s not a hard project, but the syntax is deceptively simple. Basic search terms are trivial, but then we have to deal with quoted strings, boolean operators, special term-operators like site: and inurl:, parenthesised groups…

In approximately ascending order of difficulty, these examples will be the basis for a test suite; the parser will be considered complete when it can handle all of them.

>>> tests = [
... 'paris',
... 'love paris',
... 'paris -hate',
... '"Paris, je t\'aime"',
... 'love incity:paris',
... 'love incity:paris with:"pretty french girl"',
... 'love OR "just lust?"',
... '(love OR "just lust?") incity:paris',
... '(love OR "just lust?") (in:paris with:girl)',
... '(((love OR lust) in:paris) with:girl) OR "watch TV at home"',
... '((tv at:home alone) OR (in:paris with:girl)) -(difficult decision)'
... ]
>>>
>>> def test(parser):
...     for test in tests:
...         try:
...             print test, "->", parser.parseString(test)
...         except:
...             print "FAILED"
...             return
...     print "HOORAY!"

It won’t be complete, but it’ll do. The tests aren’t automatic, because for that we’d need to decide up-front on the kind of data structure to parse the queries into, and it’s easier to learn by diving in than by collecting requirements. We’ll just adjust the output as necessary.

1.

A basic query consists of one or more words. With apologies to most of the population of the world, we’ll assume that a word can only contain the letters [a-zA-Z].

>>> from pyparsing import *
>>> term = Word(alphas)
>>> terms = OneOrMore(term) + StringEnd()

>>> test(terms)
paris -> ['paris']
love paris -> ['love', 'paris']
paris -hate -> FAILED

Already a failure! The parser can’t handle the minus sign. The easiest fix is to just add “-” to the list of allowed characters:

>>> term = Word(alphas + "-")

2.

Negative terms have different semantics, so it’s better to represent them in the grammar with a different rule. It’s easier to read, and we can get the parser to trigger different actions for different types of token.

>>> simple = Word(alphas)
>>> negative = "-" + simple
>>> terms = OneOrMore(simple | negative) + StringEnd()

>>> test(terms)
paris -> ['paris']
love paris -> ['love', 'paris']
paris -hate -> ['paris', '-', 'hate']
"Paris, je t'aime" -> FAILED

3.

The quoted string was a disaster waiting to happen, what with all those spaces and commas and things. PyParsing offers a couple of solutions, so we’ll use the built-in dblQuotedString.

>>> single = dblQuotedString | Word(alphas)

The quotes aren’t technically part of the search term, so we’ll use a parser action to strip them. Parser actions are just functions (or classes, or anything else callable like a function), and they can be added inline. These two examples are effectively the same:

quoted = dblQuotedString
quoted.setParseAction(lambda s,l,t: [t[1:-1]])
single = quoted | Word(alphas)

single = dblQuotedString.setParseAction(removeQuotes) | Word(alphas)

4.

Moving on, it’s obvious that we’re going to get an error from the colon in incity:paris, too, so it needs its own rule. Combining with the double-quote fix, we get:

>>> single = dblQuotedString.setParseAction(removeQuotes) | Word(alphas)
>>> special = Word(alphas) + ":" + single
>>> negative = "-" + (special | single)
>>> terms = OneOrMore(special | single | negative)

>>> print terms.parseString("love incity:paris")
['love', 'incity', ':', 'paris']
>>> print terms.parseString('love incity:paris with:"pretty french girl"')
['love', 'incity', ':', 'paris', 'with', ':', 'pretty french girl']

It does work, but not very well. There are three tokens for every special term, when we really only want one. We could do something fancy here, like return a different structure (a class instance?) as the token, and validate the term at the same time:

allowed_operators = ["incity", "in", "with"]
def parse_special(string_, string_pos, tokens):
    operator, colon, term = tokens
    if operator not in allowed_operators:
        raise IllegalQueryError
    return SpecialQueryTerm(term, [operator])

special.setParseAction(parse_special)

But that’s a little too complicated, since we have neither a SpecialQueryTerm class nor an IllegalQueryError exception. PyParsing has some built-in functionality we can use, like Combine, which combines the tokens into one, and Group, which groups them into a list. Let’s try Combine.

>>> special = Combine(Word(alphas) + ":" + single)
>>> negative = "-" + (special | single)
>>> terms = OneOrMore(special | single | negative) + StringEnd()

>>> print terms.parseString('love incity:paris with:"pretty french girl"')
['love', 'incity:paris', 'with:pretty french girl']

Our tests again:

>>> test(terms)
paris -> ['paris']
love paris -> ['love', 'paris']
paris -hate -> ['paris', '-', 'hate']
"Paris, je t'aime" -> ["Paris, je t'aime"]
love incity:paris -> ['love', 'incity:paris']
love incity:paris with:"pretty french girl" -> ['love', 'incity:paris', 'with:pretty french girl']
love OR "just lust?" -> ['love', 'OR', 'just lust?']
(love OR "just lust?") incity:paris -> FAILED

5.

The critical failure was caused by the parentheses, but there’s a subtler problem with the first boolean test. It seems to work, but “OR” is being matched as though it were just another word, as in the hypothetical query “OR OR AND AND AND”. We should fix that.

>>> operator = CaselessLiteral("or") | CaselessLiteral("and")
>>> terms = single + ZeroOrMore(Optional(operator) + single)
>>> query = OneOrMore(terms) + StringEnd()

6.

A single level of parentheses couldn’t be simpler. We could use a QuotedString, with the two brackets the “quotes”, but this works too:

>>> parenthetical = Suppress("(") + Group(terms) + Suppress(")")

Suppress means “match these tokens, but don’t add them to the output”.

Unfortunately, for nested parentheses we need recursion. After the extreme suffering I was put through the last time I wrote a recursive parser, I was expecting that statement to be met with an ominous crash of thunder. As it turns out, PyParsing makes recursion absurdly easy too. We just use an instance of the special class Forward() as a placeholder, then set its value with the << operator.

This is about the simplest possible recursive parser for nested parentheses:

terms = Forward()
parenthesis = "(" + terms + ")"
terms << OneOrMore(Word(alphas) | parenthesis)

Ours is only slightly more complex:

>>> terms = Forward()
>>> parenthetical = Suppress("(") + Group(terms) + Suppress(")")
>>> negative = "-" + (special | single | parenthetical)
>>> term = special | single | parenthetical | negative
>>> parser = terms << term + ZeroOrMore(Optional(operator) + terms)

>>> test(parser)
paris -> ['paris']
love paris -> ['love', 'paris']
paris -hate -> ['paris', '-', 'hate']
"Paris, je t'aime" -> ["Paris, je t'aime"]
love incity:paris -> ['love', 'incity:paris']
love incity:paris with:"pretty french girl" -> ['love', 'incity:paris', 'with:pretty french girl']
love OR "just lust?" -> ['love', 'or', 'just lust?']
(love OR "just lust?") incity:paris -> [['love', 'or', 'just lust?'], 'incity:paris']
(love OR "just lust?") (in:paris with:girl) -> [['love', 'or', 'just lust?'], ['in:paris', 'with:girl']]
(((love OR lust) in:paris) with:girl) OR "watch TV at home" -> [[[['love', 'or', 'lust'], 'in:paris'], 'with:girl'], 'or', 'watch TV at home']
((tv at:home alone) OR (in:paris with:girl)) -(difficult decision) -> [[['tv', 'at:home', 'alone'], 'or', ['in:paris', 'with:girl']], '-', ['difficult', 'decision']]
HOORAY!

7.

Hooray! Now to combine it all:

from pyparsing import *

# simple tokens
simple = Word(alphas)
quoted = dblQuotedString.setParseAction(removeQuotes)
single = simple | quoted
special = Combine(Word(alphas) + ":" + single)

# negative terms
negative = "-" + (special | single | parenthetical)

# boolean operators
operator = CaselessLiteral("or") | CaselessLiteral("and")

# recursive parenthetical groups
terms = Forward()
parenthetical = Suppress("(") + Group(terms) + Suppress(")")

# bring it all together!
term = special | single | parenthetical | negative 
terms << term + ZeroOrMore(Optional(operator) + terms)
query = OneOrMore(terms) + StringEnd()

It’s still not done, but it’ll do. It’s left as an exercise for the reader to add support for syntax like “inurl:(parenthesised terms)“, though the beauty of a parser like this is that it’s a very, very easy to do so.

It needs to support a large fraction of Unicode. The numbers 0-9 would be a good place to start. It should work around bad input, not die horribly. It should have an explicit order of operations: is “a or b and c” the same as “(a or b) and c” or “a or (b and c)”? And, of course, the hard bit is taking the data from the query and actually doing something useful with it.

I have no need whatsoever for such a parser, but it was fun to write this much of it: PyParsing is a great tool. At the very least, it’s useful as an infinitely more readable replacement for complex regular expressions.

8. Other Resources

• PyParsing homepage. Be sure to check out the examples.
• “Building Recursive Descent Parsers with Python“: from PyParsing’s author, the introduction you should have read instead of this one. (Hmm, maybe I should have mentioned that earlier…)

The Bad Beginning, The Reptile Room, The Wide Window, The Miserable Mill, The Austere Academy, The Ersatz Elevator, The Vile Village, The Hostile Hospital, The Carnivorous Carnival, The Slippery Slope, The Grim Grotto, The Penultimate Peril, The End, by Lemony Snicket.

Presentation

The best thing about the books in A Series of Unfortunate Events is their existence as actual physical artifacts. If there were ever a compelling argument against e-paper, it’s these: beautiful, rough-cut hardcovers, tastefully illustrated by Brett Hellquist. They’re just wonderful, especially given the awful covers on so many children’s books; compare, for example, the grotesquely gaudiness of Artemis Fowl. The greatest tragedy of the film adaptation is that the charmless tie-in editions of the books replaced the originals on most store shelves.

It wouldn’t be outrageous to make a similar claim for the audio versions: Tim Curry (Rocky Horror etc.) reads, and he’s perfect.

There are two different editions of the first volume, one read entirely by Mr. Curry and one which he merely narrates, accompanied by sound effects and a full cast for the dialogue. I have only heard the original. Additionally — and unfortunately — he’s replaced on the third, fourth and fifth books by Daniel Handler, under his Lemony Snicket alias. He does a creditable job, but just can’t compare.

Content

The story proper begins at Briny Beach, where Violet, Klaus and Sunny Baudelaire have their holiday interrupted by news that their parents have perished in a terrible fire, which has also destroyed their house and all of their possessions. As if that wasn’t quite enough unpleasantness for a morning, the three newly-orphaned children are adopted by a distant relative, the villainous Count Olaf, whose only interest in them is their parents’ fortune, held in trust until Violet comes of age.

It’s charmingly dark. The Bad Beginning even opens with an admonition that the book not only has no happy ending, but also no happy beginning or middle; the reader would be better-off closing it and looking elsewhere for something more pleasant. It’s true: very little happens to the Baudelaires that isn’t bad, and what few happy times they do have usually consist of a brief respite between one horrible circumstance and the next. Nor is all cartoon-style peril, where the worst that can happen is a bump on the head from a falling anvil. It’s always the nicest ones who have to die…

No matter how dark it gets, the style is delightful. It’s self-conscious, with frequent authorial asides warning the reader to lay the books aside and take up something fun, like knitting, or bemoaning the way that Mr. Snicket himself was once shipwrecked, or poisoned, or trapped in a pit full of ravenous armadilloes. There are lots of nice little touches, like the explanatory parentheticals — here, the word “paranthetical” refers to a distracting mid-sentence interruption — which begin as vocabulary lessons for young readers but become saturated in irony as the series progresses.

The first seven books follow roughly the same formula: Violet, Klaus and Sunny are sent to live with a new guardian, only to be faced with another of Count Olaf’s schemes. It doesn’t take long for the novelty to wear off, so it comes as a relief when the sixth and seventh books gently transition into a new formula, based on the investigation of a mystery first uncovered at The Austere Academy. By the time the orphans visit The Hostile Hospital, they’ve taken control of their own lives and are actively attempting to unravel the Byzantine tangle of secrets they’ve become trapped within.

Events come to a head in the aptly-titled The Penultimate Peril, with virtually every character in the entire series — at least, those that are still alive — in residence at the Hotel Denouement, though prospective readers should bear in mind that the mystery is never really resolved. What makes The End, the final book, so charming and interesting, is that it makes the not-knowing palatable. The entire volume is, in effect, a continuation of the denouement that began at the (also aptly-titled) hotel, and its character is markedly different.

The End is the end of, essentially, a long coming-of-age story, charting the Baudelaires’ intellectual, emotional and moral maturation. They begin as children, thrown from the care of one adult to another, their only hope of escaping trouble to somehow alert an adult — notably Mr. Poe, their bank manager — of Olaf’s latest treachery. By The End, they’ve become fully indepdendent. (Most blatantly, Sunny grows from unintelligible baby to monosyllabic toddler to, finally, articulate gourmet chef.)

The moral character of the novels also changes. The Bad Beginning establishes a dichotomy between villains on the one hand, such as Count Olaf, and “noble people”, like Olaf’s kindly neighbour, on the other, but it is steadily eroded through the sequels. Most of the “noble people” suffer from a weakness of character that prevents them from helping the children: Josphine is a coward, Monty is too trusting, Jerome is too self-interested; others fall to peer pressure, blind faith in the rule of law, or undue deference to authority. The villains — even Olaf — are humanized, while the Baudelaires find themselves committing more than a few morally ambiguous acts.

It’s hard to complain that the story is never fully resolved when, in fact, that seems to be the point. Nothing is simple. Even villains have their good sides; even noble people can be villainous — and the difference between the two is only a matter of perspective. Every answer raises two new questions. Every person has two parents with stories of their own, four grandparents with stories of their own, ad infinitum.

They don’t live happily ever after, of course. If there’s a moral here, it’s that there is no end to the series of unfortunate events, just the books with that name. We call that series “life”. For some people it’s better, for some it’s worse, but what matters is how you live it. Try to do the right thing, find what happiness you can along the way, and ask yourself: “What’s the opposite of what Olaf would do?”