The long hunt

A long-running cipher search over the systems Elgar could plausibly have used in July 1897, with the statistics needed to tell a real result from the best of a very large number of lottery tickets. One Python file, no dependencies, built to be started and left alone for weeks: it checkpoints, resumes exactly, bounds its own memory, and writes a status file every 30 seconds for a live dashboard.

Which ciphers, and why those

The constraint that does the most work is the date and the medium: a short personal note written by hand in 1897 by a musician, not a cryptographer, for a friend who was expected to read it. That rules out anything needing tables, machines or bookkeeping.

Always present: substitution. Every candidate is solved with a simulated annealer over the 24-symbols → letters map. A substitution solver absorbs any fixed relabelling — a different grid fill, a once-off rotation of the whole alphabet, “the orientation is the letter and the arc count is a modifier” are all the same cipher to it. That is what makes the space finite instead of astronomical.

How a long brute-force is done properly

Never trust the maximum

The best of a million candidates is a large number even when nothing is there. Three corrections are applied, and every one was added because its absence produced a false positive during development.

Reading the verdict — a claim needs both conditions

The instrument has to suit the message

lab/bakeoff.py asks the question that decides whether the search can work at all: at this search size, could the model detect the message if it were there? Every candidate raises the score the winner must reach; at 108 M candidates × 2 languages the best-of-N bar is 5.8 noise-SD. Detection power, giving each model the same plaintext and comparing it against its own noise floor:

Generic English misses the bar for both registers Dorabella could plausibly be. The search was structurally unable to detect the message even if it were sitting in the space — not because the space was too small, but because the instrument was wrong for the register. Widening the search would have made this worse. So the default target became let: a 3 M-letter corpus of period correspondence (Carroll's letters of 1898, Dickens, George Eliot, Darwin, Austen, Keats, plus model letters, a letter-writing manual and merchant correspondence), filtered to the sentences that read like letters, with controls held out.

The search budget beyond which nothing can be believed

lab/odds.py: for each language there is a budget N* beyond which the best coincidence outscores a genuine solution.

At the project's full search size — 108 M candidates × 3 languages = 326 M — the best-of-N null reaches −4.566 for English against a correct answer's −3.994. The truth still wins by 0.57, so the search is worth running and a positive English or German result would mean something. For compressed English it does not: past about 414 candidates the coincidence wins, and it charges the other languages a Šidák penalty for the privilege. That is why it is not a default target.

Watching for words

The hunt can be started with a list of words — ./run.sh words ELGAR,DORA,MUSIC,DEAR — and lists every plaintext containing one, with the position and the transform. Beside the count is the number chance alone supplies, and that is the point. A live example from testing:

hits 2   expected 4.23   ratio 0.47x   over 96 candidates

Two sightings of AND — fewer than chance gives. The count only starts to mean something when it runs well above the expected figure, and even then the candidate still has to pass the verdict. The expected figure assumes uniform letters, so real solver output (language-shaped) produces common words rather more often: treat it as a floor, not a forecast. A positional template is deliberately not offered here — each transform reorders the stream, so “position 34 is D” means a different glyph for every candidate. A word transfers; a position does not.

How long it runs

Measured throughput is 0.34 candidates/s per core (8 restarts, three languages, CPython; PyPy is 5–10× faster with no changes). That length is deliberate: the space is larger than any single run, the hunt never exits, and it takes one candidate from each family in turn — so a run that covers a fraction still covers every hypothesis. To finish a space instead of sampling one, drop the giant family: the remaining 2,770,195 candidates take about twelve days on eight cores. Widening the transposition catalogue (columnar width 7) multiplies the composite family by six.

Running for months

Memory is bounded by design. The only structure that grows with the run is the dedupe cache, capped at 3 M stream hashes (roughly 200–250 MB); the leaderboard, history, log and word hits are all capped too, and the systemd unit sets MemoryMax=4G. Disk: the status file is rewritten every 30 seconds — about 86,000 atomic writes of ~20 KB a month — and the checkpoint every 5 minutes.

Restarts cost almost nothing. Verified against a hard kill -9 mid-run, which is what an OOM kill or a power cut looks like:

restarts recorded   1
tested              1400   (checkpoint held 512)
total work          265s   vs 161s this run     -> prior work carried
baselines reused    34                          -> not rebuilt
leaders survived    25
ledger survived     16 dimensions, 63 matrix cells

• A backup checkpoint. Every save moves the previous one to .bak; a full disk or bad sector can no longer cost the whole run.
• A space fingerprint. Candidates are addressed by index, so resizing a family shifts every address after it. The checkpoint carries a hash of the families, the ciphertext and the languages, and refuses to resume against a different one — otherwise it would search different candidates while reporting the old coverage, wrong in a way that looks right for months.
• Restart=always. The hunt is designed never to finish, so any exit that is not a deliberate stop is something going wrong.
• States have names: starting, running, deepening, verdict, refine, stopping — a pause in throughput always has a visible reason.

The status file is written mode 0644 on purpose: mkstemp() creates 0600 and os.replace() keeps it, so a hunt running as root would leave the file readable by root alone — and the dashboard, running as www-data, would show nothing while the hunt worked perfectly.

The dashboard

A single page, served by a tiny stdlib server that validates the JSON before handing it over and retries once if it catches a write in progress — so the page can never render half a file. It binds to localhost and is reached through an SSH tunnel, or through nginx with TLS and basic auth; bound openly it would show the ciphertext, candidate plaintexts and recovered keys to anyone. Five tabs, in the order a person needs them:

The connection dot is the honest one: live when the file is fresh, amber after 75 s (probably a deepening or verdict cycle, which the state field names), red after 150 s (probably stopped), and server unreachable when the server itself is down. The best-score chart only ever rises, because it is a maximum over everything tried — a rising line is the search working, not evidence.

What to expect

Almost certainly: nothing. Dorabella has resisted since 1897, and the work on this site has already ruled out a regular rotating key, found no repeating key of period 2–22, and established that no substitution under any transform tested reads as English at matched effort.

The hunt is built so that if it does find something it will be possible to tell — and, more importantly, so that it will not fool anyone when it does not. The verdict is designed to say nothing loudly rather than solution softly. If it ever reports solution, the status file holds the family, the exact transform, the key and the plaintext: reproduce it in the browser decipher tool first, then check the plaintext for anachronisms — “DORABELLA” itself is one, since Elgar did not use the name before September 1898.