Email me
Updates
← Automating Garage Dreams

Updates

Automating Garage Dreams

Every update on this project, newest first.

28 updates

Updates are bullet-point summaries generated with AI from my commits and build notes, then checked by me. Project articles and learning notes are written by me.

  1. AI summary

    • The monitor reported five green checks (26 records, newest run 23.2 hours old, last run published, in sync with the remote) and all three of those facts were wrong. The remote held 27 records, the newest was eight hours old, and it had skipped. Two defects compounded: the upstream check measured only whether the branch was ahead, so a purely behind checkout scored zero and printed “in sync”; and the record checks read the local working tree, which --fetch does not update. On the publishing host the tree and the remote coincide, so the machine the monitor was designed for is the one machine where the bug is invisible.
    • Nine burn-in drafts reviewed against their sources: three carried article-introduced faults, and all three had passed both reviewers. Each is the same shape: a fact kept and the qualifier that made it true discarded. One asserted a platform lineage the source paragraph exists to deny; one dropped “non verified test” from a top-speed figure; one rendered the end of Japan’s 280 PS advertising cap as a claim about the car’s power, beside a 308 PS figure.
    • That retracts an earlier conclusion. “Both reviewers reliably catch dropped qualifiers” was drawn from two runs in which they did, which is a definition rather than a sample: a run they catch gets revised into the record, a run they miss publishes silently. The observed miss rate is three in nine and that is a floor, not an estimate. It matters most at the next stage, when the draft step disappears and the reviewers become the only check.
    • Two further faults were source defects copied faithfully: a guide calling cast-iron brake discs carbon, and another calling a three-door hatchback a four-door. The pipeline behaved correctly in both cases, which is its own kind of finding.
    • The sync damage recurred at four times the size: eight tracked files deleted with (1) twins, including four published articles. The first method used to verify them reported all eight as differing, which would have meant eight files of real work to reconcile by hand. The method was wrong. Dumping a blob through a PowerShell redirect re-encodes it, so the comparison measures the shell rather than the files. Comparing object IDs the way git itself would showed all eight identical. Note the direction of that error: the broken method produced the alarming answer, and nobody double-checks bad news.
    • Added commands to move generated articles between draft and live in both directions, sharing one transaction that validates everything, holds the originals in memory and rolls back by exact path. It exists because after the draft stage is removed there is otherwise no route back for prose. The flip and its rollback do not differ by direction, so one function takes the source and target values and the round trip to the original bytes is asserted. There is deliberately no bulk form on the withdraw side.
    • All nine drafts are now live, published in one commit by the command rather than by hand.

  2. AI summary

    • An unattended cover run failed its own integrity check and left nothing behind to diagnose it with: no candidate, no session record. Rendering the prompt locally and hashing variants identified it exactly: the attested digest is sha256(prompt + "\n"). That is the signature of hashing an echoed string in a shell. Switching the attestation from text to a digest a fortnight ago closed the Unicode serialisation boundary and not the shell’s trailing newline.
    • Ben’s call was to accept exactly one appended newline and nothing else. A second newline, a CRLF, a leading newline, a trailing space and a single dropped character all still fail, and both directions were mutation-checked.
    • The retry selected a different article, because the failed one was deprioritised by its own recorded failure, and it passed.
    • Second instance of the development-years fault: a cover for “The Development of the Mazda RX-8” asked for 2001-2003 for a car sold from 2003. Both instances are articles whose titles start “The Evolution of” or “The Development of”, which is a pattern rather than a coincidence. The images were correct both times.
    • A 23-minute article run outlived the 15-minute sudo credential, so the job queued behind it sat waiting for a human. Worth knowing before anything depends on two long steps in sequence.

  3. AI summary

    • The first draft run under systemd failed on a revision call that hit the token cap after 859 visible characters, and the cause is that the model thinks. The server streams reasoning on a separate field, the pipeline discards it, and it still spends the cap and the wall clock. A probe measured 400 tokens and 103 seconds of pure reasoning with no content at all; with thinking disabled the same call took 34 tokens and 8.4 seconds. Ben chose to keep thinking on and raise the cap to 4096, so a truncation message now reports the reasoning characters too.
    • Covers became an unattended nightly job: select one missing article, generate, accept, apply, build, commit, push. A push adding a cover record opens a GitHub issue mentioning Ben, delivered by email and confirmed working, and a phone-triggered workflow reverts it. Every side effect is injected in tests, and three of the guards were mutated and watched failing.
    • A failed cover run leaves no committed record, so the batch itself carries the state file, so a run killed mid-flight stays marked started and counts as a failure. Three failures on one article excludes it from selection rather than letting it burn a generation every night.
    • Checking the Pi for a leaked API key with env | grep through sudo reported a match that was not there: sudo sets a variable containing the full command line, so grep matched its own text.
    • The alert path fired correctly and printed “(journal unavailable)”, because the handler unit could not read the journal it was reporting on. The failure path works; the part that tells you what failed did not.
    • The first Pi cover recorded model years of 2008-2012 for a car that went on sale in mid-2012. The derivation takes the dominant cluster of years in the article body, and a development history names the announcement, the prototype and the launch, so the cluster is the development period, not the production run. The image came back correct anyway. Left alone deliberately: changing the derivation mid burn-in would muddy the tally it is being measured by.

  4. AI summary

    • A mobile performance pass found the pipeline’s own articles already score 100, which killed the font theory on the spot: the same 176 KB of woff2 loads on the page scoring 100 and the page scoring 91. Only the document varies.
    • Lighthouse’s largest single recommendation does not pay. It claimed 1,210 ms from eliminating render-blocking CSS; inlining it removed every render-blocking resource and made the score worse, 93 to 92, because the document grew from 128 KB to 172 KB and the document is the bottleneck. Reverted. What did pay was a high fetch priority and a 768w variant on the cover image, and deleting a font weight nothing referenced.
    • Chasing one slow page found 15 cross-origin requests transferring zero bytes: raw WordPress image tags still pointing at the parked domain. Pulling that thread turned up five more classes of migration debris: 1,285 Gutenberg block comments, 97 escaped target="_blank" attributes, and 68 URLs with markdown escaping inside them, which silently 404s every youtu.be link and breaks every affiliate parameter. None of it is schema-invalid, so nothing had ever failed.
    • 132 bare YouTube URLs across 38 guides now render as click-to-load facades with a real link behind them. The URL parser returns null on anything it cannot parse rather than taking the tail of the string, because with escaped ids in the corpus, a guess would embed a different video than the author linked. The transform runs at render time, so the prose the pipeline reads stays a plain URL with no markup to leak into a prompt.
    • A latent bug shipped alongside: the cover-provenance field accepts the whole agent enum, so a human-credited photograph would have rendered “Generated image · Human. Not a photograph of a real car.” over somebody’s cleared press photo.
    • Cover videos throw by name when the path resolves to nothing, because Astro’s asset helper validates images only. A dead <video> renders as its poster and looks completely fine, which is exactly why it would ship unnoticed. Requiring a human author for the field makes the stranded-video state unrepresentable rather than handled.
    • Seven new corpus assertions were each verified by injecting their own defect and watching them fire by name. Sequential assertions mean one combined injection only ever proves the first one.

  5. AI summary

    • Two runs died and the machine looked like the obvious suspect at 1.7x slower than the week’s baseline. It was not. Thermal throttling: killed, the throttle bits never set across 1,470 samples. Memory pressure: killed, peak usage was lower than during a build. Throughput degradation: killed by the server’s own per-call numbers, identical to the previous session’s. What actually varied was how many tokens the model chose to emit, 1,057 to 1,483 per call, and at 3.4 tokens a second each extra hundred costs thirty seconds.
    • The longest call that survived took 477,824 ms against a 480,000 ms ceiling. Two seconds of margin. The next one went over.
    • So the fix bounds the work rather than the clock: the request now carries a token cap, and a response that stops because it hit that cap raises outside the retry path, because truncation is not retryable. The same prompt under the same cap truncates again, and on this hardware that costs another ten minutes.
    • The timeout went to fifteen minutes, deliberately generous rather than fitted. Sizing it to the longest call observed has now been wrong three times. With a token cap as the real control, the timeout is a backstop that should almost never fire, and a tight backstop is the same mistake again.
    • The next batch drew the identical section that had failed, because a failed record is not a skip and nothing deprioritised it. Same source, same prompt, only the ceiling changed, and it published. A controlled test rather than a lucky one. Three of three published; median wall clock across five published runs is 954 seconds.
    • A previously recorded finding turned out to be wrong. “Shorter source sections produce more verbatim overlap” rested on three data points; across all seventeen records the correlation is +0.185, the opposite sign, and the three newest near-floor runs produced the lowest verbatim runs on record. The genuine worst case was a 418-word section that reached 29 against a limit of 30. It predates the finding and was missed because only three runs were compared.
    • The build peak was measured three times, not once: 907.6, 867.2 and 882.8 MB. A single pass would have reported 867 and been 40 MB light.
    • Consecutive runs accumulate heat and swap. Peaks rose 79.6 → 80.7 → 81.8 °C across one batch, with the first active soft-throttle bit ever caught on this machine in a single sample, and swap steps up at every build phase without coming back down. Small, and the only metrics with a trend rather than a steady state.
    • The measurement tool needed a group the service account is not in, so the two columns the whole exercise existed to collect came back as errors. Caught mid-run. Moving the sampler to a separate account produced a better design anyway: one continuous trace across consecutive runs shows accumulation that per-run traces would hide.

  6. AI summary

    • The Pi now generates, reviews, revises, builds, commits, pushes and publicly publishes on its own. Two articles, one hidden draft and one live, neither touched by a human. 822 and 787 seconds, so “twenty minutes or ninety” resolves to about thirteen.
    • Getting there meant two ceilings that both presented as “it timed out”. The first was a hardcoded 120-second request timeout with no override, against measured throughput of 3.4 tokens a second. The second only became visible once the first was lifted: with a non-streaming request the server sends no headers until generation finishes, and Node’s HTTP client abandons the request at 300 seconds regardless of the timeout you set. The error type was the only tell. Switching to streaming makes the abort signal the only deadline. The evidence is three calls pinned at 120.1s, three pinned at ~301s, then a 426-second call that finished.
    • The kernel’s missing Landlock turned out not to matter: the reviewer CLI does not use Landlock on Linux. It ships its own bubblewrap and sandboxes with unprivileged user namespaces. The write failed with “read-only file system” rather than “permission denied”, which identifies the mechanism instead of merely observing that something was blocked, and the probe ran the identical write unsandboxed first as a control. The new fragility is that unprivileged user namespaces must stay enabled. A hardening change would break the sandbox silently.
    • The deploy key’s write access had never once been exercised, because every previous run was a dry run. Added a git push --dry-run to the baseline: reaching “Everything up-to-date” proves the service authorised, costs two seconds, and guards against a commit that cannot push and then blocks every run after it.
    • For a draft-only commit, “the deploy succeeded” and “the deploy never ran” produce identical public output: the page 404s either way and the sitemap is unchanged either way. Nothing in the built site carries a build stamp, so every assertion passes under both readings. A live article self-evidences its own deployment, because the sitemap count moves and a stale build cannot fake that.
    • The local model’s characteristic error is deletion, not invention. Twice in two runs it kept a fact and dropped the restrictive clause that made it true: “hurt ride quality” became “hurt performance capabilities”, and “the first all-electric production car past 320 km” became “the first vehicle”. Both reviewers caught both, independently, and named the same missing qualifier.
    • Originality overlap rises as the source section gets shorter, because less material means more of the source’s phrasing gets reused. The first near-floor section the pipeline has ever drawn passed while spending 77% of the verbatim budget, within seven words of tripping the gate. The thresholds were fitted on the same 60 drafts they judge, and their own comment says to treat the next batch as the real test. This is that batch.
    • The revision loop can trade prose for accuracy. A fix for a factual objection produced “reported that the ride detracted from the car’s ride quality”, which is tautological, and the reviewer’s own pass verdict names the awkwardness before passing it at 8 out of 10. The loop optimises narrowly against the stated objection, and reviewers accept a prose regression once their own point is addressed.
    • Found by accident: the site emits no canonical link and no OpenGraph or Twitter metadata on any page, and never has. Same class as the robots.txt bug: schema-valid, so the build cannot see it, no test covers it, visible only on the deployed site. Every social share of an article currently renders as a bare link, on a site whose entire purpose is publishing articles.

  7. AI summary

    • Turned the scheduled monitor off until there is something for it to monitor. It alerted overnight, correctly, that the newest run was 45 hours old, but the staleness threshold is calibrated to a nightly timer that has not been installed yet, so it would have fired every night from now until then and trained the alert into noise before it could matter.
    • Corrected a claim that both the workflow and the plan had been making: a failed scheduled workflow does not “email the repository owner”. GitHub notifies whoever last changed the cron, subject to their own notification settings. So whoever restores the schedule becomes the recipient, which has to be a person and not a bot, and a manual dispatch is the only cheap way to confirm the mail actually lands before an unattended night depends on it.
    • Fourteen tests fail on Linux and always would have. The test fixture links the real node_modules into a temp directory using a Windows-only junction type; on Windows the OS resolves it invisibly, on Linux it becomes an ordinary symlink, the bundler resolves it back outside the fixture root, and the cache key breaks. The gate written for this stage, “661 tests, 0 fail”, was unachievable on Linux as the code stood, and nobody had ever run the suite on Linux.
    • The Pi kernel has Landlock compiled out, which is what the reviewer CLI’s read-only sandbox relies on. Found at the probe step before the 5 GB model download, which is exactly why the runbook put it there. The flag is not being dropped: it is a containment boundary around a model-driven process.
    • That probe’s own test was defective and got thrown out. It proposed treating a successful CLI response as evidence of sandbox enforcement, but a successful response cannot distinguish enforcement from a silent fallback from the model simply never attempting a write, which is the likeliest outcome since nothing in that prompt asks it to. Same species as the rollback suite that never rolled back.
    • Build peak measured properly at last: 739.6 MB on the development PC, 864.9 MB on the Pi, same 138 pages, 2.4x slower. The first build after a cover backfill measured 835.9 MB in 23.4s against a 720 MB, 9.6s steady state, so the image-derivative cost is real, about 100 MB, and sits on the supervised path rather than the nightly one. Measuring the wrong build would have overstated the nightly requirement by that much.
    • Two probes on the way to that number were wrong in instructive ways. Starting a process with -PassThru returns an object whose exit code is null, which would have let a failed build pass as a measurement; and the peak-memory counter reads zero once the process has exited.

  8. AI summary

    • The most dangerous thing found all day: publish:article --provider=stub publishes. The flag swaps in stub providers and relaxes the CLI preconditions, and that is all it does: the run still builds, commits and pushes. The runbook’s own comment said “no model, no CLIs, no network”, which describes the providers and reads as a description of the run. It must always be paired with --dry-run.
    • Built the independent run monitor, which is the real gate on putting any of this on a timer. It reads only committed run records, writes nothing, and exits non-zero on any alert so a timer can act on the exit code. A run that aborts before writing a record surfaces as staleness rather than as silence.
    • Its ordering comes from the timestamp inside each record, never the filename or the file’s mtime. Filenames are written in local time and the timestamps are UTC, so they disagree across a day boundary, and mtime is rewritten by any checkout and by the sync client on this tree. A test writes records under names that deliberately contradict their own timestamps.
    • A skipped run is not an alert. It is editorial, it is a normal night, and alerting on it would train whoever reads the output to ignore it.
    • The off-host copy of the monitor runs on GitHub’s schedule and was verified by dispatching it, not by writing it, because a workflow fails by never alerting, which no local test can observe.
    • The alert script confidently reported result=success exit=0 for a unit that had just failed. systemctl show answers for a unit it has never heard of with defaults rather than an error, so querying the wrong service manager produces a confident wrong alert, which is worse than no alert.
    • The alert script also shipped non-executable, which would have killed the one component that reports every other failure. The WSL test could not have caught it: it copied the script and made the copy executable, and the mount reports every file as 777 regardless. A test that prepares its subject cannot test how the subject ships.
    • A failed prompt attestation discarded a paid generation and was undiagnosable by construction. The error named neither the attested digest nor the expected one, so investigating it meant changing the code and spending another generation. It now reports both, with the model-controlled value escaped and truncated.

  9. AI summary

    • One of four cover prompts kept failing its own integrity check, and it was the only one containing an en dash. The instruction said “read the file”, which PowerShell is free to satisfy with a default-codepage read that decodes UTF-8 as cp1252, so one character arrived as three and a paid generation was discarded every time. The check was right; the read was wrong. The instruction now prescribes the exact command.
    • With the read fixed, the dash was still coming back as an ASCII hyphen, same length, one code point for one. Tightening the instruction to forbid normalisation did not stop it across three more renders. The fix was to stop echoing text at all: the attestation is now a SHA-256 digest, which crosses a Unicode serialisation boundary unchanged. Same trust limitation as before, minus a channel that had already failed three times.
    • Potential exposure is 331 of 4,285 guide sections with non-ASCII headings. It had never been seen because none of the six articles generated so far happened to contain one.
    • A format rule nearly shipped as dead code. Validating “is this 64 hex characters” inside the comparison killed no test when deleted, because an exact digest comparison already rejects every malformed value. Split out with its own error message, both mutations now fail: removing the comparison breaks 5 tests, removing the format rule breaks 2.
    • The deprioritise-recently-skipped fix ran for real for the first time and worked: three runs, three different sections, where before run 2 would have re-selected run 1’s section and stalled. Codex was the stricter reviewer in all three.
    • First cover under the new recipe derived model years 2002-2010, the whole first generation, which is exactly the fault it was built to fix. Accepted first candidate.
    • Installed a standing rule: comments say what the code does or why a non-obvious choice was made, and nothing is signed (no attribution trailers, no tool names in commit messages or headers), with an explicit carve-out so provider names used as functional identifiers in code are never renamed.

  10. AI summary

    • The test failure filed as a flake last week was a real defect: two processes could hold the run lock at once. Measured at 4 failures in 25 rounds. The six-process test that guards it only catches it about one run in six, which is precisely why a previous session shrugged and moved on.
    • The mechanism, from an instrumented timeline: both contenders renamed the stale lock aside and both renames succeeded, because they moved different files. One moved the stale lock; the other, still acting on a read it took earlier, moved the first one’s live lock. rename is atomic per file, and nothing checked that the file being moved was still the file that had been judged.
    • The first hypothesis was wrong and is worth keeping. Creating a file with wx publishes it before the payload lands, and the lock reader mapped both empty and torn reads to “reclaimable”: 2,104 empty and 11,139 torn reads across 17,599 cycles, so that window is entirely real. It is not this bug. Fixing it would have narrowed the failure and left it in place.
    • The lock is not a file any more. It is an exclusive listening socket on a loopback port derived from the working directory. A lock file outlives the process that wrote it, so it needs a staleness rule, and the staleness rule is where both file-based versions failed. The filesystem offers no compare-and-swap, so ownership is inferred from bytes another process can replace between the read and the act. The kernel owns the port and releases it however the holder dies: no stale state, no reclaim path, no race in the reclaim path. Nothing is served on the socket, so no connection can leave the port unbindable.
    • A proposed fix was rejected before it was written. It kept the file, added identity verification after the rename, and carried an acknowledged residual where a lost restore leaves a live holder inside the critical section without owning the published lock. That is the same safety failure through a narrower interleaving, and “unlikely” is not a correctness argument for a lock guarding a working tree, a build and a commit.
    • Review found the availability test was vacuous: the client destroyed its own connection, so it passed whether or not the holder destroyed anything. It is now proved by mutation: deleting the handler makes it fail with the right message. A sibling test that searched 400 random paths for a collision was replaced with pigeonhole, where 1001 candidates over 1000 ports guarantee a pair, because a probability of e^-80 is not the same as never.
    • The deterministic reproduction is kept in history and deleted from the suite. It drove a temporary seam between the staleness decision and the rename and failed 5 times in 5 runs while the probabilistic test passed beside it. A test of steps that no longer exist is scaffolding, not coverage.