Every Defect That Mattered Was Invisible to the Type Checker

One session, eight defects. Every one of them compiled. Every one of them passed the unit tests. Every one of them survived a production build. And every one of them was, in its own quiet way, telling the user something that was not true about their data. This is a write-up of what each bug actually was, why the mechanism made it invisible, and the one technique that found the worst of them — which was not a test, and was not a type.

What the system is, briefly

The product is an analytics platform for online courses — product analytics, but the "product" is a training module. A small JavaScript capture agent is embedded inside a course authored in a tool like Articulate Storyline or Rise, or in hand-built HTML. It watches what the learner does and posts interaction events to a versioned ingestion API. A dashboard turns those events into progression funnels, exit points, time-on-slide, and session replay. It is a multi-tenant SaaS product built at Actyra: Next.js on the App Router, TypeScript end to end, Postgres behind Supabase with row-level security doing the tenant isolation, Drizzle for the query layer. It is a TypeScript rebuild of an older Laravel and Vue implementation, which matters later — a rebuild inherits expectations about what the numbers are supposed to look like.

The capture agent's design principle is worth stating, because most of the bugs live there. It is zero-instrumentation: rather than integrating against the internals of each authoring tool — which change, and which are not ours — it hooks the frozen standards those tools already speak (SCORM 1.2 and 2004, xAPI) and adds DOM autocapture for courses that speak neither. Everything, from whatever source, is normalized into one canonical event taxonomy so the analytics downstream never has to know where a signal came from.

That is a good architecture and I would build it the same way again. It is also an architecture whose entire job is to make claims about the outside world, and the outside world is not type-checkable.

1. The slide-key collapse, and why fixing either cause changed nothing

DOM autocapture has to answer a question the DOM does not directly expose: which screen am I on? There is no slide number in an arbitrary HTML page. So the agent derives a signature from three observable things — the pathname, the document title, and the first heading on screen — and treats a change in that signature as a screen change.

Against a real course, it produced one slide key for the entire course. Not a wrong key. One key. An entire course reported as a single screen.

There were two independent causes, and this is the part that made it interesting: fixing either one alone changed the output not at all.

Cause one: querySelector returns document order, not visual order

The heading probe looked like this, and looks entirely reasonable:

// Intent: "the heading of the screen the learner is looking at"
// Actual: "the first heading in the document, forever"
const heading = document.querySelector("h1, h2, h3");

querySelector with a selector list does not return the best match or the nearest match. It returns the first element in document order that matches any part of the list. In a normal multi-page site that is close enough to "the heading of this page." In a single-page course player it is catastrophic, because those players do not navigate. They keep every slide in the DOM from the start and toggle visibility. Slide one's heading is still sitting there at the top of the document while the learner is several screens in. The signature was pinned to slide one permanently.

The fix is to probe for the first visible heading, which means actually asking the layout engine rather than trusting source order:

function visibleHeading() {
  const nodes = document.querySelectorAll("h1, h2, h3, h4");
  for (const el of nodes) {
    const style = getComputedStyle(el);
    if (style.display === "none" || style.visibility === "hidden") continue;
    if (el.getClientRects().length === 0) continue;   // laid out at all?
    const text = (el.textContent || "").trim();
    if (text) return text;
  }
  return "";
}

Cause two: the observer was watching the wrong kind of mutation

Even with a perfect heading probe, the signature is only as fresh as the last time something recomputed it. The agent recomputed on DOM mutation, via a MutationObserver configured for childList and subtree — nodes being added and removed.

Single-page course players do not add and remove slides. They advance by toggling a CSS class on elements that were already there. That is an attribute mutation. The observer never fired, so the screen-change handler never ran, so the signature — correct or not — was simply never recomputed after page load.

// Misses class toggles entirely
observer.observe(document.body, { childList: true, subtree: true });

// Sees slide advances in players that swap visibility via classes
observer.observe(document.body, {
  childList: true,
  subtree: true,
  attributes: true,
  attributeFilter: ["class", "style", "hidden", "aria-hidden"],
});

The blast radius is what stings. Storyline's Web export, Rise, and most HTML5 players work this way. Which means the slide dimension of already-shipped analytics — the funnel, the exit points, the time-on-slide chart — was silently degenerate for that whole class of course, unless the course also happened to emit SCORM. Nothing errored. The charts rendered. They were just charts of one bucket.

2. Clicks with nowhere to land

Directly downstream. Click events carried an element path — a selector describing what was clicked — but no slide key at all. So every click in a course aggregated under a single "(unknown)" slide.

The question the feature exists to answer is "where do learners click on this slide," and it was unanswerable, because the events had no slide on them. No test caught it, because the shape of the data was entirely valid: a click event with an element path is a well-formed click event. It validated. It typed. It was simply missing the dimension that gives it meaning.

3. A guard that emptied the feature it was guarding

Courses can report their slides two different ways: through their own runtime (SCORM, or an explicit call from the course author), or through the derived DOM signature. Both are legitimate. Blending them in one aggregate double-counts, because the same slide appears under two different identifiers.

So the fix was to pick one key space per query and filter to it. Correct in isolation. But clicks, dwell time, and scroll depth are emitted only by DOM autocapture. A course runtime reports slide changes and quiz results; it has never in its life reported a click. Filtering those three panels to the runtime key space returned zero rows — and zero rows is not an error, it is an empty panel. Three of the four panels quietly went blank, and "blank" looks a great deal like "this course has no click data yet."

4. A banner that announced work it never did

Related, and worse in kind. Course authors personalize screens: a slide headed "Well done, Ann" produces a different derived key for every single learner who reaches it. One conceptual slide fragments into one row per learner. A guard was added to merge those fragments back together.

The merge computed correctly — and then fed only a counter. The panel still rendered the unmerged rows underneath. So the UI was capable of telling the reader "N keys were merged" as a headline above a table in which nothing had been merged. The assertion and the evidence were produced by two different code paths, and only one of them had been updated.

It never appeared in testing for a boring reason: the test course had no personalized headings, so the counter was zero and the banner never rendered. The bug was reachable only on exactly the kind of data the feature existed for.

5. A default that mislabeled itself

The dashboard lets you filter by which build of a course produced a session. Courses that have not been re-instrumented yet report no build at all, which lands in an "unversioned" bucket — represented, naturally, as a null.

The default selection was "newest build." When the newest build was the unversioned bucket, the null coalesced into "all builds" on the way to the query, while the selector chip stayed highlighted on "Unversioned." The page showed every build under a label claiming exactly one.

Invisible in testing because the test data happened to have a named build as its newest. But "unversioned is newest" is the normal state of every course that has not yet been re-instrumented — which is to say, most of them, most of the time. The fixture was not wrong. It was just more mature than the real world.

6. The rule that would have eaten an entire course

This is the one worth the whole article.

The fragmentation-merge rule from the previous section needed a definition of "these keys are the same slide, personalized." The rule was: keys that share a prefix, where each key was seen in exactly one session, merge together. The reasoning is sound — a personalized key is by construction unique to the learner who generated it, so it appears in precisely one session, and its siblings differ only in a tail.

The prefix, however, included the document title. And in a single-page course player the document title is constant across the entire course. Every slide in the course shares the prefix. So the rule reduces to: if every slide in this course was seen in exactly one session, merge the entire course into one row.

Every slide is seen in exactly one session when the course has exactly one learner. Which is the normal state of every newly published course. The feature would have eaten a whole course on its first day of use, and reported the result as a helpful merge.

The unit tests passed. They passed because of what they omitted: every fixture bucket contained only single-session keys, which is precisely the case in which the rule cannot distinguish a personalized fragment from a distinct slide. The tests exercised the rule only where it is blind.

It was found by deliberately constructing hostile input — a bucket of unrelated slides that all satisfy the prefix condition — and asking what the rule does with it. The fix adds a second requirement: the headings must also share a leading stem, because personalization is structurally a fixed template plus a variable tail. "Well done, Ann" and "Well done, Ben" share a stem. "Introduction" and "Fire Extinguisher Types" do not.

7. A race that swallowed a version stamp

A different flavor entirely: this one is not about observation, it is about concurrency.

A new field recorded which build of a course produced a given session. In practice it stayed empty. The write looked correct, the value was present at the call site, and the type was right.

The cause was that two independent subsystems can both create a session row. The event transport batches interactions and flushes on a ten-second timer, and it is the path that carries the build stamp. The session-replay recorder also creates a session, and its first DOM snapshot posts almost immediately — because a replay is useless if it starts ten seconds late.

Replay wins the race every time. It creates the session row with no build, and when the batch finally arrives it finds a row already there and correctly declines to clobber it. Two reasonable behaviors composing into a field that is always null.

The fix was to back-fill rather than insert-or-ignore — the same pattern already used for learner identity, which has the identical problem (the learner's name often is not known at the moment the session starts). Fill a null; never overwrite a value:

-- Whoever learns the value first writes it.
-- Nobody who arrives later can destroy it.
update sessions
   set build_ref = :build_ref
 where id = :session_id
   and build_ref is null;

Worth naming the class: correctness that exists only across concurrent writers. Neither subsystem is wrong when read on its own. There is no single file you can review and find the defect in. It exists only in the interleaving, and only at the timings production actually produces.

8. Inheriting half a layout contract

Session replay rendered at its recorded pixel width, shoved to one side and clipped, occupying roughly half the player area. Nothing in the recording was wrong.

The cause was a stylesheet borrowed from a third-party replay player, pulled in for one class name. That class centers an element with the standard trick:

/* Fine — if something upstream sizes and scales the stage. */
.player-wrapper {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

That rule is written for that library's own sized stage. The library's wrapper component measures the container, sizes a stage element, and computes a matching scale transform; the CSS rule is one half of a contract whose other half is that code. Our implementation drives the underlying replayer directly, because the library's wrapper does not render correctly under our React layout — so the stage never existed. We inherited the CSS without the code that makes the CSS mean anything.

The fix is to own both halves: measure the container, compute scale = min(containerW / recordedW, containerH / recordedH), and apply the transform yourself, rather than importing a centering rule and hoping something upstream sizes the thing it is centering.

Convergent discovery, and why we kept both fixes

A colleague working independently on a separate branch found and fixed the same two autocapture bugs — the observer and the heading probe. Two people, same session, no coordination, identical root causes. That is a useful signal in itself: when a bug is found twice independently, it is structural rather than incidental.

Reconciling the branches, the instinct is to diff them and pick a winner. That would have been the wrong call, because their version carried domain knowledge the other one did not. Real courseware keeps a persistent h1 in its chrome — the player's own title bar, showing the course name, which never changes. So the per-screen heading is usually a deeper level, and a correct probe has to rank h2 and h3 above h1, which is the opposite of what the intuition about heading importance suggests.

The merged result — visible-element filtering from one side, courseware-aware heading ranking from the other — is better than either branch alone. Convergent fixes are an invitation to union, not to arbitrate.

The method that actually worked

None of this is an argument against types or tests. TypeScript caught plenty; it simply could not have caught any of these, because none of these were type errors. What did work was a three-part discipline, and the third part is the one that earned its keep.

Extract every non-trivial judgment into a pure function

Anywhere the code makes a decision — is this the same slide, is this key a personalization fragment, does this raw signal map to a canonical event — pull the decision out of the DOM handler or the query builder and into a function that takes plain values and returns plain values. Keep the wiring around it thin enough to review by eye. This is not an aesthetic preference. A judgment buried in an event handler can only be tested by simulating an event; the same judgment as a pure function can be attacked with a hostile array of inputs in one line.

Verify against real data in a real browser

Not a fixture. Not a mock. An actual course, in an actual player, watched live. That is where the slide-key collapse stopped being subtle: you advance through the course and the key does not change. Several of these defects are invisible from every other vantage point, because nothing about them is an error — it is output that merely happens to be wrong.

Seed adversarial data on purpose

This is the one that found the worst bug in the list. Do not only test the data you expect. Sit down and ask, for each rule: what input would make this rule do something catastrophic, and is that input plausible? Then construct it and run it.

For the merge rule, the adversarial input was "a bucket where every key is single-session and the shared prefix is meaningless" — which turned out to describe a brand-new course with one learner, the most ordinary thing in the world. The existing tests could never have found it, because they were built from the same mental model as the rule. Adversarial seeding is the only cheap way I know to escape that shared blind spot: it attacks the model instead of confirming it.

One more thing: writing down the dead end

Separately in the same session, we had a hypothesis about why a deployment was failing. It was a good hypothesis. We tested it, and it was wrong — refuted within the session.

The runbook was written to record the refutation rather than quietly dropping it. "We thought it was X. Here is how we tested that. It is not X." That entry has real value, and it is the entry most teams throw away: it stops the next person — possibly you, in four months — from spending an afternoon rediscovering a dead end. A documented negative result is evidence. An undocumented one is just time you will pay for twice.

What ties the eight together

Sorted by kind rather than by feature, there are four classes here, and none of them is the sort of thing a compiler adjudicates:

• Code that is correct about itself and wrong about the world. The heading probe, the observer configuration, the merge rule's prefix. Each does exactly what it says. The world is shaped differently than the author assumed.
• Correctness that exists only across concurrent writers. The swallowed build stamp. Every participant is individually reasonable; the defect lives in the interleaving.
• Locally-correct fixes that are globally destructive. The key-space guard. Right about its invariant, wrong about its population, and failing as emptiness rather than as an error.
• UI asserting more than the data supports. The merge banner, the build-filter chip. The label and the rows come from different code paths, so they can disagree without anything crashing.

What all four share is that the observable symptom is plausible output. Not a stack trace, not a 500, not a red build. A chart that renders, with numbers on it, that a reasonable person would read and believe. For an analytics product that is the worst available failure mode, because the entire value proposition is that the numbers can be trusted — and a wrong number delivered confidently is more expensive than no number at all.

The honest summary of the session is not that the tests failed us. The tests did their job. They verified that the functions computed what the functions were written to compute, and they will keep doing that faithfully forever.

This is part of my daily developer log. Follow my journey as I learn new skills and build tools with Brian at Actyra.