Code Coverage Cannot See a Cookie Banner. Screenshot Coverage Can.

We changed the rule for frontend tests: a component counts as tested only when it shows up in a screenshot. Line coverage no longer gets a vote.

Ask a coding agent for 100% line coverage and it delivers. Every component runs under a test, the report is green, the agent declares victory.

Then someone breaks the cart button. Coverage is still 100%. The visual diff is still green. The button was under a cookie banner in every screenshot the tests ever took, so nobody sees it until a customer does.

Line coverage measures whether code ran. It cannot tell you whether anyone looked.

The new goal

Roman Kuksin shipped shaka-perf 0.3.0 with a different goal for the agent:

/goal /shaka-perf-add-coverage until 100% components screenshoted

/goal keeps a Claude Code or Codex agent working until the condition holds. The condition is measured, not guessed: for every component, how much of it landed in a screenshot, and the reason when it did not.

return (                                          // A+B   |
  <nav className="site-nav" data-cy="main-nav">   //       | A=100%,B=0%
    <div className="slide">                       //       | A=33%:clipped-by-ancestor,B=0%
    <button onClick={openCart}>Cart</button>      // 0     |

Five reasons cover the misses: not rendered, hidden by CSS, clipped by a carousel, outside the capture, or covered by a banner, modal or sticky bar. The agent first fixes the test that already touches the component (dismiss the banner, use a taller viewport), writes a new test only when that is not enough, runs the suite three times to shake out flakiness, and stops when every component is in the picture.

What it costs

Screenshot coverage needs an instrumented development build: babel-plugin-istanbul or the swc plugin, plus a React dev build with source positions. Every run executes each test twice, once per side, so this is for happy paths, not every edge case. Line coverage keeps its job. It just is not this job.

Setup

shaka-perf init installs five Claude Code skills. /shaka-perf-dockerize builds the twin containers, /shaka-perf-add-coverage writes the first tests, shaka-perf compare runs them on phone, tablet and desktop against both sides. One to four hours. Start at the usage section of the README.

The same two servers find bugs

ShakaPerf runs your app twice: control on the merge base, experiment on your branch, same Docker image, same seed data. shaka-perf 0.3.1 adds a skill that turns that pair into a QA rig:

/shaka-perf-find-bugs introduced in this branch

It reads the diff, writes down three to five guesses, drives both servers through the same steps, and keeps only what breaks on experiment and works on control. "No confirmed regressions found" is a legitimate answer.

This one is from a client's online ordering app. After the ordering window closed, both sides showed the "unavailable" banner. On main the add-to-cart button was locked. On the branch it still said add. The PR had already been through Cursor's Bugbot, which reads the diff, and this bug does not live in the diff.

ShakaPerf is the toolkit we use in client engagements, where every claim has to survive a production launch.

Aloha, Sasha