Gopher shipped with a working photo integration. The model was gemma4. Everything worked. The model was swapped to granite4.1:30b. Tests passed. CI passed. Four features broke simultaneously.
What Broke
Videos returned HTTP 400 -- the proxy was passing the wrong size parameter unconditionally, a pre-existing bug the old model never exercised. Photos stopped appearing entirely -- a null-byte placeholder in the markdown renderer was silently stripped by the browser's DOM, a bug that existed since the code was written but only surfaced when the new model stopped reliably following the image formatting instruction. Random searches returned the same results every time -- the shuffle logic was never implemented, masked by the old model's query variation. The model name badge did not show in the header -- stale browser cache. Not everything is profound.
All of these were caught manually. By looking at the screen.
Traditional Practice Has the Answer
This is a solved problem. Open a browser. Sign in. Type "show me some photos." Assert that images appear in the chat. That test fails on every regression listed above. Playwright, Cypress, any UI integration framework -- the tooling is mature and not hard to set up. A handful of these tests running after each change would have made the model swap a non-event.
The answer is not exotic. It is standard practice. The issue is that LLMs do not apply it.
The Gap
Ask an LLM to write a UI integration test and it will produce something that looks right. It may even pass once. What it will not do is stay correct as the product changes. When a selector shifts or a loading state is added, the test drifts. The LLM that made the change does not update the test because it does not run the suite, does not reason about coverage, and has no mechanism to verify the assertion still reflects intended behaviour rather than incidentally passing.
The deeper problem is that LLMs lack the judgment to know whether a test is actually testing the right thing. They will write an assertion that an image element is present while missing that the element is invisible due to a silent error handler. Green test, broken feature. And when a test fails, the tendency is to make the assertion less strict rather than fix the underlying issue -- which is worse than no test at all.
The result: AI-assisted projects are systematically under-tested at the UI layer. The AI introduces regressions it cannot catch, writes tests it cannot maintain, and produces fixes that make tests pass without fixing features. The developer ends up doing all of it manually anyway. The cycle repeats with the next model swap.
The Defensive Fix
When tests are absent, the fallback is to stop relying on the model for correctness. Pre-format tool results so the model receives output it only needs to copy, not generate. Add server-side guarantees that inject expected content if the model omits it. Encode semantics in data at the retrieval layer rather than trusting the model to carry intent through instructions. Instrument the full agentic loop so when something breaks, the logs show exactly what the model received and returned. The logs become the regression record for everything the test suite does not cover.