There is a specific kind of confidence that comes from watching a browser automate your own application. Not unit tests passing in a terminal, not type checks going green -- the actual UI opening, a keyboard shortcut firing, a modal appearing, a diagram being renamed, and a test runner confirming that what you thought was there is still there. DrawmaticAI now has that.
The app is not trivial to test. At its core is an embedded draw.io canvas inside an iframe. Surrounding it are a diagram finder (Ctrl+K, a searchable modal), an AI chat panel, multi-tenant workspaces, rename controls, CSRF protection on every mutating endpoint, and a rate limiter on the login flow. Features accumulated over months. The question eventually becomes: when you touch one thing, how do you know another thing did not quietly break?
The OAuth Problem
Real Google OAuth is not testable in automation. You cannot drive a browser through a Google sign-in page in a repeatable, non-flaky way without becoming Google. The test suite sidesteps this entirely. When the server runs with a test flag set, it exposes a demo login endpoint -- a single URL that issues a real session cookie for a synthetic user. The browser hits it once, stores the cookies, and every subsequent page load arrives already authenticated. No credentials, no OAuth dance, no dependency on an external service. The session is as real as any production session; only how it was acquired differs.
Personas
DrawmaticAI has three distinct access levels: a standard user, a workspace admin, and a superadmin. Each has different capabilities and different endpoints they can reach. A test that only ever runs as one of them proves nothing about the others. So the suite has three persona fixtures, each of which logs in as the corresponding role before the test runs and attaches the resulting browser storage state to the page context. A CSRF test, for example, runs as a standard authenticated user -- because the interesting question is not whether an anonymous request is rejected, but whether a valid session without the right token is rejected. The distinction matters.
Fast Tests and Slow Tests
The AI chat panel depends on Ollama. Not every machine running the tests has a local model available. The suite treats this honestly rather than mocking it away. Tests that exercise real AI generation are marked to skip automatically if no Ollama instance is reachable. Tests that cover everything else -- the UI, the keyboard shortcuts, the API contracts, the security controls -- run in milliseconds and need nothing except the app server. This means you can run the fast suite on any machine, at any point, and get a meaningful signal. The slow suite runs when you actually want to test the AI path end to end.
The finder modal got the most coverage. Ctrl+K opens it; Escape closes it; typing filters the list in real time; arrow keys navigate; clicking opens the selected diagram; the delete button accepts a confirmation dialog and removes the entry. Each of these is a separate assertion against a real browser, not an inference from reading the JavaScript. The rename flow runs similarly -- a click on the diagram title, a browser dialog prompt, a new name submitted, and then an API fetch to confirm the database was actually updated, not just the DOM.
Security as Regression Tests
Two of the test files are purely about security properties. The CSRF suite sends authenticated requests to every mutating endpoint -- diagram creation, update, rename, delete, tenant creation -- without the required token. Every one should return 403. One positive case confirms that a request with a valid token is not also broken. The rate limiter suite hammers the login endpoint until the threshold is crossed and then asserts on the 429 response and its Retry-After header.
These are not theoretical. In the three days before the Playwright suite was written, DrawmaticAI went through three security review passes. Collectively they found and closed a lot of ground: CSRF controls, rate limiter IP extraction, WebSocket name XSS, XML size limits and XXE guardrails, audit field bounds, session umask, proxy header trust, JWT DATA_DIR scoping, and a handful of backend robustness issues. Each was fixed. The CSRF and rate limiter tests lock in two of those fixes explicitly; the API and UI suites cover the rest by asserting that the app still behaves correctly after any future change touches those paths.
Security controls that are not tested are security controls you will one day discover are gone. The gap between "we fixed it" and "we can prove it still works" is exactly what regression tests close.
What It Feels Like
There is a post on this site called "Green Tests, Broken Features" -- about the moment a model swap passed all the unit tests and broke four visible features anyway. That failure was the argument for this work. Tests that only check the internals do not answer the question users actually have: does the button do what it says? Playwright answers that question by being the user. It opens the browser, finds the button, clicks it, and tells you what happened. The rest is just details.
DrawmaticAI did not change much to get here. The test flag on the server, a demo login route, a thin persona fixture layer -- that is the entire scaffolding. The payoff is that the next time something subtle breaks in the finder, or a security control quietly regresses during a refactor, there will be a test that was already watching.