The Replay-Patch-Retest loop: closing security issues in hours, not quarters

The dominant metric in security operations is mean time to patch or, more honestly, mean time to anyone-actually-doing-something. Across the engineering teams we have worked with, that metric is measured in weeks for high-severity findings and in quarters for medium-severity ones.

The reason is not that engineering teams are lazy. It is that the handoff between finding-confirmed and patch-merged has too many steps, each of which is a place to drop the ball:

1. The pentest report arrives weeks after the engagement.
2. An engineer reads the finding and tries to reproduce.
3. They write a patch.
4. They write a test.
5. The test sits in PR review.
6. The patch ships.
7. Nobody re-tests against the original exploit because the original exploit is described in PDF prose, not as a runnable artifact.

By the time step 7 happens, if it happens, the model has been swapped, the prompt has been refactored, and nobody is sure whether the patch held against the actual original attack or only against the version of the attack the engineer remembered.

Brektra's product thesis is that this loop should be closed end-to-end inside one platform, with each step producing a runnable artifact the next step consumes.

Replay

When Brektra confirms an exploit, the output is not a PDF. It is a typed proof artifact, an interactive kill-chain replay, and a structured record of every request, response, and reasoning step the agent took.

The replay is the runnable artifact. An engineer who opens it sees the exact path. They can step through node by node, inspect the LLM call that landed the injection, and copy the payload to their local environment if they want to verify by hand.

This single change, replacing PDF with replay, is enough to compress the time between finding-confirmed and engineer-understands from days to minutes.

Patch

Findings have a Generate Patch button. The agent inspects the proof artifact, picks the right repository (the one wired into the workspace via the Brektra GitHub App), and opens a pull request.

The PR description includes the original kill-chain replay link, the finding's severity, and the OWASP mapping. Reviewers see exactly what they are fixing without leaving GitHub.

Patches are not always correct on the first attempt. The model sometimes proposes a fix that addresses the symptom but not the underlying class. That is fine: the engineer reviews the PR like any other PR and edits or rewrites if needed. The point is not that the machine writes the patch perfectly; the point is that the human starts from a draft, not from a blank file.

Re-test

This is the step most security platforms skip and the one that closes the loop.

When the patch PR merges, Brektra schedules a re-test scan. The re-test runs the same payload against the same target. If the exploit fails, the finding flips to patched and the kill chain shows both runs side-by-side. If the exploit still works, or works in an adapted form, the finding flips to bypassed and the new variant joins the agent's Exploit Memory.

We have seen patches that look correct in the diff but fail re-test because they only block the exact original payload, not the class of payloads. We have also seen patches re-test cleanly but bypass on the next quarter's model upgrade. The re-test loop catches the first kind on the same day; the second kind shows up the next time the scan runs.

Time-to-fix in practice

The dashboard's first_exploit_seconds and time_to_verified_fix metrics tell two halves of the story. The first is how long it took the agent to land an exploit; the second is how long it took to ship a verified patch.

Across early customers running Brektra against AI app surfaces, the median time from finding confirmed to patch verified is now under 24 hours for high-severity findings. The platform did not make the engineering team faster. It removed three of the seven steps.

Why this matters more for AI than for traditional web apps

Traditional web vulnerabilities have a long tail of well-understood classes. SQL injection has been the same shape for twenty years. Once you patch parameterized queries everywhere, you stay patched.

AI vulnerabilities do not behave that way. Every model upgrade, every prompt refactor, every new tool added to the agent's toolbox can re-open the same class of issue. The defense that worked against last quarter's prompt injection might fail against this quarter's because the model's instruction-following changed.

That makes the regression test on the original exploit the only reliable signal that you are still safe. Not unit tests, not synthetic benchmarks, not the assertion that "we patched it." A re-run of the actual confirmed exploit against the actual current production model.

This is the work the platform does between releases. The CISO ends up with a dashboard showing every confirmed exploit, every patch, every re-test outcome, and the cumulative compliance heatmap across OWASP LLM, OWASP Top 10, SOC 2, and ISO 27001. None of that requires a human to copy data between three systems.

What about findings that are not patchable

Some findings are configurations the customer has chosen. A scan that flags an aggressive rate-limit bypass on a free-tier endpoint may be behavior the customer wants. Findings can be marked accepted_risk with a note. Future scans skip them.

This is a common honest exit from the patch loop and the dashboard treats it as a normal terminal state, not a failure.

A few details about the technology

Three things are non-obvious about how the loop works under the hood:

• Patches are not generated from the proof alone. They use the proof, the structured remediation suggestion, and the file-level context the GitHub App provides. The model sees real source, not just a description of source.
• Re-test runs against the same target as the original scan. No staging-vs-prod drift. The verifier hits exactly what shipped.
• Exploit Memory captures bypass variants. When a patch is bypassed, the new variant becomes part of the shared corpus other scans can pull from. This is the network effect that makes the platform get better as more customers run it.

The combination is what we mean when we describe Brektra as a continuous AI app pentest rather than a one-shot scanner. You do not run it once a quarter. You run it on every PR, every release, and every model upgrade. The re-test loop is what makes that workable without burying engineers under noise.