On Thursday, AWS made Continuum generally available as a tool for identifying and remediating code vulnerabilities. The service scans repositories for bugs, validates them against the customer's AWS infrastructure, and generates pull requests containing remediation proposals.
The 89% benchmark result
AWS published its benchmark results on October 5, more than two weeks ahead of the product launch. Continuum successfully completed 819 of 920 tasks on CyberGym-E2E, achieving an 89.0% end-to-end success rate and surpassing the previous public benchmark high of 65.9%. Matt Garman, AWS's CEO, stated on X that AI is genuinely transformative for defenders and that Continuum outperformed all other AI agents tested on this benchmark.
Corey Quinn, chief cloud economist and co-founder of Duckbill, questioned whether the announcement had been accelerated, though he offered no substantiation for the claim.
Understanding CyberGym-E2E
The CyberGym-E2E benchmark allocates 90 minutes for an agent to locate and repair a vulnerability across one of 139 open-source projects. The agent receives no bug description, proof of concept, crash log, or existing patch. It must discover the flaw independently, reproduce it, and deliver a fix that halts the crash while preserving all existing tests. The benchmark focuses exclusively on memory-safety bugs in C and C++, excluding numerous other programming languages and vulnerability categories, as AWS acknowledges.
The evaluation proceeds through four cumulative phases. Continuum identified and reproduced bugs in 92.5% of tasks, prevented the crash in 89.6%, and maintained passing tests in 89.0%. The final phase determines whether the patch also resolves the specific historical vulnerability the benchmark designated. Since the agent never learns which historical bug it should target, success depends on fixing any genuine flaw in the codebase. Continuum's patches addressed the benchmark's target historical bug in 37.8% of cases, compared to the previous public record of 26.2%.
The gap between general and historical bug fixes
AWS argues there is merit to the distinction. A single repository typically contains multiple vulnerabilities, so fixing a different flaw than the one the benchmark targets does not negate the value of the work. This dynamic accounts for the disparity between the 89% general success rate and the 37.8% historical-bug fix rate. Continuum achieved 89% success by identifying and patching a genuine vulnerability without breaking tests, yet only 37.8% of those patches happened to address the specific historical vulnerability the benchmark was evaluating. Since the agent had no information about which bug the benchmark was tracking, a direct comparison between the two metrics is imperfect. The remaining unknown is Continuum's performance when teams direct it toward a particular, identified vulnerability.
Testing and validation
AWS conducted the benchmark tests internally and reports that Continuum adhered to the benchmark's rules, including restrictions on internet access. The company verified that the agent was not simply retrieving publicly available fixes. Continuum met the 89% success threshold within the 90-minute time constraint. AWS notes that extending the time limit raises the success rate to 93.7%. Although the benchmark and testing infrastructure are publicly available, the results reported so far come exclusively from AWS.