OpenAI this week highlighted progress in legacy code modernization, revealing benchmark results that point to substantial improvements in how its models handle COBOL translation and software construction in languages with sparse training data. The company's latest models show measurable gains, though the full evidence supporting the most ambitious claims remains private.
On OpenAI's proprietary COBOL migration evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort—a sharp rise from GPT-5.6 Sol's 5.4%—while consuming 39% fewer resources. When operating at higher reasoning effort, the newer model achieved 97.9% accuracy. However, the transparency of these two evaluations differs markedly.
OpenAI has declined to publish specifics of its COBOL evaluation but has made the MultiLangBench test suite and setup available on GitHub. This second benchmark measures whether models can construct a SQLite-compatible database engine from the ground up across 20 programming languages. The company characterizes the results as "substantial improvements across both evaluations, delivering much higher accuracy at lower cost."
For teams working on legacy system maintenance and upgrades, such results could carry real significance. Yet the absence of public COBOL test cases prevents the developer community from validating the evaluation independently.
COBOL modernization: A persistent challenge
According to OpenAI, its COBOL migration evaluation "tests whether a model can migrate existing programs to another language while preserving their behavior." The test specifically covers migrations from COBOL to C++, Java, Kotlin, Python, and Rust. Successful migrations must satisfy a battery of concealed tests demanding "byte-for-byte parity on observable outputs." GPT-6.1 Sol's substantially higher accuracy demonstrates real progress on this front.
The performance gains are noteworthy, though OpenAI is not alone in applying AI to COBOL modernization. COBOL remains a formidable challenge for AI coding assistants despite its critical role in global infrastructure. IBM reports that COBOL underpins over 40% of all online banking systems, supports 80% of in-person credit card transactions, and handles 95% of all ATM transactions. Yet models encounter fewer COBOL code samples and less documentation compared to languages like Python and JavaScript. The scarcity of seasoned COBOL professionals compounds the difficulty, and system documentation frequently lags behind the actual software it describes.
Broader language performance across MultiLangBench
OpenAI developed MultiLangBench as a second evaluation to assess how well models can write software from scratch. The benchmark "tests whether a model can build a SQLite-compatible database engine across 20 different programming languages." The process begins with a database and SQL query, then measures whether the model produces a correctly modified database. The reference SQLite implementation serves as the baseline for validation.
GPT-6 Astra, OpenAI's primary model, averaged an 87.4% pass rate across all 20 languages—a substantial leap from GPT-5.6 Sol's 26.8%. The model exceeded 83% on 19 of the 20 languages tested and showed gains on every single one. GPT-6 Luna, the lower-cost variant, achieved a more modest 40.3% average pass rate but still outperformed GPT-5.6 Sol on every language, with an estimated 78% reduction in API costs.
The transparency gap
OpenAI's reported results from both its COBOL evaluation and MultiLangBench appear promising, yet benchmarks alone provide an incomplete picture. As of Thursday, Real-SWE 1.1, which draws on private codebases, ranked Claude Opus 5.5 first at 56.8% resolution. An earlier Real-SWE assessment covered by The New Stack showed different rankings and lower overall scores.
A critical limitation remains: developers cannot examine OpenAI's COBOL migration benchmark. While the announcement credits input from developers and enterprise customers, the absence of public test cases and datasets prevents external parties from validating the findings. OpenAI itself acknowledges a constraint, stating that "applying it to production software still requires expert review and testing against the system's business rules and dependencies."
OpenAI enters a competitive space where others have already invested effort. Anthropic detailed how Claude Code could support COBOL modernization on February 23. AWS rolled out AI agents for mainframe migrations during 2024, as reported by The New Stack.