At OpenAI, artificial intelligence now serves as a gatekeeper for all code submissions, with the ability to prevent merges when security issues are detected. Each pull request from an OpenAI engineer undergoes automated security screening, and if the AI identifies a flaw, the code cannot proceed without resolution.

Thibault Sottiaux, who leads OpenAI's Codex team, discussed this approach during a recent conversation on The Pragmatic Engineer, noting that the security check operates as a mandatory step that functions independently of human intervention.

Beyond security screening, OpenAI's models handle additional responsibilities including code assessment, regression detection, managing library updates, and supporting engineers with tasks that previously demanded months of work. The company has begun evaluating certain code-review models as performing at "superhuman" levels.

AI reviewers block every merge

OpenAI began developing specialized code-review models during the early phases of Codex. Sottiaux explained that these models excel at identifying logical errors and reasoning flaws that human developers might require hours to discover.

When we benchmark them, it's like they're superhuman in code review. This is not just true for correctness. This is also true for security.

Thibault Sottiaux

What began as isolated review tools has now been integrated into OpenAI's primary models. For security matters, any flagged concern automatically halts the merge process without exception.

Intent replaces inspection

As AI assumes greater responsibility for code review mechanics, Sottiaux believes human involvement may shift to earlier stages of development.

OpenAI's processes for review, deployment, and catching regressions have become substantially automated, according to Sottiaux. Engineers can deploy a pull request the same day to ChatGPT, which operates with approximately one billion active users.

Really what we see, and I see, is there's this sort of discussion around the intent that takes place around the pull request. It's like, what are you even trying to do? And is that the right thing to attempt to do?

Thibault Sottiaux

Sottiaux contends this conversation should occur sooner, during the planning phase rather than in the review queue. Team members must still align on objectives and validate proposed modifications. Transferring review responsibilities to AI does not eliminate human judgment; rather, it repositions that critical evaluation to before code submission.

Agents tackle maintenance backlogs

While security captures headlines, routine maintenance may be where engineering teams notice the most immediate impact, particularly with external libraries that introduce breaking changes and accumulate in backlogs as feature work takes precedence. Sottiaux notes that provided there is a clear changelog and adequate documentation, an agent can complete these repetitive updates within a single day, and the same applies to standard security updates.

The same principle extends to substantial refactoring efforts. A team might have a clear vision of what requires improvement and possess a superior design approach. However, when estimates reach two or three months of labor, postponement becomes the default choice. The existing code functions despite its flaws, and other priorities demand attention.

This calculation shifts when an agent can shoulder much of the burden. A refactoring project that would have been shelved due to resource constraints might now require days instead of months, making approval far more feasible.

When models outgrow their scaffolding

Sottiaux identified a pattern in agent development that diverges from typical software progression.

Codex included a /goalbuilt command designed to maintain model focus on a single task across extended periods. Sottiaux characterized this as a "crutch"—a temporary fix addressing the model's difficulty sustaining attention during prolonged operations. Newer iterations eliminate this requirement.

You don't need slash goal anymore. You don't need a harness around it.

Thibault Sottiaux

The Codex team frequently constructs auxiliary systems to compensate for model limitations, only to discover that subsequent generations handle the same tasks independently, rendering the previous scaffolding obsolete.

Sottiaux indicated the team now incorporates this into strategic planning, occasionally deciding against building workarounds if researchers anticipate the upcoming model will address the issue naturally within months. As models advance, the system prompt and supporting code shrink, while previously essential components vanish.

The blind spot question

A significant concern emerges when AI generates code and AI evaluates that same code—both systems may harbor identical blind spots. This objection remains largely unaddressed in Sottiaux's remarks.

OpenAI grants these models sufficient authority to block submissions, making their errors consequential in tangible ways. Excessive caution leaves engineers stalled on acceptable code. Missed vulnerabilities allow problematic code to advance with an automated security clearance falsely suggesting safety.

Complexity increases as AI-generated code becomes more prevalent. Code may compile successfully, satisfy test suites, and still harbor issues invisible within a pull request. Some problems may not originate from the submitted code itself. When dependencies form the attack surface, the vulnerability might stem from a library compromised weeks or months prior, requiring a reviewer to identify a problem that began elsewhere.

Code can compile, pass its tests, and still have problems that aren't obvious from the pull request itself.

Source: The New Stack