When deploying AI coding agents, the temptation to minimize tokens per interaction can backfire. GitHub's engineering team discovered that cutting output short sometimes forces the model to make additional requests or repeat work, ultimately consuming more tokens and taking longer to complete tasks. The real efficiency gain comes from optimizing the entire workflow from user request to final result, not from trimming individual responses.

GitHub evaluated four specific improvements to Copilot using agentic coding benchmarks, then validated the most promising changes through controlled online experiments before deployment. These modifications span multiple Copilot products, including the CLI, the GitHub Copilot app, and Copilot code review, all of which share the same underlying infrastructure.

The local metric trap

Chart showing 3.1% 'Remove view previxes', 5.5% 'Selective output compaction', 2.9% 'Compact task-tool prompt', and 2.3% 'Reduce notification roundtrips'.
Figure 1: Four independent A/B experiments using the same AI-credit metric. The segments are shown together for comparison; their effects are not necessarily strictly additive.

One common cost-reduction strategy involves shortening tool output before the agent processes it. GitHub tested RTK (Rust Token Killer), a utility designed to compress shell output, against their agentic coding benchmarks. While RTK did reduce individual response sizes, the model frequently reopened the original output or reran commands to recover missing information when that context proved necessary. These recovery steps added extra turns and carried more context forward, resulting in higher overall token usage and longer task completion times despite shorter individual responses.

This finding revealed that tokens per tool call represents the wrong optimization target. Instead, efficiency improvements must be evaluated across the complete task lifecycle. The question became: what can be removed without forcing the model to repeat work?

Flow chart showing: RTK, compresses shell output > Local win, tool output gets shorter > Useful detail is missing > Recovery, reread or rerun > More turns and context carried forward. Then the option of finishing at 'End-to-end result, Tokens and cost up, Task duration up, Task completion: steady,' or 'Recovery repeats' going back to 'useful detail is missing'.
Figure 2: A shorter tool response can make the completed task more expensive when missing details force the agent to reread output, rerun commands, and carry more context forward.

Compress noise, preserve useful information

Analysis of benchmark runs showed that install, build, test, and lint output typically contains repetitive noise, while source-like output and arbitrary command results tend to hold information agents genuinely need. This insight informed development of a selective output compressor that preserves critical context while removing redundancy.

Early prototypes proved too aggressive, causing models to repeat work or retrieve full saved output, which increased end-to-end costs and reduced task success rates. When the team initially compressed git diff output, benchmark tasks revealed that agents reopened the original to recover missing information. These failures led to a three-part policy:

  1. Preserve source-like and arbitrary output. Commands such as cat, git diff, git show, and arbitrary scripts are returned unchanged.
  2. Reorganize search results without dropping content. Matches and file lists from tools such as grep can be grouped more efficiently while retaining every result.
  3. Compress repetitive noise selectively. Install, build, test, and progress output is compressed only when the savings are substantial.

The shipped version emerged through repeated evaluation and refinement, becoming conservative not by design but because that approach is what the data supported. Agents retain the ability to retrieve complete original output through a direct recovery path, which also serves as a safety mechanism and evaluation signal. When output compression triggered in offline tasks, no statistically significant task-success regression appeared, and agents extremely rarely accessed the saved originals. In the online experiment, average cost decreased slightly with no material regression in tracked quality metrics.

Flowchart showing how GitHub Copilot handles shell-command output. Copilot calls a shell command, classifies the output, then chooses one of three paths: keep arbitrary/source output unchanged, reorganize search results without losing any matches, or selectively compress repetitive noise (like install/build/test logs) while preserving full output and providing a recovery path. The processed result is returned to Copilot.
Figure 3: The shipped compressor preserves source-like output, reorganizes search results without loss, and compresses only predictable repetitive noise while retaining the full original.

Remove formatting before removing information

The view tool, which agents use to read file contents into context, previously prefixed every line with a number before displaying contents to the model. Earlier file-editing tools relied on these numbers to target changes, but current tools instead match surrounding code and no longer use line numbers. The prefixes remained despite becoming obsolete in the normal workflow.

Though each prefix was small, repetition across every line and every file read accumulated substantially throughout a session. Removing them reduced model-inference cost by roughly 5% in offline agentic coding benchmarks, with success rates staying within expected variance and no increase in edit failures. When tested with Copilot CLI users, the online experiment reduced average daily model-inference cost per user by about 3%, with no material regression in quality or satisfaction metrics.

This represented an ideal change: no new instructions for the model, no information source requiring recovery, and no additional decisions needed. File contents reached the model unchanged, freeing more of the context window for actual work rather than unused formatting.

Compress prompts without compressing intent

System prompts carry instructions that shape agent behavior and are sent to the model on every turn. Shortening them only improves efficiency if agents maintain the behaviors developers depend on. In GitHub Copilot, the task tool launches specialized agents for parallel work, and its guidance had accumulated across tool descriptions, schemas, agent definitions, system instructions, and companion tools.

A meta-prompting loop, in which Copilot iteratively wrote its own prompt, reduced that prompt by roughly half. The first online experiment uncovered a regression that initial offline evaluations had missed: the rewritten prompt had converted cautious parallelism guidance into a hard scheduling policy, forcing independent custom agents to run sequentially. The team halted the experiment and added regression testing for the exposed behavior.

The eventual fix replaced an explicit allowlist and denylist with a single sentence: "Independent agents can run in parallel; consider side effects." This shorter, less restrictive guidance deferred the parallelization decision to the model instead of explicit policy. The new behavior test passed without causing existing tests to fail.

The shipped prompt removes about 1,300 task-tool prompt tokens per turn, corresponding to approximately 1.8% fewer total prompt tokens per session and 2.9% lower normalized cost per active hour, with no quality regression detected in measured evaluations.

Deliver completed background work without an extra retrieval turn

Three-stage diagram labeled Compression → Regression + fix → Completed. Left panel shows an original prompt compressed by about 50%. Middle panel highlights a regression where agents became serialized, then a fix by editing one sentence to restore parallelism. Right panel shows final shipped prompt with restored behavior and cumulative savings of about 1,300 fewer tokens per turn across steps.
Figure 5 Prompt compression became safe only after a regression test exposed serialized agents and a one-sentence fix restored parallelism; the resulting token savings recur on every model turn.

Agents often run independent work in the background, such as long-running shell commands alongside sub-agent investigations. Previously, when such work completed, the harness notified the model but did not include the result, forcing the agent to spend another turn retrieving output that Copilot had already received. When multiple tasks finished close together, this detour could repeat multiple times.

Copilot now batches eligible completion notifications and delivers completed results directly in the existing tool-result format, allowing the agent to continue with needed information without an extra turn. Before this change, each completed task required one model call to request its result and another to process it. For a shell command and sub-agent working in parallel, that meant four model calls before work could proceed. Now the harness batches both completions and supplies their results together in a single model call, avoiding the need to carry full session context through unnecessary calls.

By delivering completed results directly without compressing, summarizing, or withholding anything, the harness reduced average token-related usage, as measured in AI Credits, by about 2.3%.

Before-and-after sequence diagram comparing orchestration behavior. Before: model waits on separate shell and sub-agent completions, causing retrieval detours and four LLM calls to process two results. After: a harness batches related completions and emits synthetic tool events so background work continues while waiting; both results are processed together in a single LLM call. The visual emphasizes reduced latency and fewer model round trips.
Figure 6 Before, each background completion could wake a retrieval-only model turn. After, the harness batches eligible completions and delivers completed results in the existing tool-result format.

Measure changes in context

A change that saves tokens in one Copilot workflow can increase costs in another. For example, tighter file-tool instructions inspired by positive results in Copilot code review increased cost in a Copilot CLI online experiment, so the team did not ship it. Conversely, removing line-number prefixes and selectively compressing output each reduced average prompt tokens per review by roughly 5% in independent evaluations across a large set of Copilot code review tasks using the production model, with no material change in tracked review-quality metrics.

These findings are separate from an earlier migration of Copilot code review to shared file tools, which together with review-instruction tuning reduced code review cost by about 20%. Each change requires measurement in the specific workflow where it operates.

Five lessons for building efficient AI coding agents

  1. Optimize the completed task, not the tool call. Shorter output is not cheaper if the agent spends more turns recovering what was removed.
  2. Optimize orchestration, not just model output. Eliminate model turns that perform work the harness can complete deterministically.
  3. Compress by what the output represents. Preserve exact content, prefer lossless transformations, and measure how often agents use the recovery path.
  4. Prompt rewrites sometimes have unintended consequences. Validate that intended behavior is preserved.
  5. Evidence is local to the workload. Re-evaluate changes in offline benchmarks, online experiments, and every product surface where they ship.

None of these changes made the model smarter. They removed work the model never needed to do. The improvements described are shipping across GitHub Copilot experiences that use the same underlying harness.