Developers face a persistent risk when working with credentials: accidentally committing sensitive data to repositories. GitHub Secret Protection addresses this threat by detecting exposed credentials and alerting teams before harm occurs.
The platform provides multiple safeguards against credential leaks. Push protection blocks commits containing sensitive information before they reach the codebase. Validity checks help teams prioritize alerts by confirming which secrets remain active. A partner program enables providers to receive notifications of exposed tokens in public repositories and take immediate action. Copilot secret scanning detects generic secrets such as passwords and connection strings. Custom patterns allow organizations to define expressions for detecting credentials specific to their infrastructure.
The Secret Protection team introduced validity checks in 2023, initially focusing on the most frequently leaked token types including AWS keys, GCP credentials, and Slack tokens. The system achieved roughly 80% coverage of newly created alerts, leaving less common token types unvalidated. As agentic AI gained momentum in late 2024 and early 2025, the team explored whether coding agents could help close this coverage gap.
Automating a Framework-Driven Workflow
The team examined their existing process for adding validation support to new token types. The framework-driven approach consisted of four steps: researching the provider to identify a suitable validation endpoint, writing validator code, darkshipping the validator to observe results without persisting data, and fully releasing the validator by removing darkship configuration.
The coding and release phases emerged as prime candidates for automation. However, the initial research phase proved challenging for Copilot. While the coding agent sometimes struggled to reliably identify the least-intrusive endpoint that experienced engineers would select, it excelled at generating code. The team discovered that creating and testing live tokens and interpreting subtle API behavior changes remained best suited to human engineers.
The workflow evolved to leverage Copilot's strengths. Engineers conducted research and created detailed issues containing project background, API documentation links, and implementation examples. These issues served as comprehensive prompts for the coding agent. The team learned that Copilot sometimes failed to follow documentation links, so they added a notes field for additional context. After assignment, the agent automatically generated pull requests ready for testing and human review. Engineers could request changes by adding comments to the pull request. The agent made occasional mistakes, such as missing documentation details, but the team treated its output like any human-written code, subjecting it to automated testing and review before deployment.
The darkship capability proved essential for quality assurance. Engineers observed new validator results without writing validity inferences to the database, allowing them to identify and fix discrepancies between API documentation and actual behavior. When confident in the implementation, the team asked Copilot to make a small configuration change to release the validator fully.
Results and Impact
Before the Copilot experiment, the team was validating 32 partner token types, a pace that required several months as engineers balanced new checks with ongoing feature work. Using Copilot, engineering interns @inshalak and @matthew-tzong directed the coding agent through the process and onboarded nearly 90 new token types in just a few weeks.
Coding agents proved viable for accelerating framework-driven repeatable workflows. Copilot functioned as a force multiplier, enabling parallelization of multiple research tasks across multiple agents simultaneously. The agent delivered speed and scale without replacing human engineering judgment. The team succeeded by integrating Copilot into specific parts of the framework while maintaining rigorous code review, testing, and verification.
Key Lessons
- Automation amplifies repeatability: processes with well-defined steps benefit from coding agents that scale efforts and multiply impact
- Treat Copilot like a team member: its contributions require the same careful review, testing, and feedback as human-written code
- Prompt quality drives results: detailed, example-rich prompts with additional notes helped Copilot deliver higher-quality pull requests
- Iterate on your process: prompts required refinement and workflows benefited from adjustments as the team learned what worked best
- Parallelization is a superpower: assigning many tasks simultaneously unblocked work that would otherwise queue behind a single engineer
The Secret Protection team sees significant potential for coding agents in other repeatable engineering tasks and is experimenting with similar processes in additional onboarding workflows. Many teams across the industry likely have framework-driven workflows suitable for this type of automation. Organizations looking to introduce automation should leverage existing repeatable processes, invest in quality prompts, and keep collaboration and review central to their approach.