New research: Leading indicators of AI coding agent effectiveness

Learn more

New research: Leading indicators of AI coding agent effectiveness

Learn more

NEW RESEARCH FROM SPAN

Beyond the Model: What Distinguishes

Effective Agentic Development

Evidence from real-world AI coding agent trajectories on the factors associated with lower cost, greater autonomy, and less rework.

Evidence from real-world AI coding agent trajectories on the factors associated with lower cost, greater autonomy, and less rework.

Download report

Looking beyond the model

Most conversations about coding-agent effectiveness still start with the model. Frontier model releases are visible and often impressive. But they don’t explain why teams using the same tools can see dramatically different results.

We studied what separates the trajectories that glide from those that grind, looking at effectiveness in practical terms: token efficiency, agent autonomy, and downstream rework.

The encouraging part is that many of the biggest levers are already inside your organization, and they can be measured and improved today.

Stephen Poletto

Field CTO, Span

Download report

METHODOLOGY

Scoring real-world AI coding agent activity

An effective agent trajectory produces accepted code at lower token cost, yields more accepted output for each human turn, and requires less review rework.

To identify the leading indicators of those outcomes, Span developed a research-backed framework that scores complete developer-agent sessions across three categories.

An effective agent trajectory produces accepted code at lower token cost, yields more accepted output for each human turn, and requires less review rework.

To identify the leading indicators of those outcomes, Span developed a research-backed framework that scores complete developer-agent sessions across three categories.

Prompt clarity

How clearly the task, scope, constraints, and definition of done were specified.

Environment readiness

How effectively the repository and tooling allowed the agent to work and verify its changes.

Quality stewardship

How well the interaction guided the change toward local standards and production readiness.

Three leading indicators of AI coding agent effectiveness

01

01

Clearer prompts are associated with cheaper code

Trajectories with clearer scope, constraints, and acceptance criteria consumed fewer tokens for each line of AI-generated code that ultimately shipped. Every one-point increase in prompt clarity was associated with a 27.2% reduction in cost per merged AI line. For a trajectory producing 50 merged AI lines, moving from a prompt clarity score of 3.0 to 4.0 reduced average token spend from $6.90 to $4.45.

Vague prompts create exploratory loops, incorrect implementations, and regeneration. Clear prompts narrow the search space before the expensive work begins, making prompt quality a measurable lever on AI cost.

Trajectories with clearer scope, constraints, and acceptance criteria consumed fewer tokens for each line of AI-generated code that ultimately shipped. Every one-point increase in prompt clarity was associated with a 27.2% reduction in cost per merged AI line. For a trajectory producing 50 merged AI lines, moving from a prompt clarity score of 3.0 to 4.0 reduced average token spend from $6.90 to $4.45.

Vague prompts create exploratory loops, incorrect implementations, and regeneration. Clear prompts narrow the search space before the expensive work begins, making prompt quality a measurable lever on AI cost.

01

Clearer prompts are associated with cheaper code

Trajectories with clearer scope, constraints, and acceptance criteria consumed fewer tokens for each line of AI-generated code that ultimately shipped. Every one-point increase in prompt clarity was associated with a 27.2% reduction in cost per merged AI line. For a trajectory producing 50 merged AI lines, moving from a prompt clarity score of 3.0 to 4.0 reduced average token spend from $6.90 to $4.45.

Vague prompts create exploratory loops, incorrect implementations, and regeneration. Clear prompts narrow the search space before the expensive work begins, making prompt quality a measurable lever on AI cost.

02

02

Ready environments are associated with greater agent autonomy

Agents were more autonomous when they could build the repository, run reliable tests and checks, interpret failures, and verify their own changes. We measure this as turn yield: the amount of AI-authored code that survives to merge for each human turn.

Every one-point increase in environment readiness was associated with 88% higher turn yield. Across scores of 2.5 to 4.5, agents produced roughly 5 to 17 merged AI lines per human turn.

Autonomy is less a property of the model than a property of the workspace you give it.

Reliable setup, testing, documentation, and fast feedback loops help agents make more progress before returning to the developer.

Agents were more autonomous when they could build the repository, run reliable tests and checks, interpret failures, and verify their own changes. We measure this as turn yield: the amount of AI-authored code that survives to merge for each human turn.

Every one-point increase in environment readiness was associated with 88% higher turn yield. Across scores of 2.5 to 4.5, agents produced roughly 5 to 17 merged AI lines per human turn.

Autonomy is less a property of the model than a property of the workspace you give it.

Reliable setup, testing, documentation, and fast feedback loops help agents make more progress before returning to the developer.

02

Ready environments are associated with greater agent autonomy

Agents were more autonomous when they could build the repository, run reliable tests and checks, interpret failures, and verify their own changes. We measure this as turn yield: the amount of AI-authored code that survives to merge for each human turn.

Every one-point increase in environment readiness was associated with 88% higher turn yield. Across scores of 2.5 to 4.5, agents produced roughly 5 to 17 merged AI lines per human turn.

Autonomy is less a property of the model than a property of the workspace you give it.

Reliable setup, testing, documentation, and fast feedback loops help agents make more progress before returning to the developer.

03

03

Stronger quality stewardship is associated with less rework

Trajectories with stronger stewardship addressed local conventions, failure handling, verification, production constraints, and rollout safety before the pull request was opened. They also left behind the context reviewers and future maintainers needed to understand tradeoffs, shortcuts, technical debt, and important implementation decisions.

Every one-point increase in quality stewardship was associated with 39% fewer review cycles per 1,000 lines of AI-generated code merged. At a score of 3.0, our model estimates approximately seven review cycles per 1,000 lines. At a score of 4.0, that estimate falls to roughly four or five.

When this work isn’t addressed during the trajectory, more of it surfaces later in code review.

Trajectories with stronger stewardship addressed local conventions, failure handling, verification, production constraints, and rollout safety before the pull request was opened. They also left behind the context reviewers and future maintainers needed to understand tradeoffs, shortcuts, technical debt, and important implementation decisions.

Every one-point increase in quality stewardship was associated with 39% fewer review cycles per 1,000 lines of AI-generated code merged. At a score of 3.0, our model estimates approximately seven review cycles per 1,000 lines. At a score of 4.0, that estimate falls to roughly four or five.

When this work isn’t addressed during the trajectory, more of it surfaces later in code review.

03

Stronger quality stewardship is associated with less rework

Trajectories with stronger stewardship addressed local conventions, failure handling, verification, production constraints, and rollout safety before the pull request was opened. They also left behind the context reviewers and future maintainers needed to understand tradeoffs, shortcuts, technical debt, and important implementation decisions.

Every one-point increase in quality stewardship was associated with 39% fewer review cycles per 1,000 lines of AI-generated code merged. At a score of 3.0, our model estimates approximately seven review cycles per 1,000 lines. At a score of 4.0, that estimate falls to roughly four or five.

When this work isn’t addressed during the trajectory, more of it surfaces later in code review.

The most important levers are already within reach

Model selection happens occasionally, but prompts, environments, and operator decisions shape every task. These findings point to improvements engineering teams can begin making today.

Model selection happens occasionally, but prompts, environments, and operator decisions shape every task. These findings point to improvements engineering teams can begin making today.

NEW RESEARCH

Download the full research report

Download the full research report

Explore the complete analysis, methodology, and practical recommendations for more effective agentic development.

Explore the complete analysis, methodology, and practical recommendations for more effective agentic development.

Download report

About the research

Span analyzed complete agent trajectories across real-world development activity, connecting the initial prompt, human and agent turns, tool use, resulting code, and linked review history.

We developed the framework through qualitative trace review, structured scoring rubrics, and automated evals calibrated against human ratings. We compared observable qualities of each interaction with downstream outcomes.

These findings reflect strong observational relationships, not proof of causality.

Download report

103

engineering teams

Engineering teams

12 weeks

of real-world development activity

from May through July 2026

of real-world development activity

from May through July 2026

Turn AI investment into measurable results

See how teams measure and optimize AI spend with Span.

Get a Demo

Turn AI investment into measurable results

See how teams measure and optimize AI spend with Span.

Get a Demo