Measuring AI impact without fooling yourself

Usage is easy to count and easy to misread. Measure outcomes against a baseline, with quality beside speed.

The easiest AI numbers to collect say the least. Licenses assigned, suggestions accepted and lines generated show adoption, not value, and generated-code volume is easy to game.

Perception is not enough either. In a 2025 study by METR, experienced open-source developers took 19 percent longer on tasks when using AI tools, yet afterward still believed the tools had sped them up by about 20 percent. The study was small, and its authors warn against generalizing from it, but the gap between feeling and measurement is the lesson.

Compare against a baseline

DX organizes AI measurement into utilization, impact and cost. For impact, combine direct signals, such as time saved per developer, with slower ones, such as throughput and quality tracked over months.

To judge whether an AI rollout paid off, compare cycle time, change fail rate and rework for the same teams before and after, alongside a team that adopted later. If speed rose and rework rose with it, the answer is not yet.

Set the baseline before the rollout, not after.
Pair speed with quality, every time.
Keep it at team level: DX cautions against using these measures in individual performance reviews.

No single study or dashboard settles the question, and results vary by team, codebase and task. DORA’s 2025 research explains part of the spread: AI amplifies the strengths and weaknesses a team already has, so slow reviews and manual releases absorb the gain. Use measurement to decide where AI helps.