Measure whether coding AI actually helps your team deliver software

Imagine a team trying AI to add a feature to maintenance software. An operations manager needs completed jobs for the month in a file she can use in Excel. The AI quickly writes the code producing a CSV file, a spreadsheet-compatible data format.

A developer then checks the rows, corrects a date rule and tests customer companies’ access permissions. Whether the team should use more AI depends on the time and cost of this complete job. Coding speed provides one part of the assessment.

A limited trial, often called a pilot, should answer a specific question: for this type of request, can the team produce acceptable software with less effort or shorter delays at a justified cost?

Choose a request you can evaluate

Define the need and quality criteria before starting. In our example, the file needs the right jobs, separate company records and understandable handling of download failures. Identify who can decide ambiguous cases.

A familiar feature is not necessarily harmless. A download can expose sensitive information. Begin with fictional data in an environment separate from the live customer application.

If nobody can describe the expected result, or the test copy will not run, resolving that problem may be the first investment. Include simpler options in the comparison: a better review procedure, a reusable feature or limited AI assistance might address the need.

Record current work before the trial

Examine comparable completed requests: what was delivered, coding time, review, correction and waiting. Keep difficult cases and exclusions visible.

Apply the same requirements to the AI trial. Comparing a small generated fix with a large historical feature would not establish a gain. With few observed requests, describe the results as an initial sample and explain what it does not cover.

DORA’s delivery metrics examine software delivery speed and stability together. For this trial, select measures that support your decision and avoid letting one speed figure hide correction work.

Keep essential quality conditions mandatory

A shorter delivery time cannot compensate for access to another company’s records. Before the trial, define unacceptable outcomes and the conditions required for customer release.

What the trial needs to establish	Information to keep
The feature meets the need	Expected result and actual downloaded file
Important errors are detected	Executed checks and rejected versions
Access limits hold	Permitted actions and blocked forbidden attempts
The team can repair a failure	An exercised procedure and recovery owner
Total effort is acceptable	Setup, coding, review, correction and support
The organization can maintain its use	Recurring costs and owners of rules and tests

This is a proposed decision worksheet, not a validated scoring system. An unchecked condition remains unknown; it does not pass by default.

Include unsuccessful attempts

Count model usage, execution tools and human effort. Separate initial setup from recurring expenditure while retaining both in the assessment. Rejected versions, retries and incidents attributable to the trial belong in the cost.

For requests of comparable size, one useful calculation is:

Cost per accepted change = total trial cost ÷ number of changes meeting the agreed criteria.

If no changes are accepted, report the amount spent and that outcome. The ratio cannot be calculated. Avoid artificially splitting a request into many small changes to improve the average.

Time released may help the team handle other requests without reducing spending. The article on AI time savings explains how to distinguish this additional capacity from financial savings.

Write a decision that follows from the observations

If features meet the need, total work decreases and the team can manage failures, it can propose a somewhat broader trial. Specify which additional requests AI will handle and which controls remain necessary.

If AI produces useful code but loses time in a broken setup, improving the environment may be the next investment. Define what the team will examine in a further trial to establish whether that repair helps.

If review and correction outweigh the observed benefit, keeping a narrower assistant role or stopping this approach may be reasonable. Preserve the reasons so later experiments can use what was learned.

The decision needs to include test quality, granted permissions and recovery work. A new model or tool may require those conditions to be reviewed again.

An Opportunity Sprint can investigate an agreed uncertainty and produce a decision brief. Scope, fee, timing and any supporting experiment are agreed before work begins; implementation is a separate commitment.