10x Faster at What? Measuring AI's Real Impact on Software

The promise

It is the year 2026. Everyone is losing their mind over AI. It is everywhere. 10x developer, 100x faster feature delivery, 1000 pull requests per day, 30k lines of code per month. Company owners are being promised cheaper, faster, and more (and more) software development. Developers are going crazy rewriting tools in different languages. Many more are launching AI startups, hoping to cash in on the AI hype train. Tokens are flowing like there is no tomorrow.

However, almost nobody asks the obvious follow-up question: does this 10x faster, 10x cheaper feature make any difference to our business?

AI is changing how quickly we produce code. In the teams I’ve worked with, the way we chose features, approved releases, and measured success changed much less.

Before AI: a missed customer target

Before the AI explosion, the software development process looked more or less the same as it does now: analysis, coding, testing, building and deployment. Only one step has changed drastically. AI agents generate code much faster. But that doesn’t mean the features are better.

An example from my experience. Two teams spent more than three years building 5 big features in total. Each feature was a complicated workflow that clients were supposed to use to make their daily work easier and faster. The CEO and PO assumed that we would eventually onboard 14 clients once all the features were in place. This was a measure of success. Two years, three features implemented, only two paying customers. The teams still moved on with planned features with the promise that 12 more customers would be onboarded. In the meantime it turned out that one of the features had to be half-rewritten because it did not meet customers’ expectations. 1 year of effort went down the drain. The second feature turned out to be complicated and a customer wanted a simplified version. More effort went down the drain. The upcoming features were supposed to attract the other customers. Another year went by and no new customers onboarded. The company board claimed that it was difficult to convince customers to use the product. I’m not arguing with that. This might be true. However, I found out during discussions that development had continued without the features ever being evaluated with customers.

From a resource and risk management perspective it sounds like a bold move: to make decisions based on the assumption that board members know exactly what customers want.

By any activity measure, this project was a success: many complicated features delivered, with plenty of tickets closed and code merged along the way. A lot of code, a lot of commits. The whole feature delivery cycle was busy. Only an outcome measure showed that 3 years of work didn't achieve its stated goal.

We asked why those 12 customers didn’t sign. The official response from board members was: it is a difficult industry. However, from my discussions with sales, I gathered that we were building the wrong thing. We were focusing on implementing fancy UIs for manual workflows, whereas the real need was to automate data collection and aggregation by integrating systems through a B2B API.

With AI: more code, the same problems

The same product, one year later. The developers started using AI-assisted development, and the effect was immediate: commits and pull requests landed a couple of times every day. The teams clearly produced code faster than before. But delivery time barely changed. Every release still had to wait for the Change Advisory Board meeting. And the outcome didn't change at all. The number of people using the new features did not increase.

Then leadership responded. The CTO started measuring lines of code and pull requests across the teams working on the product. What was the purpose of this measurement? The answer was: let’s make the teams’ productivity visible and make assumptions about the company’s success based on that. Therefore, in the whole lifecycle of software development, only one aspect was measured. This measurement is not new. It is as old as software development. But it does not serve the purpose.

This metric only confirms that AI is working, while the overall feature delivery process didn’t change much. Comparing two teams on these numbers will also produce what such comparisons always produce: more pull requests, not more value.

What the numbers miss

I often heard product owners say that development was too slow. Yet defining features could take three to four months, and the teams still needed discussions with experts to clarify the scope. That time was not visible in a count of lines of code or pull requests. After coding, releases still depended on user acceptance tests, integration tests and the CAB. AI had not changed that part of the process either. Producing code faster had a limited effect while the decisions and release process stayed the same.

This is a textbook case of what Eliyahu Goldratt called the Theory of Constraints: speeding up a step that isn't the bottleneck doesn't make the system faster. Even if we had removed those delays, we would still have faced the problem from the first story. My discussions with sales had pointed to a need for automated data collection and aggregation, while we were building interfaces for manual workflows. Releasing those workflows sooner would have left that mismatch in place.

I’m not saying that counting commits and pull requests are useless. They can help describe coding activity. But I cannot judge the success of a feature from how much code went into it or how many pull requests were merged.

There is another risk when those counts become targets. If a team is judged by the number of pull requests it produces, it has an incentive to increase that number. The measurement can start shaping the work. That is the problem described by Goodhart’s law.

The root cause: software as a cost

In my experience, software development has often been treated as a cost to control. I’ve heard the view that board members are the core of the company and everyone else is an expense, including the engineers building and maintaining the product. When that is how development is seen, the natural question is: how do we get more output for the money? Lines of code and pull requests fit that way of thinking. They are easy to count and easy to put into a report. If the numbers go up, it looks as though the company is getting more from its development teams. Whether the work was worth doing can remain unanswered.

The 10x promise is attractive in that setting. It promises more software for the same money, or the same software for less. With AI the expectation becomes that developers should deliver more because writing code is faster. The incentives around the writing code can reinforce this. On the three-year-long product we were mostly working on a fixed set of features. Promised, but not evaluated. The pressure was on delivering, instead of verifying. Product owners were judged by roadmap completion, while the CTO had to show the team’s productivity through the number of commits. Under those conditions, another feature finished and shipped to production was considered a great success. With this approach software development looks more like a production line. In fact we had a Jira dashboard that was named ‘Production line’.

In contrast to that, a team could spend two weeks building a rough version of a feature, test it with users, and discover that it does not solve their problem. If that finding changes the decision and prevents months of further development, the work has produced something useful. On our project, prototyping was considered a waste of time. Yet we spent a year on a feature that later had to be half-rewritten because it didn’t meet customers’ expectations.

The team should be able to explain what it learned and why it stopped development, and have that recognised as a useful result. If AI makes this kind of test cheaper, the way we judge the work needs to account for it. Otherwise, faster development just becomes another reason to demand more features, while the decisions about which ones deserve to be built stay the same.

Code was never the problem

I want to make it clear that not every company works this way. Some have evaluated features quickly for years. In the last few months I read about how Spotify and Booking.com work, and I was stunned. Spotify teams run experiments as a normal part of building features. In a similar way Booking.com works on their features. Both companies put effort on the experimentation and fail fast approach. The Booking example is also quite well described by Stefan H. Tomke in his book “Experimentation Works: The Surprising Power of Business Experiments”.

Notice that they did this before AI. They had the same expensive code as everyone else, and still tested whether features worked and decided based on evidence.

Put them next to the three-years-long workflow project. No AI, similar cost of producing the code. The difference was in how decisions were made and how success was measured. AI alone can't fix it. Faster code delivery will not solve the problem of mismatched customer needs.

Using AI where it matters

I once asked why we were developing a particular feature. The answer was: “Because I think it will work for users.” That is a reasonable place to start. But for me this is an assumption to test before committing to the full feature. We could do that before AI. The opportunity now is to reduce some of the effort involved, so we can check our assumptions earlier.

Before choosing a feature

Start with the information the company already has: support tickets, sales notes, CRM entries and feature requests. AI can help group similar problems and identify which customers reported them. Each summary should be linked to the original records. That gives us a way to check what people actually said and whether a recurring complaint comes from several customers or one customer repeating the same request.

Before committing to full development

Use AI to help build a small version of the proposed solution that customers could try. Just a dirty prototype. In the workflow project, that could have meant showing one rough workflow to a few intended users before completing the other features. We could have watched how it fitted into their work, where they still needed manual steps, and what they expected it to do differently. Their responses could then inform a decision to continue, change direction or stop.

After release

Make measurement part of the feature. AI can help write the tracking code, prepare queries and produce reports. We still need to decide what improvement we expect and check that the measurements capture it. For these workflows, I want to know whether the intended customers were using them and whether completing the work had become faster or easier. A report could then compare those results with the expectation that justified building the feature.

The summaries and generated tracking would still need verification. Customer information would also need to be handled within the company’s data protection rules. An AI-generated answer is useful only to the extent that we can trust the information and reasoning behind it.

In the three-year project, we kept committing effort while the customer target remained unmet. The time saved on implementation should be used to check those decisions sooner, while changing direction is still a manageable part of the work.

A first step: three questions for every feature

Put three questions into the ticket and use the refinement discussion to answer them:

What problem are we solving, for whom, and what evidence do we have?
What do we risk by building this feature, and by leaving the problem unsolved?
What improvement would count as success, and how and when will we check it?

The answers should distinguish what we know from what we assume. If the answer to the first question is still “I think customers will want it,” write that down as an assumption. Then agree what we need to find out before committing to the full feature. Name someone to review the result and set a date for that review. Decide how the evidence will be collected, whether through tracking, customer feedback or both. After release, return to the original answers and compare what we expected with what happened.

The activity metrics should stay in the report. That gives us something specific to work on and reason about. We can discuss why the expected improvement has not happened and what we should check or change next.

After a few features, the team will have a record of which assumptions held up, which did not, and what it learned. The evidence should be in front of us when we decide where to spend the next month of development.

The AI assistant can help keep this record up to date. With consistent feature documentation and a feature registry, it can prepare a summary every two weeks or once a month. Ask it to do three things:

Verify if there is evidence of argument against or in favor of a given feature. If no answer yet then notify the PO.
Verify if the risk assessment is in place. And again notify if there is none.
Make a summary of metrics for already released features. This one assumes that there is a well established system observability and customer feedback process.

With this setup, the PO or CTO could spend less time gathering updates and checking which reviews are overdue. Tthe assistant should bring the features that need attention into the review, with the evidence beside them. The PO or CTO still need to assess the findings and decide what to do next.

Summary

Three years. Five features. Two paying customers instead of fourteen. And a Jira dashboard called “Production line”.

When I look back at that project, I keep wondering how much work we could have saved by showing the workflows to customers earlier. The teams delivered what was planned. But the plan was wrong. Based on assumptions, not data.

In my opinion, one of the qualities of the system design should be readiness for experimentation. That means using A/B testing with a control group, multivariate testing and prototyping, as described in Thomke’s book.

AI-assisted work means that we, as humans, still need to use our brains, because this is assisted work. AI will not answer the question "Were the outcomes worth the effort?" but it can vastly help to find the answer.

Use your agents wisely: