When Measurement Becomes the Goal: From Window Tax to PR Counts

Introduction

In one of my jobs, I once wrote a small tool to count how many Pull Requests I and other team members had contributed that quarter, how many Review Comments we had left, and how many Tickets we had resolved, hoping to use it to prove my output and contributions to my manager.

After looking at my report, my manager said calmly that performance reviews are not based on output alone; they also consider impact and the value you bring. At the time, I was a little upset, thinking I had clearly done so much but hadn’t received the reward I deserved.

I was so annoyed that I asked my manager, “So you mean it’s okay if I just slack off and produce nothing?” He didn’t answer that question directly. And, as expected, I did not get the performance rating I had hoped for.

A few years later, I finally understood what my manager meant, and even felt grateful that he hadn’t simply praised my output. When measurement becomes the goal, it loses its meaning.

Stories of Metrics Gone Wrong

In 1696, England began taxing house windows. The more windows a house had, the heavier the tax. Usually, houses with more windows belonged to wealthy families, and tax collectors could count them from the street without entering the home.

To reduce taxes, windows across the country were bricked up one by one. New houses were even built without windows. At the time, infectious diseases were common in England, and without windows, houses were poorly ventilated, making people more likely to fall ill.

The ancient Egyptians built special nilometers along the Nile to measure water levels, using the height of the annual flood to determine that year’s tax rate. Water level is a natural indicator that is virtually impossible to fake, yet avoidance still didn’t disappear.

Since they couldn’t change the tax rate, farmers underreported their farmland, bribed survey scribes, and stuffed stones into the grain they turned in to make up the weight. The state then had to send people to monitor harvesting and grain transport the entire time. When tax rates were high, marginal returns diminished, so farmers naturally had little incentive to increase output.

In Hanoi, Vietnam, in 1902, the government announced a bounty for rat tails in an effort to eradicate rats. The result was rats being released after their tails were cut off, and even people who raised rats specifically for the bounty. The rat problem was not solved—it actually got worse.

When measurement becomes the goal, it is no longer a good measure.

Why Are Proxy Metrics So Tempting?

Measurement is extremely important. In software development, there is a term called “Observability.” Making numbers visible is a crucial part of software development; only with numbers can we know how to improve and fix bugs.

• A sudden increase in database connections: maybe a poorly written SQL query is causing slow queries to occupy all available connections
• High CPU usage: maybe the data volume is too large, causing computation to rise sharply
• Significantly slower API response times: maybe a third-party dependency has failed

Measurement is one of the important ways we find problems and improve products. The issue is that many people in management only vaguely understand the effects of metrics, and fail to realize that once metrics are introduced, making the numbers look good becomes one of the goals. This has always been true.

If you evaluate developers by the number of Pull Requests, number of bugs fixed, number of LLM tokens, or lines of code, developers will respond by making the numbers look better—for example, deliberately making the system worse, splitting a feature that could be solved in one Pull Request into 10, or building an automation tool just to have AI endlessly generate meaningless code. When measurement becomes the goal, it is no longer a good measure.

The more subtle effects even show up in day-to-day development. If these proxy metrics are used to evaluate performance, it sends the message that helping teammates is discouraged, because helping others means sacrificing your own PR count. It also removes any motivation to make the product better.

Important Metrics Are Hard to Measure

The metrics that truly matter are hard to measure. Because performance is usually tied to numbers, if you can’t produce numbers, then you can’t explain the impact you’ve made.

Developer output is hard to quantify, so for convenience we use obvious proxy metrics like Pull Request count, LLM token count, and bug count. Output seems to increase, engineering teams seem busier, but the product doesn’t seem any better.

One line from The Phoenix Project has stuck with me: what matters is the result, not the amount of work. The key is why we are measuring, not what we are measuring.

Yes, what are we measuring for, anyway? Proxy metrics are so convenient that, before we know it, we set aside the things that truly matter.

So what matters when it comes to team productivity? We can get some clues from Google’s Project Aristotle. This was an internal study Google began around 2012 and made public in 2015, analyzing hundreds of teams to find out what makes teams productive. They found that who the team members are mattered far less than how the team interacted. Specifically, the most important factors were ranked as follows:

1. Psychological safety: members can take risks, show vulnerability, and admit mistakes without fear of humiliation or retaliation
2. Reliability: delivering on time and meeting quality standards
3. Role clarity: roles, goals, and plans are clear
4. Meaning: the work is meaningful to individuals
5. Impact: believing the work makes a real contribution

These five factors are difficult to quantify. Psychological safety, for example, may show up in questions like:

• Can I ask questions easily in Slack without worrying about getting scolded?
• Can I ask a senior developer for help and get all the information I need?
• During my first week on the job, do I clearly know what I need to do next?

These indicators are hard to measure, but they are highly important factors that affect team output. They are things numbers cannot tell you. And because they are hard to measure, they are also hard to notice.

In one meeting, I discussed with upper management the purpose of estimation and what outcomes we expected from it.

During the meeting, I found that the two sides had very different ideas about the purpose of estimation.

I thought estimation was mainly for aligning requirements and making sure both sides shared the same understanding of implementation. It should not be used as a team output metric. Upper management, however, believed estimation was for evaluating whether developers were improving, planning future schedules, and understanding team output as a metric. Once the estimation system was introduced, the team felt pressure and obligation. If we gave a longer estimate, upper management would question why it would take so long; if we gave a shorter estimate but failed to meet it, upper management would scold us for not sticking to the estimate.

The team became afraid to estimate casually, worried that giving an estimate meant committing to a release date. They also became less willing to align requirements proactively, because they feared that differing understandings would lead to rework. We only wanted to understand the team’s output—so why did things get worse instead?

When measurement becomes the goal, it is no longer a good measure.

Conclusion

As developers, the benefits of metrics are undeniable. However, when measurement becomes the goal, it is no longer a good measure. This is especially true for those in positions of authority: you are usually the one defining the metrics, while the people responsible for achieving them are different. That easily leads to problems of mismatch between authority and responsibility.

Measurement is a powerful tool. Only by understanding its limits can we use it well. Beyond the numbers themselves, the things outside the numbers—the ones that are hard to quantify—may be what we should pursue even more.