When a Measure Becomes a Target: From the Window Tax to Pull Request Counts
Introduction
At a previous job, I once wrote a small tool to track how many Pull Requests I (and my teammates) contributed that quarter, how many review comments I left, and how many tickets I resolved, hoping to use this data to prove my output and contributions to my manager.
After reviewing my report, my manager calmly told me that performance evaluations aren’t just about output; they also take into account impact and the kind of value you bring to the table. I felt a bit frustrated at the time, thinking that despite doing so much, I wasn’t getting the recognition I deserved.
A bit annoyed, I asked, “Are you saying it’s fine if I just slack off and don’t produce any output?” He didn’t give me a direct answer. Unsurprisingly, I didn’t get the performance rating I had hoped for.
Years later, I finally understood what he meant, and I’m even grateful that he didn’t just praise my output. When a measure becomes a goal, it loses its meaning as a measurement.
When Metrics Go Wrong
In 1696, England introduced a tax on windows. The more windows a house had, the higher the tax. Generally, wealthier households had more windows, making it easy for tax collectors to count them from the street without entering the home.
To reduce their taxes, people across the country bricked up their windows one by one, and newly built houses were designed with no windows at all. Epidemics were rampant in England at the time; the lack of windows led to poor ventilation, making people even more susceptible to contagious diseases.
Ancient Egyptians built Nilometers along the Nile River, using the annual flood height to set that year’s tax rates. The water level was an objective natural metric that was impossible to fake, yet tax evasion didn’t vanish.
Since they couldn’t alter the tax rate, farmers underreported their cultivated land, bribed the scribes surveying the fields, or mixed stones into grain deliveries to make weight. Consequently, the state had to deploy officials to watch over every step of harvesting and transporting grain. With high tax rates yielding diminishing returns, farmers naturally lost any incentive to increase production.
In Hanoi, Vietnam, in 1902, the colonial government offered a bounty for rat tails to eradicate the rodent problem. The result? People simply severed the tails and released the rats back into the wild, while others even started farming rats. Rather than solving the plague, the infestation only grew worse.
When a measure becomes a target, it ceases to be a good measure.
Why Are Proxy Metrics So Seductive?
Metrics are undeniably crucial. In software engineering, there is a concept known as “Observability.” Making numbers visible is a vital part of software development; with data, we know how to improve systems and fix errors.
- A sudden spike in database connections: A poorly written query might be causing slow queries that hog all available connections.
- Excessive CPU usage: A massive increase in data volume might be driving computation costs through the roof.
- A noticeable surge in API response times: A third-party dependency might be experiencing an outage.
Measurement is an essential tool for identifying problems and improving products. The issue is that many corporate leaders have only a superficial understanding of metrics, failing to realize that once you begin measuring an indicator, making the numbers look good becomes an objective in itself. This has been true throughout history.
If you evaluate a developer’s output by PR count, resolved bug count, LLM token usage, or lines of code, developers will adapt their behavior to optimize those numbers—such as intentionally writing worse code, breaking a feature that could be solved in one PR into ten, or building an automated tool to make AI churn out endless meaningless code. When a measure becomes a target, it ceases to be a good measure.
The more subtle consequences ripple into everyday development. Evaluating performance with these proxy metrics discourages teammates from helping one another, because assisting a colleague means sacrificing your own PR count, leaving no incentive to actually make the product better.
What Truly Matters Is Hard to Measure
What truly matters is notoriously hard to measure. Since performance evaluations are usually tied to quantifiable numbers, lacking metrics makes it seem as though you cannot articulate your impact.
A developer’s output is difficult to quantify, so out of convenience, people resort to obvious proxy metrics like PR counts, LLM tokens, and bug counts. Output appears to increase, and the engineering team gets busier, but the product doesn’t seem to get any better.
A line from The Phoenix Project left a deep impression on me: what matters is the outcome, not the amount of work. The key is knowing why we measure, not just what we measure.
Indeed, why are we measuring in the first place? Proxy metrics are so convenient that we easily end up sidelining what actually matters.
When it comes to team productivity, what actually matters? We can glean some insights from Google’s Project Aristotle.
This was an internal study initiated by Google around 2012 and published in 2015. They analyzed hundreds of teams to discover what makes a team effective. They found that who is on a team matters far less than how team members interact. Specifically, they identified the following factors, ranked in order of importance:
- Psychological Safety: Team members feel safe to take risks, show vulnerability, and admit mistakes without fear of embarrassment or retribution.
- Dependability: Delivering on time and meeting quality standards.
- Structure and Clarity: Clear roles, goals, and execution plans.
- Meaning: Work has personal significance to each member.
- Impact: Believing that one’s work creates meaningful change.
These five factors are hard to quantify. Psychological safety, for instance, might show up as:
- Can I ask questions freely in Slack without fear of being reprimanded?
- Can I reach out to senior engineers and get the information I need?
- During my first week on the job, do I clearly understand what tasks are expected of me next?
These indicators are difficult to measure, yet they are crucial factors that drive team output—things that cold numbers can never reveal. And because they are hard to measure, they are easily overlooked.
In a past meeting, I discussed the purpose and expected outcomes of time estimation with leadership.
During the discussion, I realized there was a huge gap between our perceptions of why we estimate.
I believed estimation was primarily for aligning requirements and ensuring both sides shared the same understanding of implementation; it should not be used as an output metric for the team. Leadership, on the other hand, saw estimation as a way to evaluate whether developers were improving, schedule upcoming timelines, and gauge team output. When the estimation system was introduced, the team felt pressure and binding commitments. If an estimate was too long, leadership questioned why it took so much time; if an estimate was short and missed, leadership scolded them for not sticking to the schedule.
The team became hesitant to provide estimates, terrified that an estimate would be taken as a hard release date. They also hesitated to proactively clarify requirements, fearing that differing interpretations would force rework. Management simply wanted to understand team output, so why did things only get worse?
When a measure becomes a target, it ceases to be a good measure.
Conclusion
As developers, the benefits of metrics are undeniable. However, when a measure becomes a target, it ceases to be a good measure. This is especially true for managers and leaders, who are usually the ones defining the metrics. When the people setting the metrics are different from those tasked with hitting them, you easily run into the trap of an imbalance between authority and responsibility.
Measurement is a powerful tool; only by understanding its limitations can we make good use of it. Beyond the numbers themselves, the intangible things outside the data might be what is truly worth pursuing.
Related Posts
- Using Cloudflare Images for Image Storage and Transformation Putting an image on a webpage is the simplest task in frontend development. But doing it properly—including resizing, generating multiple formats, and withstanding heavy traffic—is actually an entire end-to-end solution. Eventually, I offloaded everything to Cloudflare Images, keeping only a single original image.
- Stop Using AWS Access Keys Access Keys are an easily overlooked security risk in AWS. By pairing OIDC with IAM Roles, GitHub Actions can securely operate AWS resources without storing any secrets.
- Database Primary Keys: AUTO_INCREMENT, UUID, and UUIDv7 Backend developers often face the choice of primary keys: should you use auto-increment or UUID? What about collisions? How does UUIDv7 compare to created_at + index in performance? Here are the design decisions and benchmark results from testing 20 million rows.
- My Experience with Zeabur: A Hands-on Review Most indie developers turn to platforms like Vercel to deploy their services. But when it comes to more advanced requirements like database connections, Vercel becomes less convenient, and traditional cloud providers are often too expensive for indie development. In this article, I share my experience using Zeabur and why I recommend it!