The Whitehorse Route: Three Evidence-Gathering Errors That Derail Poaching Cases and How to Correct Them

Evidence gathering in API design is the foundation of every good decision. But in many projects, the evidence collected is incomplete, biased, or just plain wrong. Poaching cases — where one team uses data to argue for a design change that benefits their own service at the expense of the system — are surprisingly common. The result? Misguided endpoints, broken contracts, and performance regressions. This guide walks through three common evidence-gathering errors and shows how to correct them with practical, repeatable methods.

Who Needs This and What Goes Wrong Without It

Every team that designs APIs — whether for internal microservices or external consumers — relies on evidence to make decisions. But the evidence itself is often gathered haphazardly. A team might collect latency data for one endpoint, ignore error rates, and then claim the endpoint is fast. Another team might sample only peak-hour traffic and conclude that a new caching layer is unnecessary. These are poaching cases: selective evidence used to push a predetermined agenda.

Without a disciplined approach to evidence gathering, the following problems emerge:

• Confirmation bias — teams seek out data that supports their preferred design, ignoring contradictory signals.
• Incomplete coverage — metrics are collected from only a subset of users, endpoints, or time periods, leading to skewed conclusions.
• Misaligned metrics — teams measure what's easy rather than what matters, such as tracking average latency instead of p99 tail latency.

These errors derail poaching cases because the evidence is too weak to withstand peer review or stakeholder scrutiny. A design change based on flawed evidence often gets rolled back, wastes engineering time, and erodes trust in the team's judgment.

This guide is for API architects, platform engineers, and product managers who need to defend design decisions with data. By the end, you'll be able to spot each error before it infects your evidence gathering, and you'll have a set of corrective practices to apply immediately.

When Evidence Gathering Goes Wrong: A Composite Scenario

Consider a team that wants to add a new aggregation endpoint to an existing API. They gather evidence by monitoring the current endpoints for two days and find that average response time is 50 ms. They conclude the API is fast enough to support the new endpoint without optimization. But they only monitored during business hours, excluding batch jobs that run overnight. Those batch jobs cause latency spikes to 2 seconds, which the new endpoint would amplify. The poaching case succeeds initially, but the endpoint causes cascading timeouts in production. A broader evidence set would have revealed the risk.

Prerequisites and Context Readers Should Settle First

Before you start gathering evidence for an API design decision, you need a clear understanding of your system's observability infrastructure. This includes logging, metrics, tracing, and the tools you use to query them (e.g., Prometheus, Datadog, OpenTelemetry). Without reliable instrumentation, any evidence you collect is suspect from the start.

You also need a shared definition of what constitutes valid evidence within your organization. Some teams accept anecdotal observations from a single developer's local testing; others require statistically significant samples from production traffic. Setting these expectations upfront prevents arguments later about whether the evidence is sufficient.

Another prerequisite is a clear hypothesis. What exactly are you trying to prove? For example: "Adding a /batch endpoint will reduce overall client latency by 30% because it reduces the number of round trips." A well-defined hypothesis makes it easier to choose the right metrics and avoid cherry-picking.

Finally, ensure you have access to historical data. A single snapshot of current performance is rarely enough. You need baseline data from before any changes, as well as data that covers different load patterns, such as weekends, holidays, and deployment windows. Without historical context, you can't distinguish normal variation from the effect of your proposed change.

Common Pitfalls in Setting Up Evidence Gathering

One common mistake is skipping the baseline. Teams often start monitoring only after they've already decided to make a change. This makes it impossible to compare before and after. Another pitfall is using different measurement tools for different endpoints, which introduces inconsistencies. Standardize on one observability stack and ensure all endpoints are instrumented identically.

Also, be wary of relying solely on aggregate metrics. Averages hide outliers. For API performance, tail latency (p99, p99.9) is often more informative than the mean. Similarly, error rates should be broken down by error type (4xx vs. 5xx, timeouts vs. connection resets) to understand the root cause.

Core Workflow: Sequential Steps for Robust Evidence Gathering

To correct the three evidence-gathering errors, follow this three-phase workflow: Define, Collect, Validate.

Phase 1: Define the Evidence Requirements

Start by writing down exactly what evidence you need to support your design decision. Use the SMART criteria: specific, measurable, achievable, relevant, time-bound. For example: "We need evidence that the current /users endpoint has a p99 latency under 200 ms over a 7-day period, including at least two peak traffic events." This definition forces you to confront what "good enough" looks like and prevents vague claims.

Next, list all alternative explanations that could undermine your case. If you're arguing that a new endpoint is faster, what else could explain the speedup? A recent infrastructure upgrade? A drop in overall traffic? Addressing these alternatives in advance makes your evidence more robust.

Phase 2: Collect Data Systematically

Set up data collection to run for a predetermined period — typically at least one full business cycle (e.g., a week). Use automated scripts to pull metrics from your monitoring system at regular intervals. Avoid manual snapshots, which are prone to selection bias.

Collect data from all relevant endpoints, not just the ones you're changing. This provides a control group. For example, if you're modifying the /orders endpoint, also collect data from /products and /customers to see if any changes are systemic.

Include metadata with each data point: timestamp, request method, response status, client type, and deployment version. This allows you to slice the data later and check for confounding factors.

Phase 3: Validate the Evidence

Before presenting your evidence, run it through a validation checklist:

• Is the sample size large enough? (A rule of thumb: at least 1,000 requests per endpoint for latency metrics.)
• Does the data cover both normal and extreme conditions? (Include error responses, not just successful ones.)
• Are there any gaps in the data? (Missing time ranges, excluded endpoints, dropped traces.)
• Can the results be reproduced by another team member? (Share the raw data and query scripts.)

If any check fails, go back to Phase 2 and extend the collection period or broaden the scope. Do not skip this step; it's where most poaching cases are caught.

Tools, Setup, and Environment Realities

The tools you choose can make or break your evidence-gathering process. Here are three commonly used stacks and their trade-offs:

Whichever stack you choose, ensure it captures both metrics and traces. Metrics tell you what happened (e.g., latency increased), while traces tell you why (e.g., a database query slowed down). For a poaching case, you need both levels of detail.

Environment Considerations

Production is the ideal environment for evidence gathering, but not always accessible. If you must use staging, replicate production traffic patterns as closely as possible. Use traffic replay tools like GoReplay or VCR to feed real requests into your staging environment. Even then, note that staging hardware and concurrency differ — so treat staging evidence as indicative, not definitive.

Also, be aware of the observer effect: when you start collecting detailed metrics, the instrumentation itself can affect performance. For example, enabling verbose logging on a critical endpoint may increase latency. Mitigate this by sampling a subset of requests or using low-overhead instrumentation libraries.

Variations for Different Constraints

Not every team has the luxury of running a full-week evidence-gathering campaign. Here are variations for common constraints:

Limited Time

If you only have 24 hours, focus on the most critical endpoint and collect data during peak traffic hours. Use percentiles (p95, p99) instead of averages to get a more accurate picture in a short window. Accept that your evidence will have higher uncertainty and flag this when presenting.

Limited Access to Production

When production access is restricted, use client-side monitoring. Instrument the API client (e.g., a mobile app or web frontend) to capture latency and error rates from the user's perspective. This evidence is often more persuasive because it reflects real user experience.

Small Team or Startup

If you're a two-person team, automate as much as possible. Use simple scripts that run on a schedule and output to a shared spreadsheet. Don't overinvest in fancy dashboards — focus on collecting raw data that can be analyzed later. A simple CSV file with timestamps and response times is often enough to support a poaching case.

High Security / Compliance

In regulated industries, evidence gathering must be auditable. Ensure all data collection is logged, with timestamps and user IDs. Use immutable storage for raw metrics (e.g., append-only logs in S3). This prevents accusations of data tampering and satisfies compliance requirements.

Pitfalls, Debugging, and What to Check When It Fails

Even with a solid workflow, evidence gathering can fail. Here are the most common pitfalls and how to debug them.

Pitfall 1: The Evidence Is Too Noisy

If your data shows high variance and no clear pattern, the problem is often insufficient granularity. For example, if you're measuring latency per endpoint but not per HTTP method, the mix of GET and POST requests could be hiding the real story. Break down metrics by method, status code, and client version. Also check if your monitoring system is aggregating data incorrectly (e.g., averaging over 5-minute windows when you need 1-second resolution).

Pitfall 2: The Evidence Contradicts Your Hypothesis

This is actually good — it means your process is working. Don't discard the contradictory data; instead, investigate why. Perhaps your hypothesis was wrong, or there's a confounding factor you missed. For example, you might find that latency increased after your change, but only for users in a specific region due to a CDN misconfiguration. Document the contradiction and update your hypothesis accordingly.

Pitfall 3: Stakeholders Reject the Evidence

Sometimes the evidence is solid, but stakeholders don't trust it. This often happens because the evidence was collected by the same team advocating for the change. To avoid this, involve a neutral third party (e.g., a platform team or QA) in the data collection process. Also, share the raw data and methodology openly so anyone can reproduce the results.

Debugging Checklist

When your evidence fails to convince, run through this checklist:

1. Did you define success criteria before collecting data? If not, redo the process with clear criteria.
2. Is the sample size statistically significant? Use a sample size calculator (e.g., for a 5% margin of error at 95% confidence, you need around 400 requests for a binary outcome).
3. Did you control for confounding variables? Check if other changes were deployed during the collection period.
4. Did you include error cases? Evidence that only shows successful requests is incomplete.
5. Is the data collection period long enough to capture natural variation? A single day is rarely sufficient.

After debugging, you'll likely need to run the collection again with adjustments. This is normal — evidence gathering is an iterative process, not a one-shot activity.

To solidify your new skills, take these three actions next: (1) Review your last API design decision and reconstruct the evidence used to support it — identify any of the three errors. (2) Set up a standardized evidence-gathering template for your team, including baseline requirements and a validation checklist. (3) For your next design proposal, run a pilot evidence collection for one endpoint and ask a colleague to audit it before presenting. This builds a culture of rigorous evidence gathering that makes poaching cases harder to push through.