When your service talks to external APIs (payment processors, shipping providers, user directories), your tests need to make decisions based on those APIs’ responses. You can’t call the real endpoint in CI. So you reach for fixtures.
The two instincts most teams have, writing them by hand or grabbing a real response from production, both create problems that compound over time. This guide walks through what those problems are, the three main approaches to solving them, and what “realistic” actually requires of a test fixture.
The hand-written fixture problem
Someone writes a JSON blob from memory or a quick copy-paste, commits it, and the tests pass. Eighteen months later, the upstream API has added required fields, changed value formats, and introduced new enum members. That fixture quietly lies. Tests stay green. Production breaks.
Worse: hand-written fixtures are written by people who already understand the happy path. Edge cases rarely make it into manually crafted data: empty arrays where you expected at least one element, null where you expected a string, numeric IDs that exceed 32-bit range, unicode in unexpected fields. The bugs those edge cases would surface are exactly the ones that reach production.
The drift problem is subtle. You update the fixture when you update the code, so the fixture only represents the response shape your code currently handles. A realistic fixture also includes the things your code gets wrong.
The production data problem
The obvious fix: grab a real response from production and commit it. Now your fixture is realistic. It’s also a compliance liability.
A production response body may contain:
- Personal names, email addresses, physical addresses
- Financial amounts tied to a real account
- Device identifiers, session tokens, or IP addresses
- Fields your schema doesn’t explicitly model, meaning surprise PII from a third party
Even if you strip the obvious fields, you’re making a judgment call about what counts as sensitive. That judgment can be wrong. The consequence is a test file containing real customer data, often committed to a repository accessible to far more people than production.
For GDPR-regulated businesses, this isn’t a theoretical concern. Processing personal data in a test environment is a lawful basis problem. The data was collected for a specific purpose; re-using it across developer laptops and CI pipelines is not that purpose.
Three approaches
1. Hand-written mocks
You write fixture JSON by hand, based on API docs or the shape of a real response you’ve scrubbed manually.
Strengths: Zero infrastructure. Fast to get started. No data leaves your system.
Limits: Fixtures drift from reality. Edge cases are underrepresented because people write what they think about, not what production actually sends. Maintenance burden grows with API surface. When an upstream API ships a breaking change, your tests don’t notice until something breaks in production.
This approach works well enough for narrow, stable APIs where you control both sides. It falls apart when you’re integrating with external services you don’t own, or when the API surface is large enough that manual maintenance becomes a full-time job.
2. Synthetic data generators
Tools like Faker, factory_boy, or custom generators produce structurally valid but fully invented values. You define the schema; the generator produces plausible data.
Strengths: No real data at all. Randomization can surface edge cases that hand-written fixtures miss. Pairs well with property-based testing frameworks.
Limits: The generator only knows what you tell it. If your schema says email: string, the generator produces valid email strings, but it doesn’t know that 8% of your production traffic includes email addresses with a dot immediately before the @, or that one customer’s name contains a zero-width joiner that breaks your PDF renderer. Synthetic data is constrained by your current understanding of the problem space. The unknown unknowns stay unknown.
There’s a useful distinction: a Faker email address is a valid email address. A real email address that your code fails to handle is also a valid string; it just happens to trigger a bug you don’t know about yet. Syntactic validity isn’t the same as production coverage.
Synthetic generation is the right call when your APIs are well-specified, your main concern is exercising your own logic rather than upstream API behavior, and you can express the interesting edge cases as explicit generator rules.
3. Masked captures from real traffic
You instrument your service to record real API request/response pairs as they occur. Before that data leaves your infrastructure, masking rules run: sensitive fields are replaced with shape-preserving placeholders. Strings become "<masked>" by default, or format-preserving synthetic values if the opt-in salt is configured; numbers become 0; booleans become false. The masked payloads are stored and replayed in tests.
Strengths: Fixtures reflect what actually happens, including edge cases you didn’t anticipate. If a customer with an unusual input hits your endpoint today, that shape is captured. If an upstream API starts returning a new optional field, you’ll have it in your fixtures before your code needs to handle it. Coverage grows automatically with production traffic.
Limits: Requires instrumentation: you need to deploy the capture middleware to a production or realistic staging environment. There’s a cold-start period before you’ve captured enough traffic for good coverage. The masking rules must be correct; misconfigured masking is the main failure mode.
What “realistic” actually requires
“Realistic” isn’t just about data volume or plausible-looking values. A fixture that a senior engineer would trust needs to satisfy three properties:
Schema validity with production-observed values. Not just values that are valid according to your understanding of the schema, but values that production traffic actually produces. This includes values at boundary conditions, values with unexpected encodings, and values your schema doesn’t enumerate.
Representative edge cases. Empty arrays, null fields, missing optional keys, maximum-length strings, numeric values at type boundaries. These appear at low frequency in production but cause a disproportionate share of bugs. Fixtures from real traffic capture them at their natural rate; hand-written fixtures almost never include them until after a production incident.
Shape stability. A fixture that changes structure between test runs without a corresponding code change is noise. Useful fixtures maintain a stable structural signature (method, path, response schema) across updates, so a test failure means something real changed.
How masked capture works in practice
The core invariant is that masking happens at the edge, inside your own infrastructure, before anything is transmitted elsewhere. The sequence looks like this:
- Your service handles a request normally.
- A middleware intercepts the request/response pair before logging.
- Your masking rules run: fields you’ve marked sensitive are replaced in-place. The shape is preserved; the values are gone.
- Only the masked payload crosses the network boundary; raw values are not transmitted.
This is what makes masked capture viable for GDPR-regulated data: you’re not moving personal data to a third-party service. You’re moving a structural shadow of the data. The shadow is sufficient for generating realistic test fixtures: the field names, nesting, types, and edge-case structures are all there. The values that would identify someone are not.
Stubsmith implements this approach: an Express middleware captures traffic, your masking configuration specifies which fields to blank, and the SDK applies masking before any payload leaves your infrastructure. Stubsmith’s servers receive masked bodies and structural fingerprints, never raw values. Fingerprinting deduplicates structurally identical traffic shapes, so you accumulate one canonical example of each unique request/response shape rather than thousands of near-identical copies.
Choosing an approach
If your APIs are stable and you control both sides: hand-written mocks with careful maintenance discipline work. The cost is proportional to how much the API changes.
If you’re writing a library and want to exercise your logic against a large range of inputs: synthetic generation with property-based testing is the right tool. The cost is the gap between your generator model and production reality.
If you’re integrating with external APIs you don’t control, operating under data minimization obligations, or tired of fixing tests that were wrong all along: masked capture earns its instrumentation overhead. The test suite grows more honest as traffic flows.
See pricing for plan details if you want to evaluate Stubsmith for the masked-capture approach. The free plan covers most solo and small-team workflows.
The goal in any case is the same: a test suite that fails when real things are broken, not one that passes because the fixtures were written by someone who already knew what the code does.