Running tests in CI against live third-party APIs creates three problems at once: your tests can fail because the external service is down, they can fail because you’ve hit a rate limit, and they require production credentials to be present in your CI environment. Any one of those is a good reason to stop doing it.
Replaying Stubsmith stubs solves all three simultaneously. The Python SDK intercepts outbound HTTP calls inside your test process and returns recorded responses from a bundle file. Tests see the same shapes they’d get from the live API. No outbound calls. No server to start. No credentials needed at test time.
Why replay beats hand-written mocks
When you write API mocks by hand, the fixture reflects your understanding of the upstream API at the time you wrote it. That understanding is accurate until the upstream API changes, and it does change without telling you, until something breaks in production.
Stubsmith fixtures are derived from real traffic. When an upstream API starts returning a new optional field, you’ll have it in your captured fixtures before your code needs to handle it. When a response shape changes, the fingerprint dedup surfaces the new shape as a distinct fixture. Tests that depend on the old shape break in CI rather than in production.
The other gain is edge case coverage you didn’t plan for. An unusual customer input that hit your endpoint last week is now a fixture. A 429 response shape from a rate-limited upstream is a fixture. Edge cases accumulate in your fixture store automatically as production traffic flows, without anyone having to anticipate them.
How in-process replay works
stubsmith.replay() is a context manager. When active, it patches the requests library’s transport layer. Each outbound call is fingerprinted the same way capture did, matched against the bundle, and answered from the recorded response. No network call is ever made.
import stubsmith
def test_checkout():
with stubsmith.replay():
result = my_app.checkout(cart) # uses requests internally
assert result.order_idThe with block restores normal transport on exit, including after exceptions. For unittest.TestCase, use the explicit start() / stop() pair:
import unittest
import stubsmith
class CheckoutTest(unittest.TestCase):
def setUp(self):
self._replay = stubsmith.replay()
self._replay.start()
def tearDown(self):
self._replay.stop()
def test_order_lookup(self):
resp = self.client.get_order("ORD-99")
self.assertEqual(resp.status_code, 200)
self.assertIn("order_id", resp.json())Currently requests is the supported transport. Calls made through httpx or any other library are not intercepted, and they attempt real network connections. In a CI environment without network egress or third-party credentials, those calls will fail. If your application uses a library other than requests, replay gives no coverage for it.
What happens on a miss
A request that has no matching stub raises stubsmith.StubNotFound. The error message names the closest recorded stub and lists which request fields differ, so you know exactly what to fix or re-pull:
stubsmith.replay.StubNotFound: no recorded stub for POST /api/checkout (fingerprint 0eed7242949f0e58)
Closest recording 8b96d0f0140e1798 (seen 88x):
+ coupon sent, not in the recording
refresh the bundle: stubsmith pullThere is no passthrough or permissive mode. A miss is always an error in the current release.
The bundle
The bundle is the file stubsmith pull writes to .stubsmith/bundle.json. It contains all recorded stubs for your project, sorted deterministically so repeated pulls produce no spurious diff when nothing changed on the server.
Flags:
--out PATH: write to a different location (default:.stubsmith/bundle.json)--endpoint "METHOD /path/template": filter to a single endpoint
Environment variables:
STUBSMITH_API_KEY: required, your project bearer tokenSTUBSMITH_API_URL: override the backend URL (falls back toSTUBSMITH_BACKEND_URL, thenhttp://localhost:3000)
The bundle is committed to your repository. This is the key design decision: once the bundle is committed, CI needs no API key to run tests. The bundle is the source of truth for the test run, checked out with the rest of the code.
A concrete GitHub Actions workflow
The workflow below is valid GitHub Actions syntax. Check out the code (the bundle comes with it), install the SDK, and run tests. That is the entire setup.
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: pip install stubsmith pytest
- name: Run tests
run: pytestNo API key. No server step. No readiness poll. The bundle is in the repository, so the workflow is self-contained.
Keeping the bundle current
The bundle was committed at some point, which means it can go stale as your upstream APIs evolve. The recommended pattern is a scheduled workflow that pulls a fresh bundle and opens a pull request when it changes:
name: Refresh stubs
on:
schedule:
- cron: '0 6 * * 1' # every Monday at 06:00 UTC
workflow_dispatch:
jobs:
refresh:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install SDK
run: pip install stubsmith
- name: Pull bundle
run: stubsmith pull
env:
STUBSMITH_API_KEY: ${{ secrets.STUBSMITH_API_KEY }}
- name: Open PR if bundle changed
# Third-party action, pin to a commit SHA in production use.
# Requires "Allow GitHub Actions to create and approve pull requests"
# enabled under Settings → Actions → General → Workflow permissions.
uses: peter-evans/create-pull-request@v6
with:
commit-message: 'chore: refresh .stubsmith/bundle.json'
title: 'Refresh Stubsmith bundle'
branch: chore/refresh-stubsThe permissions: block is required: GITHUB_TOKEN defaults to read-only in many organisation and repository configurations, and without contents: write and pull-requests: write the step will either fail with a 403 or silently create nothing. You also need to enable “Allow GitHub Actions to create and approve pull requests” in your repository’s Settings → Actions → General → Workflow permissions; that setting is off by default in many organisations and no amount of correct YAML overrides it.
If you prefer not to depend on a third-party action, you can replace that last step with a direct gh pr create call:
- name: Open PR if bundle changed
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b chore/refresh-stubs
git add .stubsmith/bundle.json
git diff --cached --quiet || (
git commit -m 'chore: refresh .stubsmith/bundle.json' &&
git push -f origin chore/refresh-stubs &&
gh pr create --title 'Refresh Stubsmith bundle' \
--body 'Automated bundle refresh. Review the diff on .stubsmith/bundle.json.' \
--base main --head chore/refresh-stubs || true
)
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}Either approach needs the same permissions: block.
The pull request’s diff is .stubsmith/bundle.json, the exact set of stubs that changed since the last pull. Reviewing that diff is your API-drift review: new fields, new enum values, new error shapes all appear explicitly. If the diff shows something unexpected, you know before tests run, not after.
What you’ve eliminated
A CI workflow running Stubsmith stubs requires one secret: the Stubsmith API key, and only in the scheduled refresh job, not in the main test run. It does not require:
- API keys or tokens for any third-party service your application calls
- A staging environment with live credentials configured
- Network access to any external API during the test run
- A stub server process to start, health-check, or tear down
- Any production or near-production data present in the CI environment
The third-party credentials that previously had to live in your CI secret store are no longer needed for test runs. The test suite runs in a fully isolated environment against stubs derived from real traffic, traffic that was masked before it ever left your own infrastructure.