Testing Guide¶
This guide covers testing strategies, IDE setup, and debugging for the Freeze Design webshop.
The backend uses pytest 9.2 with pytest-django, pytest-cov, pytest-xdist and factory-boy; its defaults live in backend/pyproject.toml and backend/.coveragerc. The frontend uses Jest 30.5 with Testing Library and jest-environment-jsdom, configured by frontend/jest.config.js. Browser tests use Playwright 1.64 through frontend/playwright.config.ts. Load tests use k6 scripts in loadtests/.
Run ./manage.sh setup once, or install backend dependencies from backend/requirements.txt into backend/venv and frontend dependencies with npm install in frontend/. Install Playwright's browsers and system dependencies with npx playwright install --with-deps when they are absent.
Test Pyramid¶
┌───────────┐
│ E2E │ ← Few, slow, high confidence
├───────────┤
│Integration│ ← Medium number
├───────────┤
│ Unit │ ← Many, fast, focused
└───────────┘
Backend Testing (Django)¶
Backend tests run on pytest (with pytest-django, pytest-xdist, and factory-boy) — this is what CI and the pre-push hook run and what you should run locally. python manage.py test still works (./manage.sh ci-check uses it) but is not the configured path. The tests need PostgreSQL on 5432 and Redis on 6379, e.g. via docker compose -f docker-compose.dev.yml up -d db redis.
Run Tests¶
cd backend
pytest
# Run specific app tests
pytest apps/payments
pytest apps/invoices
pytest apps/orders
# Run specific test modules
pytest apps/payments/tests/test_webhook.py
pytest apps/payments/tests/test_refund_service.py
pytest apps/invoices/tests/test_invoice_service.py
# Run a single test by name
pytest -k "test_webhook_marks_order_paid"
# Extra verbosity
pytest -vv
# One process, no coverage (faster for a small subset; this is what the pre-push hook does)
pytest apps/pricing/tests.py -o addopts="" -q
With Coverage¶
Coverage is switched on in backend/pyproject.toml (addopts): parallel (-n auto --dist=loadscope) and a term-missing report. What is measured and the 75% fail-under threshold (lines and branches together) live in backend/.coveragerc. Running pytest bare already applies all of this — CI runs the same command and combines the per-worker coverage data afterwards. testpaths is tests, apps and config, so the configuration tests in backend/config/test_checks.py run too.
# Same as CI (coverage via pyproject addopts, fail-under 75% from .coveragerc)
pytest
# HTML coverage report
pytest --cov-report=html # open htmlcov/index.html
Markers: integration (needs all services) and slow (> 2 seconds) are registered in pyproject.toml.
Test Structure¶
# tests/test_products.py
from django.test import TestCase
from rest_framework.test import APITestCase
class ProductAPITests(APITestCase):
def setUp(self):
self.product = Product.objects.create(
name='Test Product',
base_price=29.95
)
def test_list_products(self):
response = self.client.get('/api/products/')
self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.data['results']), 1)
def test_product_detail(self):
response = self.client.get(f'/api/products/{self.product.slug}/')
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data['name'], 'Test Product')
Plain pytest functions with the fixtures from backend/conftest.py and the per-app factories.py (orders, payments, products, designs, quotes, users) are equally fine; match the style of the surrounding tests.
Frontend Testing¶
Unit Tests (Jest)¶
cd frontend
npm run test
npm run test:coverage
npm run test:watch # Watch mode (auto-rerun)
npm test CartContext # Specific test file
Coverage thresholds are enforced in frontend/jest.config.js (branches 54%, functions 59%, lines 61%, statements 60%) and are raised incrementally as coverage improves. Coverage is collected from app/, components/, lib/, contexts/ and hooks/. CI runs npm run test:ci (jest --watchman=false --coverage --forceExit).
Test Structure¶
// Component.test.tsx
import { render, screen, fireEvent } from '@testing-library/react';
import { QuantitySelector } from './QuantitySelector';
describe('QuantitySelector', () => {
it('increments quantity when plus clicked', () => {
const onChange = jest.fn();
render(<QuantitySelector quantity={1} onQuantityChange={onChange} />);
fireEvent.click(screen.getByLabelText('Verhoog aantal'));
expect(onChange).toHaveBeenCalledWith(2);
});
it('disables decrement at minimum', () => {
render(<QuantitySelector quantity={1} onQuantityChange={() => {}} min={1} />);
expect(screen.getByLabelText('Verlaag aantal')).toBeDisabled();
});
});
Running Specific Tests¶
# Single test file
npm test CartContext.test.tsx
# Single test case
npm test -- -t "should add item to cart"
# All tests matching pattern (Jest 30: --testPathPatterns, plural)
npm test -- --testPathPatterns=contexts
E2E Testing (Playwright)¶
Prerequisites¶
Playwright talks to a real backend. Start it with E2E_TESTING=True (disables throttling, django-axes and mandatory email verification; never set it outside DEBUG) and a migrated database. Without E2E_BASE_URL or CI set, playwright.config.ts starts npm run dev on port 3000 itself (reuseExistingServer); set E2E_BASE_URL to point at a stack you manage yourself.
Projects¶
Defined in frontend/playwright.config.ts:
| Project | Runs | Notes |
|---|---|---|
chromium |
All plain specs | What e2e-smoke-test and e2e-full-test use in CI; excludes admin/seeded and visual specs |
chromium-admin |
Seed-dependent specs (admin-*, cart-mobiel, collectie, designer-order, quote-design, multi-zone-views, personalisatie, and designer/clipart-tools for the seeded clipart) |
Depends on the setup project (admin.setup.ts: one TOTP admin login whose storageState all admin specs reuse); needs ensure_e2e_admin and seed_e2e_extras on the backend. clipart-tools also runs in the plain projects; only the clipart tests skip there |
firefox, webkit, Mobile Chrome, Mobile Safari |
Plain specs in other browsers/viewports | CI runs pdp-hydration.spec.ts in Mobile Safari as part of the smoke job |
visual |
e2e/visual/visual.spec.ts |
Visual regression with reducedMotion: 'reduce'; see below |
On CI: 2 workers and 1 retry by default (the reusable workflow passes its own --workers/--retries).
Run Tests¶
cd frontend
npx playwright test --project=chromium --grep @smoke # deploy gate, fastest
npx playwright test --project=chromium # full plain suite
npm run test:e2e # every project, incl. chromium-admin and visual
npm run test:e2e:headed # See browser while testing
npm run test:e2e:ui # Interactive Playwright UI
npm run test:e2e runs all projects; chromium-admin fails without a seeded admin, so prefer --project=chromium locally.
Running Specific E2E Tests¶
# Single test file
npx playwright test team-order-flow.spec.ts
# Single test case
npx playwright test -g "van productpagina tot winkelwagen"
# By tag
npx playwright test --grep="@smoke"
# Specific browser
npx playwright test --project=firefox
npx playwright test --project=webkit
Tags in use: @smoke (deploy gate), @critical, @journey, @regression, @design, @frontend-only, @payment-integration.
Test Structure¶
// e2e/checkout.spec.ts
import { test, expect } from '@playwright/test';
test.describe('Checkout Flow', () => {
test('complete order', async ({ page }) => {
await page.goto('/products/classic-t-shirt');
await page.click('[data-testid="color-swatch-1"]');
await page.click('[data-testid="size-option-3"]');
await page.click('[data-testid="add-to-cart"]');
await page.click('[data-testid="checkout-button"]');
await page.fill('[name="email"]', 'test@example.com');
await page.click('[data-testid="submit-order"]');
await expect(page.locator('.order-confirmation')).toBeVisible();
});
});
Visual Regression¶
e2e/visual/visual.spec.ts compares screenshots against the baselines in e2e/visual/visual.spec.ts-snapshots/ (*-linux.png, so baselines must come from Linux). The e2e-visual-test job runs them on every PR and push. To refresh the baselines, start the Update Visual Snapshots workflow (update-snapshots.yml, workflow_dispatch, optional branch input): it runs npx playwright test --project=visual --update-snapshots, pushes the new PNGs to vrt/update-baselines-<timestamp> and opens a PR. Locally the same command works on Linux only.
Accessibility¶
e2e/accessibility-audit.spec.ts runs axe (@axe-core/playwright) against the main pages as part of the plain suite.
Quarantine¶
Tests that fail only in the e2e full run while the feature works in production are parked with test.fixme / test.describe.fixme and a // QUARANTINE (e2e-debt): ... comment. The list and the deblocking procedure are in the E2E-quarantaine runbook. The deploy gate (@smoke + visual) is unaffected.
Viewing E2E Results¶
- HTML Report: Auto-opens after test run, or manually:
npx playwright show-report - Trace Viewer: For failed tests:
npx playwright show-trace trace.zip
Load Tests (k6)¶
The k6 scripts live in loadtests/ (product-listing.js, cart-operations.js, design-upload.js, homepage-load.js, checkout-load.js; thresholds and history in loadtests/README.md and loadtests/baseline-results.md).
- Locally:
./manage.sh loadtest [all|products|cart|upload]against the running backend (needsk6installed). - Load Tests workflow (
load-tests.yml, manual): inputstest_suite(all,products,cart,upload,homepage,checkout),base_urlandfrontend_url(defaulthttps://staging.freezedesign.nl). It is a manual re-baselining tool, not a gate. - Staging performance gate (
staging-perf-gate.yml): started bydeploy-staging.ymlafter every successful staging deploy; runsloadtests/homepage-load.jsagainst staging and posts to Discord when it fails. It does not block the deploy.
VS Code Setup¶
Recommended Extensions¶
| Extension | ID | Purpose |
|---|---|---|
| Jest | Orta.vscode-jest |
Inline test results, coverage, run/debug tests |
| Playwright Test | ms-playwright.playwright |
Run/debug E2E tests, record tests, trace viewer |
Install via terminal:
Configure Jest Extension¶
Add to .vscode/settings.json:
VS Code Shortcuts¶
Unit Tests (Jest):
- Hover over test and click play button to run
- Right-click test for "Debug Test"
- Cmd+Shift+P then "Jest: Toggle Coverage"
E2E Tests (Playwright):
- Click play button in gutter next to test
- Right-click for "Debug Test"
- Cmd+Shift+P then "Test: Record New"
Debugging Tests¶
Debug Unit Tests (Jest)¶
Debug E2E Tests (Playwright)¶
CI/CD Integration¶
Tests run automatically via GitHub Actions (.github/workflows/ci.yml) on every push to main/staging (except docs-only pushes, paths-ignore) and on every PR against those branches. CI runs Python 3.12 and Node 22 with PostgreSQL 15 and Redis 7 service containers. See the GitHub Workflow for the gates and the deploy.
CI Jobs on Every PR / Push¶
- precheck — decides what runs: promotion PRs (
staging→main) and docs-only PRs skip the test jobs; a push tostagingwhose git tree was already green in a PR run (tested-tree-<tree>artifact) skips them too - backend-test — flake8 (
E9,F63,F7,F82blocking; complexity/style report with a 127-character limit),makemigrations --check,check_migrations --ci, migrations, full pytest suite (parallel, coverage ≥75%) - frontend-test — TypeScript check, ESLint,
npm run test:ci(coverage thresholds fromjest.config.js) - frontend-build — Next.js production build (artifact reused by the E2E jobs)
- e2e-smoke-test — Playwright
@smoketests inchromium, pluspdp-hydration.spec.tsin Mobile Safari (via the reusablee2e-setup.yml) - e2e-visual-test — Playwright
visualproject - api-types-drift ("API types drift check") — regenerates the frontend API types from the backend OpenAPI schema and fails if the committed types drifted
- nginx-config — PRs only:
scripts/tests/test_nginx_security_headers.sh,test_restart_unhealthy.sh,test_notify_ntfy.sh,test_deploy_rolling.sh,test_werkoverzicht.pyandtest_werkbord.py(unittest) andtest_intake_ready.sh;test_check_done_todos.shruns intodo-done-check.yml - ci-gate — PRs only: green when all of the above succeeded or were skipped; the required check on
staging
Runs only on the weekly schedule (Monday 03:00 UTC) or manual workflow_dispatch:
- e2e-full-test — full plain Playwright suite (
chromium, 2 retries) — schedule orrun_full_e2e - e2e-admin-test —
chromium-adminproject withseed-admin: true(admin login via TOTPstorageState+ designer/multi-zone fixtures) — schedule orrun_full_e2e - security-scan-dependencies — pip-audit, npm audit, Snyk (when
SNYK_TOKENis set) — schedule orrun_security_scan - docker-build — Docker image build validation + Trivy scans — schedule or
run_docker_build - seed data export/import round-trip (inside
backend-test) — schedule orrun_extended_checks; see Seed Data
On pushes to staging, build-images builds the Docker images in parallel with the tests and the deploy job (calls deploy-staging.yml) runs once backend-test, frontend-test, frontend-build, e2e-smoke-test and e2e-visual-test are green (or were skipped because the tree was already tested). Staging→main promotion PRs skip the heavy jobs; check-base is the only required check on main.
E2E Strategy in CI¶
| Trigger | Tests Run | Browsers |
|---|---|---|
Pull requests & pushes (main/staging) |
Smoke tests (@smoke tag) + visual regression |
Chromium (+ Mobile Safari for pdp-hydration.spec.ts) |
Weekly schedule or manual dispatch (run_full_e2e) |
Full plain suite + admin suite | Chromium |
After every staging deploy (deploy-staging.yml → e2e-staging.yml) |
staging-smoke.spec.ts against staging: homepage, product list, product page, cart; Discord on failure, no rollback |
Chromium |
Manual E2E tegen staging |
One spec against staging (see below) | Chromium |
The reusable workflow e2e-setup.yml (workflow_call) starts PostgreSQL, Redis, the backend with E2E_TESTING=True, and the frontend from the frontend-next-build artifact; its inputs are project, seed-admin, test-grep, workers (default 4), retries, webkit-specs, report-name and timeout-minutes. Each job uploads its Playwright report as an artifact.
Pre-push Hook¶
The pre-push git hook (.githooks/pre-push) runs a backend test subset with pytest (apps/orders/tests/test_api.py, apps/users/tests, apps/designs/tests.py, apps/pricing/tests.py, without coverage or xdist; requires local PostgreSQL on 5432 and Redis on 6379, e.g. via docker-compose.dev.yml), the TypeScript check, a non-blocking CodeRabbit review, and the Jest suite with coverage. If checks fail, the push is blocked. A push that only touches docs, Markdown or todo's (the same paths CI ignores via paths-ignore) skips the checks. In a git worktree the hook falls back to the venv, .env and node_modules of the main checkout and uses its own test database. Never bypass with --no-verify.
Test File Locations¶
backend/
├── conftest.py # Shared pytest fixtures
├── apps/
│ ├── <app>/factories.py # factory-boy factories (orders, payments, products, designs, quotes, users)
│ ├── payments/tests/ # Payment tests
│ │ ├── test_models.py # Payment, Refund, AuditLog model tests
│ │ ├── test_services.py # PaymentService unit tests
│ │ ├── test_services_mollie_sdk.py
│ │ ├── test_views.py # API endpoint tests
│ │ ├── test_webhook.py # Mollie webhook handler tests
│ │ ├── test_refund_service.py # Refund creation/validation
│ │ ├── test_refund_webhook.py # Refund webhook processing
│ │ ├── test_refund_mislukt.py
│ │ ├── test_retry.py # Payment retry tests
│ │ ├── test_alle_betaalmethoden.py
│ │ ├── test_e2e_payment_flow.py
│ │ └── test_audit.py # Audit logging tests
│ ├── invoices/tests/ # Invoice tests
│ │ ├── test_invoice_service.py # Generation, numbering, idempotency
│ │ ├── test_invoice_pdf_render.py
│ │ ├── test_invoice_tasks.py
│ │ └── test_invoice_views.py # Download authorization tests
│ ├── orders/tests/ # Orders, checkout, admin actions (test_api.py, test_refund_admin.py, ...)
│ └── <app>/tests.py # Smaller apps keep a single module (e.g. designs, pricing)
frontend/
├── __tests__/ # Jest unit tests (app/, components/, contexts/, hooks/, lib/, e2e-config/)
│ ├── test-utils.tsx
│ └── README.md
├── jest.setup.js # Browser mocks and matchers
├── e2e/ # Playwright tests (*.spec.ts)
│ ├── admin.setup.ts # Admin login for the chromium-admin project
│ ├── fixtures/, helpers/, pages/
│ ├── flows/
│ │ └── customer-journey.spec.ts # Full payment journey
│ ├── visual/
│ │ ├── visual.spec.ts
│ │ └── visual.spec.ts-snapshots/ # Baseline PNGs (linux)
│ ├── accessibility-audit.spec.ts
│ ├── staging-smoke.spec.ts # Used by "E2E tegen staging"
│ └── ...
├── playwright.staging.config.ts # Config for the staging workflow
├── coverage/ # Coverage reports (generated)
├── playwright-report/ # E2E reports (generated)
└── test-results/ # E2E artifacts (generated)
loadtests/ # k6 scripts
Writing New Tests¶
- Backend discovery accepts
test_*.py,*_test.pyandtests.pyunderbackend/testsandbackend/apps. Prefer the existing per-apptests/packages and shared factories or fixtures frombackend/conftest.pyand each app'sfactories.py. - Frontend Jest tests live under
frontend/__tests__/and use*.test.tsor*.test.tsx. Shared browser mocks and matchers are initialized byfrontend/jest.setup.js;e2e/is excluded from Jest. - Playwright tests live under
frontend/e2e/and use*.spec.ts. Reuse fixtures and page objects already present in that tree, and tag deployment-critical scenarios with@smokewhen appropriate. Specs that need the seeded admin must be added toSEEDED_SPECS/NON_PLAIN_SPECSinplaywright.config.ts. - Test observable behavior and regression boundaries; avoid assertions that merely mirror implementation details.
Coverage Requirements¶
| Suite | Branches | Functions | Lines | Statements |
|---|---|---|---|---|
| Backend pytest | in aggregate | — | 75% aggregate fail-under (lines + branches) | — |
| Frontend Jest | 54% | 59% | 61% | 60% |
The backend uses one aggregate 75% threshold (fail_under in backend/.coveragerc) rather than separate category thresholds. Branch coverage is on (branch = True), so the aggregate covers lines and branches together. Migrations, test files, factories, the backup commands/services and the seed-data commands are excluded (omit). Frontend values are enforced in frontend/jest.config.js; CI runs npm run test:ci with coverage.
Troubleshooting¶
"Cannot find module '@testing-library/react'"¶
"Playwright browsers not installed"¶
Tests timing out¶
Increase timeout in test file:
Port 3000 already in use (E2E tests)¶
Coverage not showing¶
Quick Reference¶
| Task | Command |
|---|---|
| Backend (pytest) | |
| All backend tests | cd backend && pytest |
| Payment tests | pytest apps/payments |
| Invoice tests | pytest apps/invoices |
| Webhook tests | pytest apps/payments/tests/test_webhook.py |
| Subset without coverage | pytest <path> -o addopts="" -q |
| Frontend (Jest) | |
| All unit tests | npm test |
| Watch mode | npm run test:watch |
| With coverage | npm run test:coverage |
| Frontend (Playwright) | |
| Smoke tests only | npx playwright test --project=chromium --grep="@smoke" |
| Plain suite | npx playwright test --project=chromium |
| Visual regression | npx playwright test --project=visual |
| All projects | npm run test:e2e |
| Headed mode | npm run test:e2e:headed |
| Interactive UI | npm run test:e2e:ui |
| Debug E2E | npx playwright test --debug |
| View report | npx playwright show-report |
| Load (k6) | |
| Local load test | ./manage.sh loadtest all |
Testen op staging¶
Staging (staging.freezedesign.nl) vraagt basic auth voor alle pagina's; /api/, allauth
en de healthcheck zijn vrijgesteld. Een agent mag dat wachtwoord niet zien, typen of
bewaren. Daarom twee routes:
Automatisch: de workflow E2E tegen staging (.github/workflows/e2e-staging.yml)
draait één Playwright-spec tegen staging (met playwright.staging.config.ts, Node 22). Het
wachtwoord komt uit het secret STAGING_BASIC_AUTH_BASE64 en blijft in GitHub. Sinds v1.52
(#861) start deploy-staging.yml hem na elke geslaagde deploy zelf met
e2e/staging-smoke.spec.ts (homepage, productoverzicht, productpagina, winkelwagen) en de
gedeployde sha; faalt die run, dan meldt hij het in Discord. Geen rollback: de deploy is
dan al klaar.
url=$(gh workflow run e2e-staging.yml --ref staging -f spec=e2e/staging-smoke.spec.ts 2>&1 \
| grep -o 'https://github.com/[^ ]*/actions/runs/[0-9]*')
run_id=${url##*/}
gh run watch "$run_id" --exit-status
gh run download "$run_id" --dir /tmp/e2e-staging
- Start altijd met
--ref staging: staging draait de branchstaging, en een nieuwe spec staat tot de promotie alleen daar. Start hem niet vanaf een feature-branch (zie het restrisico hieronder). - Gebruik het run-nummer uit de URL die
gh workflow runteruggeeft.gh run list --limit 1direct na het starten geeft vaak nog de vorige run. Geeftgh workflow rungeen URL (ouderegh), zoek de run dan metgh run list --workflow e2e-staging.yml --event workflow_dispatch --json databaseId,createdAten kies die met eencreatedAtná je start. - Alleen specs onder
e2e/met een naam alse2e/<naam>.spec.ts. - Geen traces, video of HAR; schermafbeeldingen maakt de test zelf met
testInfo.attach. Log in een spec geen omgevingsvariabelen of request-headers: maskeren werkt alleen in de log van de run, niet in het rapport. - Er draait één run tegelijk (
concurrency: e2e-staging). Start je er een derde terwijl er één loopt en één wacht, dan annuleert GitHub de wachtende. - Artifacts blijven 3 dagen bewaard.
- GitHub start een
workflow_dispatchalleen als het bestand opmainstaat: de workflow is bruikbaar na de promotie die hem bevat.
Restrisico: deze opzet voorkomt dat het wachtwoord per ongeluk in een log of rapport
belandt. Hij houdt een agent die het wíl uitlezen niet tegen: agents gebruiken gh als
Voorman, met beheerrechten op de repo. Zo'n agent kan:
- een spec via een PR op de branch
stagingzetten (die PR heeft geen review nodig, alleen een groeneci-gate) en de workflow daarop starten; load-tests.ymlstarten met een eigenbase_url; die stuurt deAuthorization-header naar dat adres;staging-perf-gate.ymlstarten met een willekeurigesha; die checkt code van elke commit uit;- als beheerder de instellingen van de omgeving
stagingaanpassen.
Het secret naar de omgeving staging verplaatsen helpt pas als die vier ook dicht zijn. Dat
staat in de todo staging-secrets-afschermen.
Handmatig: voor losse controles en alles achter de admin-login opent de agent staging in de ingebouwde browser van de Claude-app. Ivar typt daar zelf de basic auth en zo nodig de admin-login; de agent gebruikt daarna die sessie.