Skip to content

Testing Guide

This guide covers testing strategies, IDE setup, and debugging for the Freeze Design webshop.

The backend uses pytest 9.2 with pytest-django, pytest-cov, pytest-xdist and factory-boy; its defaults live in backend/pyproject.toml and backend/.coveragerc. The frontend uses Jest 30.5 with Testing Library and jest-environment-jsdom, configured by frontend/jest.config.js. Browser tests use Playwright 1.64 through frontend/playwright.config.ts. Load tests use k6 scripts in loadtests/.

Run ./manage.sh setup once, or install backend dependencies from backend/requirements.txt into backend/venv and frontend dependencies with npm install in frontend/. Install Playwright's browsers and system dependencies with npx playwright install --with-deps when they are absent.

Test Pyramid

        ┌───────────┐
        │   E2E     │  ← Few, slow, high confidence
        ├───────────┤
        │Integration│  ← Medium number
        ├───────────┤
        │   Unit    │  ← Many, fast, focused
        └───────────┘

Backend Testing (Django)

Backend tests run on pytest (with pytest-django, pytest-xdist, and factory-boy) — this is what CI and the pre-push hook run and what you should run locally. python manage.py test still works (./manage.sh ci-check uses it) but is not the configured path. The tests need PostgreSQL on 5432 and Redis on 6379, e.g. via docker compose -f docker-compose.dev.yml up -d db redis.

Run Tests

cd backend
pytest

# Run specific app tests
pytest apps/payments
pytest apps/invoices
pytest apps/orders

# Run specific test modules
pytest apps/payments/tests/test_webhook.py
pytest apps/payments/tests/test_refund_service.py
pytest apps/invoices/tests/test_invoice_service.py

# Run a single test by name
pytest -k "test_webhook_marks_order_paid"

# Extra verbosity
pytest -vv

# One process, no coverage (faster for a small subset; this is what the pre-push hook does)
pytest apps/pricing/tests.py -o addopts="" -q

With Coverage

Coverage is switched on in backend/pyproject.toml (addopts): parallel (-n auto --dist=loadscope) and a term-missing report. What is measured and the 75% fail-under threshold (lines and branches together) live in backend/.coveragerc. Running pytest bare already applies all of this — CI runs the same command and combines the per-worker coverage data afterwards. testpaths is tests, apps and config, so the configuration tests in backend/config/test_checks.py run too.

# Same as CI (coverage via pyproject addopts, fail-under 75% from .coveragerc)
pytest

# HTML coverage report
pytest --cov-report=html   # open htmlcov/index.html

Markers: integration (needs all services) and slow (> 2 seconds) are registered in pyproject.toml.

Test Structure

# tests/test_products.py
from django.test import TestCase
from rest_framework.test import APITestCase

class ProductAPITests(APITestCase):
    def setUp(self):
        self.product = Product.objects.create(
            name='Test Product',
            base_price=29.95
        )

    def test_list_products(self):
        response = self.client.get('/api/products/')
        self.assertEqual(response.status_code, 200)
        self.assertEqual(len(response.data['results']), 1)

    def test_product_detail(self):
        response = self.client.get(f'/api/products/{self.product.slug}/')
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.data['name'], 'Test Product')

Plain pytest functions with the fixtures from backend/conftest.py and the per-app factories.py (orders, payments, products, designs, quotes, users) are equally fine; match the style of the surrounding tests.

Frontend Testing

Unit Tests (Jest)

cd frontend
npm run test
npm run test:coverage
npm run test:watch          # Watch mode (auto-rerun)
npm test CartContext        # Specific test file

Coverage thresholds are enforced in frontend/jest.config.js (branches 54%, functions 59%, lines 61%, statements 60%) and are raised incrementally as coverage improves. Coverage is collected from app/, components/, lib/, contexts/ and hooks/. CI runs npm run test:ci (jest --watchman=false --coverage --forceExit).

Test Structure

// Component.test.tsx
import { render, screen, fireEvent } from '@testing-library/react';
import { QuantitySelector } from './QuantitySelector';

describe('QuantitySelector', () => {
  it('increments quantity when plus clicked', () => {
    const onChange = jest.fn();
    render(<QuantitySelector quantity={1} onQuantityChange={onChange} />);

    fireEvent.click(screen.getByLabelText('Verhoog aantal'));

    expect(onChange).toHaveBeenCalledWith(2);
  });

  it('disables decrement at minimum', () => {
    render(<QuantitySelector quantity={1} onQuantityChange={() => {}} min={1} />);

    expect(screen.getByLabelText('Verlaag aantal')).toBeDisabled();
  });
});

Running Specific Tests

# Single test file
npm test CartContext.test.tsx

# Single test case
npm test -- -t "should add item to cart"

# All tests matching pattern (Jest 30: --testPathPatterns, plural)
npm test -- --testPathPatterns=contexts

E2E Testing (Playwright)

Prerequisites

Playwright talks to a real backend. Start it with E2E_TESTING=True (disables throttling, django-axes and mandatory email verification; never set it outside DEBUG) and a migrated database. Without E2E_BASE_URL or CI set, playwright.config.ts starts npm run dev on port 3000 itself (reuseExistingServer); set E2E_BASE_URL to point at a stack you manage yourself.

Projects

Defined in frontend/playwright.config.ts:

Project Runs Notes
chromium All plain specs What e2e-smoke-test and e2e-full-test use in CI; excludes admin/seeded and visual specs
chromium-admin Seed-dependent specs (admin-*, cart-mobiel, collectie, designer-order, quote-design, multi-zone-views, personalisatie, and designer/clipart-tools for the seeded clipart) Depends on the setup project (admin.setup.ts: one TOTP admin login whose storageState all admin specs reuse); needs ensure_e2e_admin and seed_e2e_extras on the backend. clipart-tools also runs in the plain projects; only the clipart tests skip there
firefox, webkit, Mobile Chrome, Mobile Safari Plain specs in other browsers/viewports CI runs pdp-hydration.spec.ts in Mobile Safari as part of the smoke job
visual e2e/visual/visual.spec.ts Visual regression with reducedMotion: 'reduce'; see below

On CI: 2 workers and 1 retry by default (the reusable workflow passes its own --workers/--retries).

Run Tests

cd frontend
npx playwright test --project=chromium --grep @smoke   # deploy gate, fastest
npx playwright test --project=chromium                 # full plain suite
npm run test:e2e                                       # every project, incl. chromium-admin and visual
npm run test:e2e:headed                                # See browser while testing
npm run test:e2e:ui                                    # Interactive Playwright UI

npm run test:e2e runs all projects; chromium-admin fails without a seeded admin, so prefer --project=chromium locally.

Running Specific E2E Tests

# Single test file
npx playwright test team-order-flow.spec.ts

# Single test case
npx playwright test -g "van productpagina tot winkelwagen"

# By tag
npx playwright test --grep="@smoke"

# Specific browser
npx playwright test --project=firefox
npx playwright test --project=webkit

Tags in use: @smoke (deploy gate), @critical, @journey, @regression, @design, @frontend-only, @payment-integration.

Test Structure

// e2e/checkout.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Checkout Flow', () => {
  test('complete order', async ({ page }) => {
    await page.goto('/products/classic-t-shirt');
    await page.click('[data-testid="color-swatch-1"]');
    await page.click('[data-testid="size-option-3"]');
    await page.click('[data-testid="add-to-cart"]');
    await page.click('[data-testid="checkout-button"]');
    await page.fill('[name="email"]', 'test@example.com');
    await page.click('[data-testid="submit-order"]');
    await expect(page.locator('.order-confirmation')).toBeVisible();
  });
});

Visual Regression

e2e/visual/visual.spec.ts compares screenshots against the baselines in e2e/visual/visual.spec.ts-snapshots/ (*-linux.png, so baselines must come from Linux). The e2e-visual-test job runs them on every PR and push. To refresh the baselines, start the Update Visual Snapshots workflow (update-snapshots.yml, workflow_dispatch, optional branch input): it runs npx playwright test --project=visual --update-snapshots, pushes the new PNGs to vrt/update-baselines-<timestamp> and opens a PR. Locally the same command works on Linux only.

Accessibility

e2e/accessibility-audit.spec.ts runs axe (@axe-core/playwright) against the main pages as part of the plain suite.

Quarantine

Tests that fail only in the e2e full run while the feature works in production are parked with test.fixme / test.describe.fixme and a // QUARANTINE (e2e-debt): ... comment. The list and the deblocking procedure are in the E2E-quarantaine runbook. The deploy gate (@smoke + visual) is unaffected.

Viewing E2E Results

  • HTML Report: Auto-opens after test run, or manually: npx playwright show-report
  • Trace Viewer: For failed tests: npx playwright show-trace trace.zip

Load Tests (k6)

The k6 scripts live in loadtests/ (product-listing.js, cart-operations.js, design-upload.js, homepage-load.js, checkout-load.js; thresholds and history in loadtests/README.md and loadtests/baseline-results.md).

  • Locally: ./manage.sh loadtest [all|products|cart|upload] against the running backend (needs k6 installed).
  • Load Tests workflow (load-tests.yml, manual): inputs test_suite (all, products, cart, upload, homepage, checkout), base_url and frontend_url (default https://staging.freezedesign.nl). It is a manual re-baselining tool, not a gate.
  • Staging performance gate (staging-perf-gate.yml): started by deploy-staging.yml after every successful staging deploy; runs loadtests/homepage-load.js against staging and posts to Discord when it fails. It does not block the deploy.

VS Code Setup

Extension ID Purpose
Jest Orta.vscode-jest Inline test results, coverage, run/debug tests
Playwright Test ms-playwright.playwright Run/debug E2E tests, record tests, trace viewer

Install via terminal:

code --install-extension Orta.vscode-jest
code --install-extension ms-playwright.playwright

Configure Jest Extension

Add to .vscode/settings.json:

{
  "jest.autoRun": "off",
  "jest.showCoverageOnLoad": false,
  "jest.rootPath": "frontend"
}

VS Code Shortcuts

Unit Tests (Jest):

  • Hover over test and click play button to run
  • Right-click test for "Debug Test"
  • Cmd+Shift+P then "Jest: Toggle Coverage"

E2E Tests (Playwright):

  • Click play button in gutter next to test
  • Right-click for "Debug Test"
  • Cmd+Shift+P then "Test: Record New"

Debugging Tests

Debug Unit Tests (Jest)

  1. Set breakpoint (click left of line number)
  2. Right-click test and select "Debug Test"
  3. Use debug toolbar to step through

node --inspect-brk node_modules/.bin/jest --runInBand CartContext
Then open chrome://inspect in Chrome.

Debug E2E Tests (Playwright)

npx playwright test --debug
  1. Set breakpoint in test file
  2. Right-click test and select "Debug Test"
  3. Playwright Inspector opens automatically

npm run test:e2e:headed
See browser actions in real-time.

CI/CD Integration

Tests run automatically via GitHub Actions (.github/workflows/ci.yml) on every push to main/staging (except docs-only pushes, paths-ignore) and on every PR against those branches. CI runs Python 3.12 and Node 22 with PostgreSQL 15 and Redis 7 service containers. See the GitHub Workflow for the gates and the deploy.

CI Jobs on Every PR / Push

  • precheck — decides what runs: promotion PRs (staging→main) and docs-only PRs skip the test jobs; a push to staging whose git tree was already green in a PR run (tested-tree-<tree> artifact) skips them too
  • backend-test — flake8 (E9,F63,F7,F82 blocking; complexity/style report with a 127-character limit), makemigrations --check, check_migrations --ci, migrations, full pytest suite (parallel, coverage ≥75%)
  • frontend-test — TypeScript check, ESLint, npm run test:ci (coverage thresholds from jest.config.js)
  • frontend-build — Next.js production build (artifact reused by the E2E jobs)
  • e2e-smoke-test — Playwright @smoke tests in chromium, plus pdp-hydration.spec.ts in Mobile Safari (via the reusable e2e-setup.yml)
  • e2e-visual-test — Playwright visual project
  • api-types-drift ("API types drift check") — regenerates the frontend API types from the backend OpenAPI schema and fails if the committed types drifted
  • nginx-config — PRs only: scripts/tests/test_nginx_security_headers.sh, test_restart_unhealthy.sh, test_notify_ntfy.sh, test_deploy_rolling.sh, test_werkoverzicht.py and test_werkbord.py (unittest) and test_intake_ready.sh; test_check_done_todos.sh runs in todo-done-check.yml
  • ci-gate — PRs only: green when all of the above succeeded or were skipped; the required check on staging

Runs only on the weekly schedule (Monday 03:00 UTC) or manual workflow_dispatch:

  • e2e-full-test — full plain Playwright suite (chromium, 2 retries) — schedule or run_full_e2e
  • e2e-admin-test — chromium-admin project with seed-admin: true (admin login via TOTP storageState + designer/multi-zone fixtures) — schedule or run_full_e2e
  • security-scan-dependencies — pip-audit, npm audit, Snyk (when SNYK_TOKEN is set) — schedule or run_security_scan
  • docker-build — Docker image build validation + Trivy scans — schedule or run_docker_build
  • seed data export/import round-trip (inside backend-test) — schedule or run_extended_checks; see Seed Data

On pushes to staging, build-images builds the Docker images in parallel with the tests and the deploy job (calls deploy-staging.yml) runs once backend-test, frontend-test, frontend-build, e2e-smoke-test and e2e-visual-test are green (or were skipped because the tree was already tested). Staging→main promotion PRs skip the heavy jobs; check-base is the only required check on main.

E2E Strategy in CI

Trigger Tests Run Browsers
Pull requests & pushes (main/staging) Smoke tests (@smoke tag) + visual regression Chromium (+ Mobile Safari for pdp-hydration.spec.ts)
Weekly schedule or manual dispatch (run_full_e2e) Full plain suite + admin suite Chromium
After every staging deploy (deploy-staging.yml → e2e-staging.yml) staging-smoke.spec.ts against staging: homepage, product list, product page, cart; Discord on failure, no rollback Chromium
Manual E2E tegen staging One spec against staging (see below) Chromium

The reusable workflow e2e-setup.yml (workflow_call) starts PostgreSQL, Redis, the backend with E2E_TESTING=True, and the frontend from the frontend-next-build artifact; its inputs are project, seed-admin, test-grep, workers (default 4), retries, webkit-specs, report-name and timeout-minutes. Each job uploads its Playwright report as an artifact.

Pre-push Hook

The pre-push git hook (.githooks/pre-push) runs a backend test subset with pytest (apps/orders/tests/test_api.py, apps/users/tests, apps/designs/tests.py, apps/pricing/tests.py, without coverage or xdist; requires local PostgreSQL on 5432 and Redis on 6379, e.g. via docker-compose.dev.yml), the TypeScript check, a non-blocking CodeRabbit review, and the Jest suite with coverage. If checks fail, the push is blocked. A push that only touches docs, Markdown or todo's (the same paths CI ignores via paths-ignore) skips the checks. In a git worktree the hook falls back to the venv, .env and node_modules of the main checkout and uses its own test database. Never bypass with --no-verify.

Test File Locations

backend/
├── conftest.py                      # Shared pytest fixtures
├── apps/
│   ├── <app>/factories.py           # factory-boy factories (orders, payments, products, designs, quotes, users)
│   ├── payments/tests/              # Payment tests
│   │   ├── test_models.py           # Payment, Refund, AuditLog model tests
│   │   ├── test_services.py         # PaymentService unit tests
│   │   ├── test_services_mollie_sdk.py
│   │   ├── test_views.py            # API endpoint tests
│   │   ├── test_webhook.py          # Mollie webhook handler tests
│   │   ├── test_refund_service.py   # Refund creation/validation
│   │   ├── test_refund_webhook.py   # Refund webhook processing
│   │   ├── test_refund_mislukt.py
│   │   ├── test_retry.py            # Payment retry tests
│   │   ├── test_alle_betaalmethoden.py
│   │   ├── test_e2e_payment_flow.py
│   │   └── test_audit.py            # Audit logging tests
│   ├── invoices/tests/              # Invoice tests
│   │   ├── test_invoice_service.py  # Generation, numbering, idempotency
│   │   ├── test_invoice_pdf_render.py
│   │   ├── test_invoice_tasks.py
│   │   └── test_invoice_views.py    # Download authorization tests
│   ├── orders/tests/                # Orders, checkout, admin actions (test_api.py, test_refund_admin.py, ...)
│   └── <app>/tests.py               # Smaller apps keep a single module (e.g. designs, pricing)

frontend/
├── __tests__/                    # Jest unit tests (app/, components/, contexts/, hooks/, lib/, e2e-config/)
│   ├── test-utils.tsx
│   └── README.md
├── jest.setup.js                 # Browser mocks and matchers
├── e2e/                          # Playwright tests (*.spec.ts)
│   ├── admin.setup.ts            # Admin login for the chromium-admin project
│   ├── fixtures/, helpers/, pages/
│   ├── flows/
│   │   └── customer-journey.spec.ts  # Full payment journey
│   ├── visual/
│   │   ├── visual.spec.ts
│   │   └── visual.spec.ts-snapshots/ # Baseline PNGs (linux)
│   ├── accessibility-audit.spec.ts
│   ├── staging-smoke.spec.ts     # Used by "E2E tegen staging"
│   └── ...
├── playwright.staging.config.ts  # Config for the staging workflow
├── coverage/                     # Coverage reports (generated)
├── playwright-report/            # E2E reports (generated)
└── test-results/                 # E2E artifacts (generated)

loadtests/                        # k6 scripts

Writing New Tests

  • Backend discovery accepts test_*.py, *_test.py and tests.py under backend/tests and backend/apps. Prefer the existing per-app tests/ packages and shared factories or fixtures from backend/conftest.py and each app's factories.py.
  • Frontend Jest tests live under frontend/__tests__/ and use *.test.ts or *.test.tsx. Shared browser mocks and matchers are initialized by frontend/jest.setup.js; e2e/ is excluded from Jest.
  • Playwright tests live under frontend/e2e/ and use *.spec.ts. Reuse fixtures and page objects already present in that tree, and tag deployment-critical scenarios with @smoke when appropriate. Specs that need the seeded admin must be added to SEEDED_SPECS/NON_PLAIN_SPECS in playwright.config.ts.
  • Test observable behavior and regression boundaries; avoid assertions that merely mirror implementation details.

Coverage Requirements

Suite Branches Functions Lines Statements
Backend pytest in aggregate — 75% aggregate fail-under (lines + branches) —
Frontend Jest 54% 59% 61% 60%

The backend uses one aggregate 75% threshold (fail_under in backend/.coveragerc) rather than separate category thresholds. Branch coverage is on (branch = True), so the aggregate covers lines and branches together. Migrations, test files, factories, the backup commands/services and the seed-data commands are excluded (omit). Frontend values are enforced in frontend/jest.config.js; CI runs npm run test:ci with coverage.

Troubleshooting

"Cannot find module '@testing-library/react'"

cd frontend && npm install

"Playwright browsers not installed"

cd frontend && npx playwright install --with-deps

Tests timing out

Increase timeout in test file:

test.setTimeout(60000); // 60 seconds

Port 3000 already in use (E2E tests)

lsof -ti:3000 | xargs kill -9

Coverage not showing

npm test -- --clearCache
npm run test:coverage

Quick Reference

Task Command
Backend (pytest)
All backend tests cd backend && pytest
Payment tests pytest apps/payments
Invoice tests pytest apps/invoices
Webhook tests pytest apps/payments/tests/test_webhook.py
Subset without coverage pytest <path> -o addopts="" -q
Frontend (Jest)
All unit tests npm test
Watch mode npm run test:watch
With coverage npm run test:coverage
Frontend (Playwright)
Smoke tests only npx playwright test --project=chromium --grep="@smoke"
Plain suite npx playwright test --project=chromium
Visual regression npx playwright test --project=visual
All projects npm run test:e2e
Headed mode npm run test:e2e:headed
Interactive UI npm run test:e2e:ui
Debug E2E npx playwright test --debug
View report npx playwright show-report
Load (k6)
Local load test ./manage.sh loadtest all

Testen op staging

Staging (staging.freezedesign.nl) vraagt basic auth voor alle pagina's; /api/, allauth en de healthcheck zijn vrijgesteld. Een agent mag dat wachtwoord niet zien, typen of bewaren. Daarom twee routes:

Automatisch: de workflow E2E tegen staging (.github/workflows/e2e-staging.yml) draait één Playwright-spec tegen staging (met playwright.staging.config.ts, Node 22). Het wachtwoord komt uit het secret STAGING_BASIC_AUTH_BASE64 en blijft in GitHub. Sinds v1.52 (#861) start deploy-staging.yml hem na elke geslaagde deploy zelf met e2e/staging-smoke.spec.ts (homepage, productoverzicht, productpagina, winkelwagen) en de gedeployde sha; faalt die run, dan meldt hij het in Discord. Geen rollback: de deploy is dan al klaar.

url=$(gh workflow run e2e-staging.yml --ref staging -f spec=e2e/staging-smoke.spec.ts 2>&1 \
  | grep -o 'https://github.com/[^ ]*/actions/runs/[0-9]*')
run_id=${url##*/}
gh run watch "$run_id" --exit-status
gh run download "$run_id" --dir /tmp/e2e-staging
  • Start altijd met --ref staging: staging draait de branch staging, en een nieuwe spec staat tot de promotie alleen daar. Start hem niet vanaf een feature-branch (zie het restrisico hieronder).
  • Gebruik het run-nummer uit de URL die gh workflow run teruggeeft. gh run list --limit 1 direct na het starten geeft vaak nog de vorige run. Geeft gh workflow run geen URL (oudere gh), zoek de run dan met gh run list --workflow e2e-staging.yml --event workflow_dispatch --json databaseId,createdAt en kies die met een createdAt ná je start.
  • Alleen specs onder e2e/ met een naam als e2e/<naam>.spec.ts.
  • Geen traces, video of HAR; schermafbeeldingen maakt de test zelf met testInfo.attach. Log in een spec geen omgevingsvariabelen of request-headers: maskeren werkt alleen in de log van de run, niet in het rapport.
  • Er draait één run tegelijk (concurrency: e2e-staging). Start je er een derde terwijl er één loopt en één wacht, dan annuleert GitHub de wachtende.
  • Artifacts blijven 3 dagen bewaard.
  • GitHub start een workflow_dispatch alleen als het bestand op main staat: de workflow is bruikbaar na de promotie die hem bevat.

Restrisico: deze opzet voorkomt dat het wachtwoord per ongeluk in een log of rapport belandt. Hij houdt een agent die het wíl uitlezen niet tegen: agents gebruiken gh als Voorman, met beheerrechten op de repo. Zo'n agent kan:

  1. een spec via een PR op de branch staging zetten (die PR heeft geen review nodig, alleen een groene ci-gate) en de workflow daarop starten;
  2. load-tests.yml starten met een eigen base_url; die stuurt de Authorization-header naar dat adres;
  3. staging-perf-gate.yml starten met een willekeurige sha; die checkt code van elke commit uit;
  4. als beheerder de instellingen van de omgeving staging aanpassen.

Het secret naar de omgeving staging verplaatsen helpt pas als die vier ook dicht zijn. Dat staat in de todo staging-secrets-afschermen.

Handmatig: voor losse controles en alles achter de admin-login opent de agent staging in de ingebouwde browser van de Claude-app. Ivar typt daar zelf de basic auth en zo nodig de admin-login; de agent gebruikt daarna die sessie.