From 036a81b6de1352ce44ff192b426117106a20e12e Mon Sep 17 00:00:00 2001 From: Thomas Richter Date: Tue, 3 Feb 2026 23:17:33 +0100 Subject: [PATCH] docs(09): research CI pipeline hardening domain Phase 9: CI Pipeline Hardening - Standard stack: Vitest + browser mode, Playwright, svelte-check - Architecture: Multi-project config for client/server tests - Pitfalls: jsdom limitations, database parallelism, SvelteKit mocking Co-Authored-By: Claude Opus 4.5 --- .../phases/09-ci-pipeline/09-RESEARCH.md | 503 ++++++++++++++++++ 1 file changed, 503 insertions(+) create mode 100644 .planning/phases/09-ci-pipeline/09-RESEARCH.md diff --git a/.planning/phases/09-ci-pipeline/09-RESEARCH.md b/.planning/phases/09-ci-pipeline/09-RESEARCH.md new file mode 100644 index 0000000..90ce950 --- /dev/null +++ b/.planning/phases/09-ci-pipeline/09-RESEARCH.md @@ -0,0 +1,503 @@ +# Phase 9: CI Pipeline Hardening - Research + +**Researched:** 2026-02-03 +**Domain:** Testing infrastructure (Vitest, Playwright, svelte-check) + CI/CD (Gitea Actions) +**Confidence:** HIGH + +## Summary + +This phase establishes a comprehensive testing pipeline that blocks deployment on test or type failures. The research covers three testing layers: unit tests (Vitest), type checking (svelte-check), and E2E tests (Playwright), integrated into the existing Gitea Actions workflow. + +The standard approach for SvelteKit testing in 2026 uses Vitest with browser mode for component tests (real browser via Playwright, not jsdom), traditional Vitest for server/utility tests, and standalone Playwright for full E2E tests. The user decisions lock in 80% coverage threshold, Playwright component tests for DOM interactions, and Slack notifications on failure. + +Key finding: Vitest browser mode with `vitest-browser-svelte` is the modern approach for Svelte 5 component testing, replacing the older jsdom + @testing-library/svelte pattern. This provides real browser testing with runes support (`.svelte.test.ts` files). + +**Primary recommendation:** Use multi-project Vitest configuration separating client (browser mode) and server (node) tests, with standalone Playwright for E2E, all gated before Docker build in CI. + +## Standard Stack + +The established libraries/tools for this domain: + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| vitest | ^3.x | Unit/component test runner | Official SvelteKit recommendation, Vite-native | +| @vitest/browser | ^3.x | Browser mode for component tests | Real browser testing without jsdom limitations | +| vitest-browser-svelte | ^0.x | Svelte component rendering in browser mode | Official Svelte 5 support with runes | +| @vitest/browser-playwright | ^3.x | Playwright provider for Vitest browser mode | Real Chrome DevTools Protocol, not simulated events | +| @vitest/coverage-v8 | ^3.x | V8-based coverage collection | Fast native coverage, identical accuracy to Istanbul since v3.2 | +| @playwright/test | ^1.58 | E2E test framework | Already installed, mature E2E solution | +| svelte-check | ^4.x | TypeScript/Svelte type checking | Already installed, CI-compatible output | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| @testing-library/svelte | ^5.x | Alternative component testing | Only if not using browser mode (jsdom fallback) | +| drizzle-seed | ^0.x | Database seeding for tests | E2E test fixtures with Drizzle ORM | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| Vitest browser mode | jsdom + @testing-library/svelte | jsdom simulates browser, misses real CSS/runes issues | +| v8 coverage | istanbul | istanbul 300% slower, v8 now equally accurate | +| Playwright for E2E | Cypress | Playwright already in project, better multi-browser support | + +**Installation:** +```bash +npm install -D vitest @vitest/browser vitest-browser-svelte @vitest/browser-playwright @vitest/coverage-v8 drizzle-seed +npx playwright install chromium +``` + +## Architecture Patterns + +### Recommended Project Structure +``` +src/ +├── lib/ +│ ├── components/ +│ │ ├── Button.svelte +│ │ └── Button.svelte.test.ts # Component tests (browser mode) +│ ├── utils/ +│ │ ├── format.ts +│ │ └── format.test.ts # Utility tests (node mode) +│ └── server/ +│ ├── db/ +│ │ └── queries.test.ts # Server tests (node mode) +│ └── api.test.ts +├── routes/ +│ └── +page.server.test.ts # Server route tests (node mode) +tests/ +├── e2e/ # Playwright E2E tests +│ ├── fixtures/ +│ │ └── db.ts # Database seeding fixture +│ ├── user-journeys.spec.ts +│ └── index.ts # Custom test with fixtures +├── docker-deployment.spec.ts # Existing deployment tests +vitest-setup-client.ts # Browser mode setup +vitest.config.ts # Multi-project config (or in vite.config.ts) +playwright.config.ts # E2E config (already exists) +``` + +### Pattern 1: Multi-Project Vitest Configuration +**What:** Separate test projects for different environments (browser vs node) +**When to use:** SvelteKit apps with both client components and server code +**Example:** +```typescript +// vite.config.ts +// Source: https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide +import { sveltekit } from '@sveltejs/kit/vite'; +import tailwindcss from '@tailwindcss/vite'; +import { playwright } from '@vitest/browser-playwright'; +import { defineConfig } from 'vite'; + +export default defineConfig({ + plugins: [tailwindcss(), sveltekit()], + test: { + coverage: { + provider: 'v8', + reporter: ['text', 'json', 'html'], + include: ['src/**/*.{ts,svelte}'], + exclude: ['src/**/*.test.ts', 'src/**/*.spec.ts'], + thresholds: { + global: { + statements: 80, + branches: 80, + functions: 80, + lines: 80, + }, + }, + }, + projects: [ + { + extends: true, + test: { + name: 'client', + testTimeout: 5000, + browser: { + enabled: true, + provider: playwright(), + instances: [{ browser: 'chromium' }], + }, + include: ['src/**/*.svelte.{test,spec}.{js,ts}'], + setupFiles: ['./vitest-setup-client.ts'], + }, + }, + { + extends: true, + test: { + name: 'server', + environment: 'node', + include: ['src/**/*.{test,spec}.{js,ts}'], + exclude: ['src/**/*.svelte.{test,spec}.{js,ts}'], + }, + }, + ], + }, +}); +``` + +### Pattern 2: Component Test with Runes Support +**What:** Test Svelte 5 components with $state and $derived in real browser +**When to use:** Any component using Svelte 5 runes +**Example:** +```typescript +// src/lib/components/Counter.svelte.test.ts +// Source: https://svelte.dev/docs/svelte/testing +import { render } from 'vitest-browser-svelte'; +import { page } from '@vitest/browser/context'; +import { describe, expect, it } from 'vitest'; +import { flushSync } from 'svelte'; +import Counter from './Counter.svelte'; + +describe('Counter Component', () => { + it('increments count on click', async () => { + render(Counter, { props: { initial: 0 } }); + + const button = page.getByRole('button', { name: /increment/i }); + await button.click(); + + // flushSync needed for external state changes + flushSync(); + + await expect.element(page.getByText('Count: 1')).toBeInTheDocument(); + }); +}); +``` + +### Pattern 3: E2E Database Fixture with Drizzle +**What:** Seed database before tests, clean up after +**When to use:** E2E tests requiring known data state +**Example:** +```typescript +// tests/e2e/fixtures/db.ts +// Source: https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/ +import { test as base } from '@playwright/test'; +import { db } from '../../../src/lib/server/db/index.js'; +import * as schema from '../../../src/lib/server/db/schema.js'; +import { reset, seed } from 'drizzle-seed'; + +export const test = base.extend<{ + seededDb: typeof db; +}>({ + seededDb: async ({}, use) => { + // Seed with known test data + await seed(db, schema, { count: 10 }); + + await use(db); + + // Clean up after test + await reset(db, schema); + }, +}); + +export { expect } from '@playwright/test'; +``` + +### Pattern 4: SvelteKit Module Mocking +**What:** Mock $app/stores and $app/navigation in unit tests +**When to use:** Testing components that use SvelteKit-specific imports +**Example:** +```typescript +// vitest-setup-client.ts +// Source: https://www.closingtags.com/blog/mocking-svelte-stores-in-vitest +/// + +import { vi } from 'vitest'; +import { writable } from 'svelte/store'; + +// Mock $app/navigation +vi.mock('$app/navigation', () => ({ + goto: vi.fn(() => Promise.resolve()), + invalidate: vi.fn(() => Promise.resolve()), + invalidateAll: vi.fn(() => Promise.resolve()), + beforeNavigate: vi.fn(), + afterNavigate: vi.fn(), +})); + +// Mock $app/stores +vi.mock('$app/stores', () => ({ + page: writable({ + url: new URL('http://localhost'), + params: {}, + route: { id: null }, + status: 200, + error: null, + data: {}, + form: null, + }), + navigating: writable(null), + updated: { check: vi.fn(), subscribe: writable(false).subscribe }, +})); +``` + +### Anti-Patterns to Avoid +- **Testing with jsdom for Svelte 5 components:** jsdom cannot properly handle runes reactivity. Use browser mode instead. +- **Parallel E2E tests with shared database:** Will cause race conditions. Set `workers: 1` in playwright.config.ts. +- **Using deprecated @playwright/experimental-ct-svelte:** Use vitest-browser-svelte instead for component tests. +- **Mocking everything in E2E tests:** E2E tests should test real integrations. Only mock external services if necessary. + +## Don't Hand-Roll + +Problems that look simple but have existing solutions: + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Coverage collection | Custom instrumentation | @vitest/coverage-v8 | Handles source maps, thresholds, reporters automatically | +| Database seeding | Manual INSERT statements | drizzle-seed | Generates consistent, seeded random data with schema awareness | +| Component mounting | Manual DOM manipulation | vitest-browser-svelte render() | Handles Svelte 5 lifecycle, context, and cleanup | +| Screenshot on failure | Custom error handlers | Playwright built-in `screenshot: 'only-on-failure'` | Integrated with test lifecycle and artifacts | +| CI test output parsing | Regex parsing | svelte-check --output machine | Structured, timestamp-prefixed output designed for CI | + +**Key insight:** The testing ecosystem has mature solutions for all common needs. Hand-rolling any of these leads to edge cases around cleanup, async timing, and framework integration that the official tools have already solved. + +## Common Pitfalls + +### Pitfall 1: jsdom Limitations with Svelte 5 Runes +**What goes wrong:** Tests pass locally but fail to detect reactivity issues, or throw cryptic errors about $state +**Why it happens:** jsdom simulates browser APIs but doesn't actually run JavaScript in a browser context. Svelte 5 runes compile differently and expect real browser reactivity. +**How to avoid:** Use Vitest browser mode with Playwright provider for all `.svelte` component tests +**Warning signs:** Tests involving $state, $derived, or $effect behave inconsistently or require excessive `await tick()` + +### Pitfall 2: Missing flushSync for External State +**What goes wrong:** Assertions fail because DOM hasn't updated after state change +**Why it happens:** Svelte batches updates. When state changes outside component (e.g., store update in test), DOM update is async. +**How to avoid:** Call `flushSync()` from 'svelte' after modifying external state before asserting +**Warning signs:** Tests that work with longer timeouts but fail with short ones + +### Pitfall 3: Parallel E2E with Shared Database +**What goes wrong:** Flaky tests that sometimes pass, sometimes fail with data conflicts +**Why it happens:** Multiple test workers modify the same database simultaneously +**How to avoid:** Set `workers: 1` in playwright.config.ts for E2E tests. Use separate database per worker if parallelism is needed. +**Warning signs:** Tests pass individually but fail in full suite runs + +### Pitfall 4: Coverage Threshold Breaking Existing Code +**What goes wrong:** CI fails immediately after enabling 80% threshold because existing code has 0% coverage +**Why it happens:** Enabling coverage thresholds on existing codebase without tests +**How to avoid:** Start with `thresholds: { autoUpdate: true }` to establish baseline, then incrementally raise thresholds as tests are added +**Warning signs:** Immediate CI failure when coverage is first enabled + +### Pitfall 5: SvelteKit Module Import Errors +**What goes wrong:** Tests fail with "Cannot find module '$app/stores'" or similar +**Why it happens:** $app/* modules are virtual modules provided by SvelteKit at build time, not available in test environment +**How to avoid:** Mock all $app/* imports in vitest setup file. Keep mocks simple (don't use importOriginal with SvelteKit modules - causes SSR issues). +**Warning signs:** Import errors mentioning $app, $env, or other SvelteKit virtual modules + +### Pitfall 6: Playwright Browsers Not Installed in CI +**What goes wrong:** CI fails with "browserType.launch: Executable doesn't exist" +**Why it happens:** Playwright browsers need explicit installation, not included in npm install +**How to avoid:** Add `npx playwright install --with-deps chromium` step before tests +**Warning signs:** Works locally (where browsers are cached), fails in fresh CI environment + +## Code Examples + +Verified patterns from official sources: + +### vitest-setup-client.ts +```typescript +// Source: https://vitest.dev/guide/browser/ +/// +/// +``` + +### Package.json Scripts +```json +{ + "scripts": { + "test": "vitest", + "test:unit": "vitest run", + "test:unit:watch": "vitest", + "test:coverage": "vitest run --coverage", + "test:e2e": "playwright test", + "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json" + } +} +``` + +### CI Workflow (Gitea Actions) +```yaml +# Source: https://docs.gitea.com/usage/actions/quickstart +name: Test and Build + +on: + push: + branches: [master, main] + pull_request: + branches: [master, main] + +env: + REGISTRY: git.kube2.tricnet.de + IMAGE_NAME: admin/taskplaner + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Run type check + run: npm run check -- --output machine + + - name: Install Playwright browsers + run: npx playwright install --with-deps chromium + + - name: Run unit tests with coverage + run: npm run test:coverage + + - name: Run E2E tests + run: npm run test:e2e + env: + CI: true + + - name: Upload test artifacts + uses: actions/upload-artifact@v4 + if: always() + with: + name: test-results + path: | + coverage/ + playwright-report/ + test-results/ + + build: + needs: test + runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.merged == true + steps: + # ... existing build steps ... + + notify: + needs: [test, build] + runs-on: ubuntu-latest + if: failure() + steps: + - name: Notify Slack on failure + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + run: | + curl -X POST -H 'Content-type: application/json' \ + --data '{"text":"Pipeline failed for ${{ gitea.repository }} on ${{ gitea.ref }}"}' \ + $SLACK_WEBHOOK_URL +``` + +### Playwright Config for E2E with Screenshots +```typescript +// playwright.config.ts +// Source: https://playwright.dev/docs/test-configuration +import { defineConfig, devices } from '@playwright/test'; + +export default defineConfig({ + testDir: './tests/e2e', + fullyParallel: false, // Sequential for shared database + forbidOnly: !!process.env.CI, + retries: process.env.CI ? 2 : 0, + workers: 1, // Single worker for database tests + reporter: [ + ['html', { open: 'never' }], + ['github'], // GitHub/Gitea compatible annotations + ], + use: { + baseURL: process.env.BASE_URL || 'http://localhost:5173', + trace: 'on-first-retry', + screenshot: 'only-on-failure', + video: 'off', // Per user decision: screenshots only, no video + }, + projects: [ + { + name: 'chromium-desktop', + use: { ...devices['Desktop Chrome'] }, + }, + { + name: 'chromium-mobile', + use: { ...devices['Pixel 5'] }, + }, + ], + webServer: { + command: 'npm run build && npm run preview', + port: 4173, + reuseExistingServer: !process.env.CI, + }, +}); +``` + +### svelte-check CI Output Format +```bash +# Machine-readable output for CI parsing +# Source: https://svelte.dev/docs/cli/sv-check +npx svelte-check --output machine --tsconfig ./tsconfig.json + +# Output format: +# 1590680326283 ERROR "/path/file.svelte" 10:5 "Type error message" +# 1590680326807 COMPLETED 50 FILES 2 ERRORS 0 WARNINGS +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| jsdom + @testing-library/svelte | Vitest browser mode + vitest-browser-svelte | 2025 | Real browser testing, runes support | +| Istanbul coverage | V8 coverage with AST remapping | Vitest 3.2 | 10x faster, same accuracy | +| @playwright/experimental-ct-svelte | vitest-browser-svelte | 2025 | Better integration, official support | +| Jest with svelte-jester | Vitest | 2024 | Native Vite support, faster | + +**Deprecated/outdated:** +- `vitest-svelte-kit` package: Deprecated, no longer needed with modern Vitest +- `@playwright/experimental-ct-svelte`: Use vitest-browser-svelte for component tests instead +- `jsdom` for Svelte 5 components: Does not properly support runes reactivity + +## Open Questions + +Things that couldn't be fully resolved: + +1. **Exact drizzle-seed API for this schema** + - What we know: drizzle-seed works with Drizzle ORM schemas + - What's unclear: Specific configuration for the project's schema structure + - Recommendation: Review drizzle-seed docs during implementation with actual schema + +2. **Gitea Actions Slack notification action availability** + - What we know: GitHub Actions Slack actions exist (rtCamp/action-slack-notify, etc.) + - What's unclear: Whether these work identically in Gitea Actions + - Recommendation: Use direct curl to Slack webhook (shown in examples) for maximum compatibility + +3. **Vitest browser mode stability** + - What we know: Vitest documents browser mode as "experimental" with stable core + - What's unclear: Edge cases in production CI environments + - Recommendation: Pin Vitest version, monitor for issues + +## Sources + +### Primary (HIGH confidence) +- [Svelte Official Testing Docs](https://svelte.dev/docs/svelte/testing) - Official Vitest + browser mode recommendations +- [Vitest Guide](https://vitest.dev/guide/) - Installation, configuration, browser mode +- [Vitest Coverage Config](https://vitest.dev/config/coverage) - Threshold configuration +- [Vitest Browser Mode](https://vitest.dev/guide/browser/) - Playwright provider setup +- [svelte-check CLI](https://svelte.dev/docs/cli/sv-check) - CI output formats +- [Gitea Actions Quickstart](https://docs.gitea.com/usage/actions/quickstart) - Workflow syntax + +### Secondary (MEDIUM confidence) +- [Scott Spence - Vitest Browser Mode Guide](https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide) - Multi-project configuration +- [Mainmatter - Database Fixtures](https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/) - Drizzle seed pattern +- [Roy Bakker - Playwright CI Guide](https://www.roybakker.dev/blog/playwright-in-ci-with-github-actions-and-docker-endtoend-guide) - Artifact upload, caching +- [@testing-library/svelte Setup](https://testing-library.com/docs/svelte-testing-library/setup/) - Alternative jsdom approach + +### Tertiary (LOW confidence) +- Slack webhook notification patterns from various blog posts - curl approach is safest + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - Official Svelte docs explicitly recommend Vitest + browser mode +- Architecture: HIGH - Multi-project pattern documented in Vitest and community guides +- Pitfalls: HIGH - Common issues well-documented in GitHub issues and guides +- E2E fixtures: MEDIUM - Drizzle-seed pattern documented but specific schema integration untested + +**Research date:** 2026-02-03 +**Valid until:** 2026-03-03 (Vitest browser mode evolving, re-verify before major updates)