docs(09): research CI pipeline hardening domain

Phase 9: CI Pipeline Hardening - Standard stack: Vitest + browser mode, Playwright, svelte-check - Architecture: Multi-project config for client/server tests - Pitfalls: jsdom limitations, database parallelism, SvelteKit mocking Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 23:17:33 +01:00
parent 7f3942eb7c
commit 036a81b6de
1 changed files with 503 additions and 0 deletions
--- a/.planning/phases/09-ci-pipeline/09-RESEARCH.md
+++ b/.planning/phases/09-ci-pipeline/09-RESEARCH.md
@@ -0,0 +1,503 @@
 # Phase 9: CI Pipeline Hardening - Research
 **Researched:** 2026-02-03
 **Domain:** Testing infrastructure (Vitest, Playwright, svelte-check) + CI/CD (Gitea Actions)
 **Confidence:** HIGH
 ## Summary
 This phase establishes a comprehensive testing pipeline that blocks deployment on test or type failures. The research covers three testing layers: unit tests (Vitest), type checking (svelte-check), and E2E tests (Playwright), integrated into the existing Gitea Actions workflow.
 The standard approach for SvelteKit testing in 2026 uses Vitest with browser mode for component tests (real browser via Playwright, not jsdom), traditional Vitest for server/utility tests, and standalone Playwright for full E2E tests. The user decisions lock in 80% coverage threshold, Playwright component tests for DOM interactions, and Slack notifications on failure.
 Key finding: Vitest browser mode with `vitest-browser-svelte` is the modern approach for Svelte 5 component testing, replacing the older jsdom + @testing-library/svelte pattern. This provides real browser testing with runes support (`.svelte.test.ts` files).
 **Primary recommendation:** Use multi-project Vitest configuration separating client (browser mode) and server (node) tests, with standalone Playwright for E2E, all gated before Docker build in CI.
 ## Standard Stack
 The established libraries/tools for this domain:
 ### Core
 | Library | Version | Purpose | Why Standard |
 |---------|---------|---------|--------------|
 | vitest | ^3.x | Unit/component test runner | Official SvelteKit recommendation, Vite-native |
 | @vitest/browser | ^3.x | Browser mode for component tests | Real browser testing without jsdom limitations |
 | vitest-browser-svelte | ^0.x | Svelte component rendering in browser mode | Official Svelte 5 support with runes |
 | @vitest/browser-playwright | ^3.x | Playwright provider for Vitest browser mode | Real Chrome DevTools Protocol, not simulated events |
 | @vitest/coverage-v8 | ^3.x | V8-based coverage collection | Fast native coverage, identical accuracy to Istanbul since v3.2 |
 | @playwright/test | ^1.58 | E2E test framework | Already installed, mature E2E solution |
 | svelte-check | ^4.x | TypeScript/Svelte type checking | Already installed, CI-compatible output |
 ### Supporting
 | Library | Version | Purpose | When to Use |
 |---------|---------|---------|-------------|
 | @testing-library/svelte | ^5.x | Alternative component testing | Only if not using browser mode (jsdom fallback) |
 | drizzle-seed | ^0.x | Database seeding for tests | E2E test fixtures with Drizzle ORM |
 ### Alternatives Considered
 | Instead of | Could Use | Tradeoff |
 |------------|-----------|----------|
 | Vitest browser mode | jsdom + @testing-library/svelte | jsdom simulates browser, misses real CSS/runes issues |
 | v8 coverage | istanbul | istanbul 300% slower, v8 now equally accurate |
 | Playwright for E2E | Cypress | Playwright already in project, better multi-browser support |
 **Installation:**
 ```bash
 npm install -D vitest @vitest/browser vitest-browser-svelte @vitest/browser-playwright @vitest/coverage-v8 drizzle-seed
 npx playwright install chromium
 ```
 ## Architecture Patterns
 ### Recommended Project Structure
 ```
 src/
 ├── lib/
 │   ├── components/
 │   │   ├── Button.svelte
 │   │   └── Button.svelte.test.ts    # Component tests (browser mode)
 │   ├── utils/
 │   │   ├── format.ts
 │   │   └── format.test.ts           # Utility tests (node mode)
 │   └── server/
 │       ├── db/
 │       │   └── queries.test.ts      # Server tests (node mode)
 │       └── api.test.ts
 ├── routes/
 │   └── +page.server.test.ts         # Server route tests (node mode)
 tests/
 ├── e2e/                             # Playwright E2E tests
 │   ├── fixtures/
 │   │   └── db.ts                    # Database seeding fixture
 │   ├── user-journeys.spec.ts
 │   └── index.ts                     # Custom test with fixtures
 ├── docker-deployment.spec.ts        # Existing deployment tests
 vitest-setup-client.ts               # Browser mode setup
 vitest.config.ts                     # Multi-project config (or in vite.config.ts)
 playwright.config.ts                 # E2E config (already exists)
 ```
 ### Pattern 1: Multi-Project Vitest Configuration
 **What:** Separate test projects for different environments (browser vs node)
 **When to use:** SvelteKit apps with both client components and server code
 **Example:**
 ```typescript
 // vite.config.ts
 // Source: https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide
 import { sveltekit } from '@sveltejs/kit/vite';
 import tailwindcss from '@tailwindcss/vite';
 import { playwright } from '@vitest/browser-playwright';
 import { defineConfig } from 'vite';
 export default defineConfig({
  plugins: [tailwindcss(), sveltekit()],
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'json', 'html'],
      include: ['src/**/*.{ts,svelte}'],
      exclude: ['src/**/*.test.ts', 'src/**/*.spec.ts'],
      thresholds: {
        global: {
          statements: 80,
          branches: 80,
          functions: 80,
          lines: 80,
        },
      },
    },
    projects: [
      {
        extends: true,
        test: {
          name: 'client',
          testTimeout: 5000,
          browser: {
            enabled: true,
            provider: playwright(),
            instances: [{ browser: 'chromium' }],
          },
          include: ['src/**/*.svelte.{test,spec}.{js,ts}'],
          setupFiles: ['./vitest-setup-client.ts'],
        },
      },
      {
        extends: true,
        test: {
          name: 'server',
          environment: 'node',
          include: ['src/**/*.{test,spec}.{js,ts}'],
          exclude: ['src/**/*.svelte.{test,spec}.{js,ts}'],
        },
      },
    ],
  },
 });
 ```
 ### Pattern 2: Component Test with Runes Support
 **What:** Test Svelte 5 components with $state and $derived in real browser
 **When to use:** Any component using Svelte 5 runes
 **Example:**
 ```typescript
 // src/lib/components/Counter.svelte.test.ts
 // Source: https://svelte.dev/docs/svelte/testing
 import { render } from 'vitest-browser-svelte';
 import { page } from '@vitest/browser/context';
 import { describe, expect, it } from 'vitest';
 import { flushSync } from 'svelte';
 import Counter from './Counter.svelte';
 describe('Counter Component', () => {
  it('increments count on click', async () => {
    render(Counter, { props: { initial: 0 } });
    const button = page.getByRole('button', { name: /increment/i });
    await button.click();
    // flushSync needed for external state changes
    flushSync();
    await expect.element(page.getByText('Count: 1')).toBeInTheDocument();
  });
 });
 ```
 ### Pattern 3: E2E Database Fixture with Drizzle
 **What:** Seed database before tests, clean up after
 **When to use:** E2E tests requiring known data state
 **Example:**
 ```typescript
 // tests/e2e/fixtures/db.ts
 // Source: https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/
 import { test as base } from '@playwright/test';
 import { db } from '../../../src/lib/server/db/index.js';
 import * as schema from '../../../src/lib/server/db/schema.js';
 import { reset, seed } from 'drizzle-seed';
 export const test = base.extend<{
  seededDb: typeof db;
 }>({
  seededDb: async ({}, use) => {
    // Seed with known test data
    await seed(db, schema, { count: 10 });
    await use(db);
    // Clean up after test
    await reset(db, schema);
  },
 });
 export { expect } from '@playwright/test';
 ```
 ### Pattern 4: SvelteKit Module Mocking
 **What:** Mock $app/stores and $app/navigation in unit tests
 **When to use:** Testing components that use SvelteKit-specific imports
 **Example:**
 ```typescript
 // vitest-setup-client.ts
 // Source: https://www.closingtags.com/blog/mocking-svelte-stores-in-vitest
 /// <reference types="@vitest/browser/matchers" />
 import { vi } from 'vitest';
 import { writable } from 'svelte/store';
 // Mock $app/navigation
 vi.mock('$app/navigation', () => ({
  goto: vi.fn(() => Promise.resolve()),
  invalidate: vi.fn(() => Promise.resolve()),
  invalidateAll: vi.fn(() => Promise.resolve()),
  beforeNavigate: vi.fn(),
  afterNavigate: vi.fn(),
 }));
 // Mock $app/stores
 vi.mock('$app/stores', () => ({
  page: writable({
    url: new URL('http://localhost'),
    params: {},
    route: { id: null },
    status: 200,
    error: null,
    data: {},
    form: null,
  }),
  navigating: writable(null),
  updated: { check: vi.fn(), subscribe: writable(false).subscribe },
 }));
 ```
 ### Anti-Patterns to Avoid
 - **Testing with jsdom for Svelte 5 components:** jsdom cannot properly handle runes reactivity. Use browser mode instead.
 - **Parallel E2E tests with shared database:** Will cause race conditions. Set `workers: 1` in playwright.config.ts.
 - **Using deprecated @playwright/experimental-ct-svelte:** Use vitest-browser-svelte instead for component tests.
 - **Mocking everything in E2E tests:** E2E tests should test real integrations. Only mock external services if necessary.
 ## Don't Hand-Roll
 Problems that look simple but have existing solutions:
 | Problem | Don't Build | Use Instead | Why |
 |---------|-------------|-------------|-----|
 | Coverage collection | Custom instrumentation | @vitest/coverage-v8 | Handles source maps, thresholds, reporters automatically |
 | Database seeding | Manual INSERT statements | drizzle-seed | Generates consistent, seeded random data with schema awareness |
 | Component mounting | Manual DOM manipulation | vitest-browser-svelte render() | Handles Svelte 5 lifecycle, context, and cleanup |
 | Screenshot on failure | Custom error handlers | Playwright built-in `screenshot: 'only-on-failure'` | Integrated with test lifecycle and artifacts |
 | CI test output parsing | Regex parsing | svelte-check --output machine | Structured, timestamp-prefixed output designed for CI |
 **Key insight:** The testing ecosystem has mature solutions for all common needs. Hand-rolling any of these leads to edge cases around cleanup, async timing, and framework integration that the official tools have already solved.
 ## Common Pitfalls
 ### Pitfall 1: jsdom Limitations with Svelte 5 Runes
 **What goes wrong:** Tests pass locally but fail to detect reactivity issues, or throw cryptic errors about $state
 **Why it happens:** jsdom simulates browser APIs but doesn't actually run JavaScript in a browser context. Svelte 5 runes compile differently and expect real browser reactivity.
 **How to avoid:** Use Vitest browser mode with Playwright provider for all `.svelte` component tests
 **Warning signs:** Tests involving $state, $derived, or $effect behave inconsistently or require excessive `await tick()`
 ### Pitfall 2: Missing flushSync for External State
 **What goes wrong:** Assertions fail because DOM hasn't updated after state change
 **Why it happens:** Svelte batches updates. When state changes outside component (e.g., store update in test), DOM update is async.
 **How to avoid:** Call `flushSync()` from 'svelte' after modifying external state before asserting
 **Warning signs:** Tests that work with longer timeouts but fail with short ones
 ### Pitfall 3: Parallel E2E with Shared Database
 **What goes wrong:** Flaky tests that sometimes pass, sometimes fail with data conflicts
 **Why it happens:** Multiple test workers modify the same database simultaneously
 **How to avoid:** Set `workers: 1` in playwright.config.ts for E2E tests. Use separate database per worker if parallelism is needed.
 **Warning signs:** Tests pass individually but fail in full suite runs
 ### Pitfall 4: Coverage Threshold Breaking Existing Code
 **What goes wrong:** CI fails immediately after enabling 80% threshold because existing code has 0% coverage
 **Why it happens:** Enabling coverage thresholds on existing codebase without tests
 **How to avoid:** Start with `thresholds: { autoUpdate: true }` to establish baseline, then incrementally raise thresholds as tests are added
 **Warning signs:** Immediate CI failure when coverage is first enabled
 ### Pitfall 5: SvelteKit Module Import Errors
 **What goes wrong:** Tests fail with "Cannot find module '$app/stores'" or similar
 **Why it happens:** $app/* modules are virtual modules provided by SvelteKit at build time, not available in test environment
 **How to avoid:** Mock all $app/* imports in vitest setup file. Keep mocks simple (don't use importOriginal with SvelteKit modules - causes SSR issues).
 **Warning signs:** Import errors mentioning $app, $env, or other SvelteKit virtual modules
 ### Pitfall 6: Playwright Browsers Not Installed in CI
 **What goes wrong:** CI fails with "browserType.launch: Executable doesn't exist"
 **Why it happens:** Playwright browsers need explicit installation, not included in npm install
 **How to avoid:** Add `npx playwright install --with-deps chromium` step before tests
 **Warning signs:** Works locally (where browsers are cached), fails in fresh CI environment
 ## Code Examples
 Verified patterns from official sources:
 ### vitest-setup-client.ts
 ```typescript
 // Source: https://vitest.dev/guide/browser/
 /// <reference types="@vitest/browser/matchers" />
 /// <reference types="@vitest/browser/providers/playwright" />
 ```
 ### Package.json Scripts
 ```json
 {
  "scripts": {
    "test": "vitest",
    "test:unit": "vitest run",
    "test:unit:watch": "vitest",
    "test:coverage": "vitest run --coverage",
    "test:e2e": "playwright test",
    "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json"
  }
 }
 ```
 ### CI Workflow (Gitea Actions)
 ```yaml
 # Source: https://docs.gitea.com/usage/actions/quickstart
 name: Test and Build
 on:
  push:
    branches: [master, main]
  pull_request:
    branches: [master, main]
 env:
  REGISTRY: git.kube2.tricnet.de
  IMAGE_NAME: admin/taskplaner
 jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - name: Install dependencies
        run: npm ci
      - name: Run type check
        run: npm run check -- --output machine
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
      - name: Run unit tests with coverage
        run: npm run test:coverage
      - name: Run E2E tests
        run: npm run test:e2e
        env:
          CI: true
      - name: Upload test artifacts
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: |
            coverage/
            playwright-report/
            test-results/
  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request' || github.event.pull_request.merged == true
    steps:
      # ... existing build steps ...
  notify:
    needs: [test, build]
    runs-on: ubuntu-latest
    if: failure()
    steps:
      - name: Notify Slack on failure
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
        run: |
          curl -X POST -H 'Content-type: application/json' \
            --data '{"text":"Pipeline failed for ${{ gitea.repository }} on ${{ gitea.ref }}"}' \
            $SLACK_WEBHOOK_URL
 ```
 ### Playwright Config for E2E with Screenshots
 ```typescript
 // playwright.config.ts
 // Source: https://playwright.dev/docs/test-configuration
 import { defineConfig, devices } from '@playwright/test';
 export default defineConfig({
  testDir: './tests/e2e',
  fullyParallel: false, // Sequential for shared database
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: 1, // Single worker for database tests
  reporter: [
    ['html', { open: 'never' }],
    ['github'], // GitHub/Gitea compatible annotations
  ],
  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:5173',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'off', // Per user decision: screenshots only, no video
  },
  projects: [
    {
      name: 'chromium-desktop',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'chromium-mobile',
      use: { ...devices['Pixel 5'] },
    },
  ],
  webServer: {
    command: 'npm run build && npm run preview',
    port: 4173,
    reuseExistingServer: !process.env.CI,
  },
 });
 ```
 ### svelte-check CI Output Format
 ```bash
 # Machine-readable output for CI parsing
 # Source: https://svelte.dev/docs/cli/sv-check
 npx svelte-check --output machine --tsconfig ./tsconfig.json
 # Output format:
 # 1590680326283 ERROR "/path/file.svelte" 10:5 "Type error message"
 # 1590680326807 COMPLETED 50 FILES 2 ERRORS 0 WARNINGS
 ```
 ## State of the Art
 | Old Approach | Current Approach | When Changed | Impact |
 |--------------|------------------|--------------|--------|
 | jsdom + @testing-library/svelte | Vitest browser mode + vitest-browser-svelte | 2025 | Real browser testing, runes support |
 | Istanbul coverage | V8 coverage with AST remapping | Vitest 3.2 | 10x faster, same accuracy |
 | @playwright/experimental-ct-svelte | vitest-browser-svelte | 2025 | Better integration, official support |
 | Jest with svelte-jester | Vitest | 2024 | Native Vite support, faster |
 **Deprecated/outdated:**
 - `vitest-svelte-kit` package: Deprecated, no longer needed with modern Vitest
 - `@playwright/experimental-ct-svelte`: Use vitest-browser-svelte for component tests instead
 - `jsdom` for Svelte 5 components: Does not properly support runes reactivity
 ## Open Questions
 Things that couldn't be fully resolved:
 1. **Exact drizzle-seed API for this schema**
   - What we know: drizzle-seed works with Drizzle ORM schemas
   - What's unclear: Specific configuration for the project's schema structure
   - Recommendation: Review drizzle-seed docs during implementation with actual schema
 2. **Gitea Actions Slack notification action availability**
   - What we know: GitHub Actions Slack actions exist (rtCamp/action-slack-notify, etc.)
   - What's unclear: Whether these work identically in Gitea Actions
   - Recommendation: Use direct curl to Slack webhook (shown in examples) for maximum compatibility
 3. **Vitest browser mode stability**
   - What we know: Vitest documents browser mode as "experimental" with stable core
   - What's unclear: Edge cases in production CI environments
   - Recommendation: Pin Vitest version, monitor for issues
 ## Sources
 ### Primary (HIGH confidence)
 - [Svelte Official Testing Docs](https://svelte.dev/docs/svelte/testing) - Official Vitest + browser mode recommendations
 - [Vitest Guide](https://vitest.dev/guide/) - Installation, configuration, browser mode
 - [Vitest Coverage Config](https://vitest.dev/config/coverage) - Threshold configuration
 - [Vitest Browser Mode](https://vitest.dev/guide/browser/) - Playwright provider setup
 - [svelte-check CLI](https://svelte.dev/docs/cli/sv-check) - CI output formats
 - [Gitea Actions Quickstart](https://docs.gitea.com/usage/actions/quickstart) - Workflow syntax
 ### Secondary (MEDIUM confidence)
 - [Scott Spence - Vitest Browser Mode Guide](https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide) - Multi-project configuration
 - [Mainmatter - Database Fixtures](https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/) - Drizzle seed pattern
 - [Roy Bakker - Playwright CI Guide](https://www.roybakker.dev/blog/playwright-in-ci-with-github-actions-and-docker-endtoend-guide) - Artifact upload, caching
 - [@testing-library/svelte Setup](https://testing-library.com/docs/svelte-testing-library/setup/) - Alternative jsdom approach
 ### Tertiary (LOW confidence)
 - Slack webhook notification patterns from various blog posts - curl approach is safest
 ## Metadata
 **Confidence breakdown:**
 - Standard stack: HIGH - Official Svelte docs explicitly recommend Vitest + browser mode
 - Architecture: HIGH - Multi-project pattern documented in Vitest and community guides
 - Pitfalls: HIGH - Common issues well-documented in GitHub issues and guides
 - E2E fixtures: MEDIUM - Drizzle-seed pattern documented but specific schema integration untested
 **Research date:** 2026-02-03
 **Valid until:** 2026-03-03 (Vitest browser mode evolving, re-verify before major updates)