docs(09): research CI pipeline hardening domain

Phase 9: CI Pipeline Hardening - Standard stack: Vitest + browser mode, Playwright, svelte-check - Architecture: Multi-project config for client/server tests - Pitfalls: jsdom limitations, database parallelism, SvelteKit mocking Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 23:17:33 +01:00
parent 7f3942eb7c
commit 036a81b6de
1 changed files with 503 additions and 0 deletions
--- a/.planning/phases/09-ci-pipeline/09-RESEARCH.md
+++ b/.planning/phases/09-ci-pipeline/09-RESEARCH.md
@@ -0,0 +1,503 @@
+# Phase 9: CI Pipeline Hardening - Research
+
+**Researched:** 2026-02-03
+**Domain:** Testing infrastructure (Vitest, Playwright, svelte-check) + CI/CD (Gitea Actions)
+**Confidence:** HIGH
+
+## Summary
+
+This phase establishes a comprehensive testing pipeline that blocks deployment on test or type failures. The research covers three testing layers: unit tests (Vitest), type checking (svelte-check), and E2E tests (Playwright), integrated into the existing Gitea Actions workflow.
+
+The standard approach for SvelteKit testing in 2026 uses Vitest with browser mode for component tests (real browser via Playwright, not jsdom), traditional Vitest for server/utility tests, and standalone Playwright for full E2E tests. The user decisions lock in 80% coverage threshold, Playwright component tests for DOM interactions, and Slack notifications on failure.
+
+Key finding: Vitest browser mode with `vitest-browser-svelte` is the modern approach for Svelte 5 component testing, replacing the older jsdom + @testing-library/svelte pattern. This provides real browser testing with runes support (`.svelte.test.ts` files).
+
+**Primary recommendation:** Use multi-project Vitest configuration separating client (browser mode) and server (node) tests, with standalone Playwright for E2E, all gated before Docker build in CI.
+
+## Standard Stack
+
+The established libraries/tools for this domain:
+
+### Core
+| Library | Version | Purpose | Why Standard |
+|---------|---------|---------|--------------|
+| vitest | ^3.x | Unit/component test runner | Official SvelteKit recommendation, Vite-native |
+| @vitest/browser | ^3.x | Browser mode for component tests | Real browser testing without jsdom limitations |
+| vitest-browser-svelte | ^0.x | Svelte component rendering in browser mode | Official Svelte 5 support with runes |
+| @vitest/browser-playwright | ^3.x | Playwright provider for Vitest browser mode | Real Chrome DevTools Protocol, not simulated events |
+| @vitest/coverage-v8 | ^3.x | V8-based coverage collection | Fast native coverage, identical accuracy to Istanbul since v3.2 |
+| @playwright/test | ^1.58 | E2E test framework | Already installed, mature E2E solution |
+| svelte-check | ^4.x | TypeScript/Svelte type checking | Already installed, CI-compatible output |
+
+### Supporting
+| Library | Version | Purpose | When to Use |
+|---------|---------|---------|-------------|
+| @testing-library/svelte | ^5.x | Alternative component testing | Only if not using browser mode (jsdom fallback) |
+| drizzle-seed | ^0.x | Database seeding for tests | E2E test fixtures with Drizzle ORM |
+
+### Alternatives Considered
+| Instead of | Could Use | Tradeoff |
+|------------|-----------|----------|
+| Vitest browser mode | jsdom + @testing-library/svelte | jsdom simulates browser, misses real CSS/runes issues |
+| v8 coverage | istanbul | istanbul 300% slower, v8 now equally accurate |
+| Playwright for E2E | Cypress | Playwright already in project, better multi-browser support |
+
+**Installation:**
+```bash
+npm install -D vitest @vitest/browser vitest-browser-svelte @vitest/browser-playwright @vitest/coverage-v8 drizzle-seed
+npx playwright install chromium
+```
+
+## Architecture Patterns
+
+### Recommended Project Structure
+```
+src/
+├── lib/
+│   ├── components/
+│   │   ├── Button.svelte
+│   │   └── Button.svelte.test.ts    # Component tests (browser mode)
+│   ├── utils/
+│   │   ├── format.ts
+│   │   └── format.test.ts           # Utility tests (node mode)
+│   └── server/
+│       ├── db/
+│       │   └── queries.test.ts      # Server tests (node mode)
+│       └── api.test.ts
+├── routes/
+│   └── +page.server.test.ts         # Server route tests (node mode)
+tests/
+├── e2e/                             # Playwright E2E tests
+│   ├── fixtures/
+│   │   └── db.ts                    # Database seeding fixture
+│   ├── user-journeys.spec.ts
+│   └── index.ts                     # Custom test with fixtures
+├── docker-deployment.spec.ts        # Existing deployment tests
+vitest-setup-client.ts               # Browser mode setup
+vitest.config.ts                     # Multi-project config (or in vite.config.ts)
+playwright.config.ts                 # E2E config (already exists)
+```
+
+### Pattern 1: Multi-Project Vitest Configuration
+**What:** Separate test projects for different environments (browser vs node)
+**When to use:** SvelteKit apps with both client components and server code
+**Example:**
+```typescript
+// vite.config.ts
+// Source: https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide
+import { sveltekit } from '@sveltejs/kit/vite';
+import tailwindcss from '@tailwindcss/vite';
+import { playwright } from '@vitest/browser-playwright';
+import { defineConfig } from 'vite';
+
+export default defineConfig({
+  plugins: [tailwindcss(), sveltekit()],
+  test: {
+    coverage: {
+      provider: 'v8',
+      reporter: ['text', 'json', 'html'],
+      include: ['src/**/*.{ts,svelte}'],
+      exclude: ['src/**/*.test.ts', 'src/**/*.spec.ts'],
+      thresholds: {
+        global: {
+          statements: 80,
+          branches: 80,
+          functions: 80,
+          lines: 80,
+        },
+      },
+    },
+    projects: [
+      {
+        extends: true,
+        test: {
+          name: 'client',
+          testTimeout: 5000,
+          browser: {
+            enabled: true,
+            provider: playwright(),
+            instances: [{ browser: 'chromium' }],
+          },
+          include: ['src/**/*.svelte.{test,spec}.{js,ts}'],
+          setupFiles: ['./vitest-setup-client.ts'],
+        },
+      },
+      {
+        extends: true,
+        test: {
+          name: 'server',
+          environment: 'node',
+          include: ['src/**/*.{test,spec}.{js,ts}'],
+          exclude: ['src/**/*.svelte.{test,spec}.{js,ts}'],
+        },
+      },
+    ],
+  },
+});
+```
+
+### Pattern 2: Component Test with Runes Support
+**What:** Test Svelte 5 components with $state and $derived in real browser
+**When to use:** Any component using Svelte 5 runes
+**Example:**
+```typescript
+// src/lib/components/Counter.svelte.test.ts
+// Source: https://svelte.dev/docs/svelte/testing
+import { render } from 'vitest-browser-svelte';
+import { page } from '@vitest/browser/context';
+import { describe, expect, it } from 'vitest';
+import { flushSync } from 'svelte';
+import Counter from './Counter.svelte';
+
+describe('Counter Component', () => {
+  it('increments count on click', async () => {
+    render(Counter, { props: { initial: 0 } });
+
+    const button = page.getByRole('button', { name: /increment/i });
+    await button.click();
+
+    // flushSync needed for external state changes
+    flushSync();
+
+    await expect.element(page.getByText('Count: 1')).toBeInTheDocument();
+  });
+});
+```
+
+### Pattern 3: E2E Database Fixture with Drizzle
+**What:** Seed database before tests, clean up after
+**When to use:** E2E tests requiring known data state
+**Example:**
+```typescript
+// tests/e2e/fixtures/db.ts
+// Source: https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/
+import { test as base } from '@playwright/test';
+import { db } from '../../../src/lib/server/db/index.js';
+import * as schema from '../../../src/lib/server/db/schema.js';
+import { reset, seed } from 'drizzle-seed';
+
+export const test = base.extend<{
+  seededDb: typeof db;
+}>({
+  seededDb: async ({}, use) => {
+    // Seed with known test data
+    await seed(db, schema, { count: 10 });
+
+    await use(db);
+
+    // Clean up after test
+    await reset(db, schema);
+  },
+});
+
+export { expect } from '@playwright/test';
+```
+
+### Pattern 4: SvelteKit Module Mocking
+**What:** Mock $app/stores and $app/navigation in unit tests
+**When to use:** Testing components that use SvelteKit-specific imports
+**Example:**
+```typescript
+// vitest-setup-client.ts
+// Source: https://www.closingtags.com/blog/mocking-svelte-stores-in-vitest
+/// <reference types="@vitest/browser/matchers" />
+
+import { vi } from 'vitest';
+import { writable } from 'svelte/store';
+
+// Mock $app/navigation
+vi.mock('$app/navigation', () => ({
+  goto: vi.fn(() => Promise.resolve()),
+  invalidate: vi.fn(() => Promise.resolve()),
+  invalidateAll: vi.fn(() => Promise.resolve()),
+  beforeNavigate: vi.fn(),
+  afterNavigate: vi.fn(),
+}));
+
+// Mock $app/stores
+vi.mock('$app/stores', () => ({
+  page: writable({
+    url: new URL('http://localhost'),
+    params: {},
+    route: { id: null },
+    status: 200,
+    error: null,
+    data: {},
+    form: null,
+  }),
+  navigating: writable(null),
+  updated: { check: vi.fn(), subscribe: writable(false).subscribe },
+}));
+```
+
+### Anti-Patterns to Avoid
+- **Testing with jsdom for Svelte 5 components:** jsdom cannot properly handle runes reactivity. Use browser mode instead.
+- **Parallel E2E tests with shared database:** Will cause race conditions. Set `workers: 1` in playwright.config.ts.
+- **Using deprecated @playwright/experimental-ct-svelte:** Use vitest-browser-svelte instead for component tests.
+- **Mocking everything in E2E tests:** E2E tests should test real integrations. Only mock external services if necessary.
+
+## Don't Hand-Roll
+
+Problems that look simple but have existing solutions:
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| Coverage collection | Custom instrumentation | @vitest/coverage-v8 | Handles source maps, thresholds, reporters automatically |
+| Database seeding | Manual INSERT statements | drizzle-seed | Generates consistent, seeded random data with schema awareness |
+| Component mounting | Manual DOM manipulation | vitest-browser-svelte render() | Handles Svelte 5 lifecycle, context, and cleanup |
+| Screenshot on failure | Custom error handlers | Playwright built-in `screenshot: 'only-on-failure'` | Integrated with test lifecycle and artifacts |
+| CI test output parsing | Regex parsing | svelte-check --output machine | Structured, timestamp-prefixed output designed for CI |
+
+**Key insight:** The testing ecosystem has mature solutions for all common needs. Hand-rolling any of these leads to edge cases around cleanup, async timing, and framework integration that the official tools have already solved.
+
+## Common Pitfalls
+
+### Pitfall 1: jsdom Limitations with Svelte 5 Runes
+**What goes wrong:** Tests pass locally but fail to detect reactivity issues, or throw cryptic errors about $state
+**Why it happens:** jsdom simulates browser APIs but doesn't actually run JavaScript in a browser context. Svelte 5 runes compile differently and expect real browser reactivity.
+**How to avoid:** Use Vitest browser mode with Playwright provider for all `.svelte` component tests
+**Warning signs:** Tests involving $state, $derived, or $effect behave inconsistently or require excessive `await tick()`
+
+### Pitfall 2: Missing flushSync for External State
+**What goes wrong:** Assertions fail because DOM hasn't updated after state change
+**Why it happens:** Svelte batches updates. When state changes outside component (e.g., store update in test), DOM update is async.
+**How to avoid:** Call `flushSync()` from 'svelte' after modifying external state before asserting
+**Warning signs:** Tests that work with longer timeouts but fail with short ones
+
+### Pitfall 3: Parallel E2E with Shared Database
+**What goes wrong:** Flaky tests that sometimes pass, sometimes fail with data conflicts
+**Why it happens:** Multiple test workers modify the same database simultaneously
+**How to avoid:** Set `workers: 1` in playwright.config.ts for E2E tests. Use separate database per worker if parallelism is needed.
+**Warning signs:** Tests pass individually but fail in full suite runs
+
+### Pitfall 4: Coverage Threshold Breaking Existing Code
+**What goes wrong:** CI fails immediately after enabling 80% threshold because existing code has 0% coverage
+**Why it happens:** Enabling coverage thresholds on existing codebase without tests
+**How to avoid:** Start with `thresholds: { autoUpdate: true }` to establish baseline, then incrementally raise thresholds as tests are added
+**Warning signs:** Immediate CI failure when coverage is first enabled
+
+### Pitfall 5: SvelteKit Module Import Errors
+**What goes wrong:** Tests fail with "Cannot find module '$app/stores'" or similar
+**Why it happens:** $app/* modules are virtual modules provided by SvelteKit at build time, not available in test environment
+**How to avoid:** Mock all $app/* imports in vitest setup file. Keep mocks simple (don't use importOriginal with SvelteKit modules - causes SSR issues).
+**Warning signs:** Import errors mentioning $app, $env, or other SvelteKit virtual modules
+
+### Pitfall 6: Playwright Browsers Not Installed in CI
+**What goes wrong:** CI fails with "browserType.launch: Executable doesn't exist"
+**Why it happens:** Playwright browsers need explicit installation, not included in npm install
+**How to avoid:** Add `npx playwright install --with-deps chromium` step before tests
+**Warning signs:** Works locally (where browsers are cached), fails in fresh CI environment
+
+## Code Examples
+
+Verified patterns from official sources:
+
+### vitest-setup-client.ts
+```typescript
+// Source: https://vitest.dev/guide/browser/
+/// <reference types="@vitest/browser/matchers" />
+/// <reference types="@vitest/browser/providers/playwright" />
+```
+
+### Package.json Scripts
+```json
+{
+  "scripts": {
+    "test": "vitest",
+    "test:unit": "vitest run",
+    "test:unit:watch": "vitest",
+    "test:coverage": "vitest run --coverage",
+    "test:e2e": "playwright test",
+    "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json"
+  }
+}
+```
+
+### CI Workflow (Gitea Actions)
+```yaml
+# Source: https://docs.gitea.com/usage/actions/quickstart
+name: Test and Build
+
+on:
+  push:
+    branches: [master, main]
+  pull_request:
+    branches: [master, main]
+
+env:
+  REGISTRY: git.kube2.tricnet.de
+  IMAGE_NAME: admin/taskplaner
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run type check
+        run: npm run check -- --output machine
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps chromium
+
+      - name: Run unit tests with coverage
+        run: npm run test:coverage
+
+      - name: Run E2E tests
+        run: npm run test:e2e
+        env:
+          CI: true
+
+      - name: Upload test artifacts
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: test-results
+          path: |
+            coverage/
+            playwright-report/
+            test-results/
+
+  build:
+    needs: test
+    runs-on: ubuntu-latest
+    if: github.event_name != 'pull_request' || github.event.pull_request.merged == true
+    steps:
+      # ... existing build steps ...
+
+  notify:
+    needs: [test, build]
+    runs-on: ubuntu-latest
+    if: failure()
+    steps:
+      - name: Notify Slack on failure
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+        run: |
+          curl -X POST -H 'Content-type: application/json' \
+            --data '{"text":"Pipeline failed for ${{ gitea.repository }} on ${{ gitea.ref }}"}' \
+            $SLACK_WEBHOOK_URL
+```
+
+### Playwright Config for E2E with Screenshots
+```typescript
+// playwright.config.ts
+// Source: https://playwright.dev/docs/test-configuration
+import { defineConfig, devices } from '@playwright/test';
+
+export default defineConfig({
+  testDir: './tests/e2e',
+  fullyParallel: false, // Sequential for shared database
+  forbidOnly: !!process.env.CI,
+  retries: process.env.CI ? 2 : 0,
+  workers: 1, // Single worker for database tests
+  reporter: [
+    ['html', { open: 'never' }],
+    ['github'], // GitHub/Gitea compatible annotations
+  ],
+  use: {
+    baseURL: process.env.BASE_URL || 'http://localhost:5173',
+    trace: 'on-first-retry',
+    screenshot: 'only-on-failure',
+    video: 'off', // Per user decision: screenshots only, no video
+  },
+  projects: [
+    {
+      name: 'chromium-desktop',
+      use: { ...devices['Desktop Chrome'] },
+    },
+    {
+      name: 'chromium-mobile',
+      use: { ...devices['Pixel 5'] },
+    },
+  ],
+  webServer: {
+    command: 'npm run build && npm run preview',
+    port: 4173,
+    reuseExistingServer: !process.env.CI,
+  },
+});
+```
+
+### svelte-check CI Output Format
+```bash
+# Machine-readable output for CI parsing
+# Source: https://svelte.dev/docs/cli/sv-check
+npx svelte-check --output machine --tsconfig ./tsconfig.json
+
+# Output format:
+# 1590680326283 ERROR "/path/file.svelte" 10:5 "Type error message"
+# 1590680326807 COMPLETED 50 FILES 2 ERRORS 0 WARNINGS
+```
+
+## State of the Art
+
+| Old Approach | Current Approach | When Changed | Impact |
+|--------------|------------------|--------------|--------|
+| jsdom + @testing-library/svelte | Vitest browser mode + vitest-browser-svelte | 2025 | Real browser testing, runes support |
+| Istanbul coverage | V8 coverage with AST remapping | Vitest 3.2 | 10x faster, same accuracy |
+| @playwright/experimental-ct-svelte | vitest-browser-svelte | 2025 | Better integration, official support |
+| Jest with svelte-jester | Vitest | 2024 | Native Vite support, faster |
+
+**Deprecated/outdated:**
+- `vitest-svelte-kit` package: Deprecated, no longer needed with modern Vitest
+- `@playwright/experimental-ct-svelte`: Use vitest-browser-svelte for component tests instead
+- `jsdom` for Svelte 5 components: Does not properly support runes reactivity
+
+## Open Questions
+
+Things that couldn't be fully resolved:
+
+1. **Exact drizzle-seed API for this schema**
+   - What we know: drizzle-seed works with Drizzle ORM schemas
+   - What's unclear: Specific configuration for the project's schema structure
+   - Recommendation: Review drizzle-seed docs during implementation with actual schema
+
+2. **Gitea Actions Slack notification action availability**
+   - What we know: GitHub Actions Slack actions exist (rtCamp/action-slack-notify, etc.)
+   - What's unclear: Whether these work identically in Gitea Actions
+   - Recommendation: Use direct curl to Slack webhook (shown in examples) for maximum compatibility
+
+3. **Vitest browser mode stability**
+   - What we know: Vitest documents browser mode as "experimental" with stable core
+   - What's unclear: Edge cases in production CI environments
+   - Recommendation: Pin Vitest version, monitor for issues
+
+## Sources
+
+### Primary (HIGH confidence)
+- [Svelte Official Testing Docs](https://svelte.dev/docs/svelte/testing) - Official Vitest + browser mode recommendations
+- [Vitest Guide](https://vitest.dev/guide/) - Installation, configuration, browser mode
+- [Vitest Coverage Config](https://vitest.dev/config/coverage) - Threshold configuration
+- [Vitest Browser Mode](https://vitest.dev/guide/browser/) - Playwright provider setup
+- [svelte-check CLI](https://svelte.dev/docs/cli/sv-check) - CI output formats
+- [Gitea Actions Quickstart](https://docs.gitea.com/usage/actions/quickstart) - Workflow syntax
+
+### Secondary (MEDIUM confidence)
+- [Scott Spence - Vitest Browser Mode Guide](https://scottspence.com/posts/testing-with-vitest-browser-svelte-guide) - Multi-project configuration
+- [Mainmatter - Database Fixtures](https://mainmatter.com/blog/2025/08/21/mock-database-in-svelte-tests/) - Drizzle seed pattern
+- [Roy Bakker - Playwright CI Guide](https://www.roybakker.dev/blog/playwright-in-ci-with-github-actions-and-docker-endtoend-guide) - Artifact upload, caching
+- [@testing-library/svelte Setup](https://testing-library.com/docs/svelte-testing-library/setup/) - Alternative jsdom approach
+
+### Tertiary (LOW confidence)
+- Slack webhook notification patterns from various blog posts - curl approach is safest
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH - Official Svelte docs explicitly recommend Vitest + browser mode
+- Architecture: HIGH - Multi-project pattern documented in Vitest and community guides
+- Pitfalls: HIGH - Common issues well-documented in GitHub issues and guides
+- E2E fixtures: MEDIUM - Drizzle-seed pattern documented but specific schema integration untested
+
+**Research date:** 2026-02-03
+**Valid until:** 2026-03-03 (Vitest browser mode evolving, re-verify before major updates)