taskplaner/.planning/research/STACK-v2-cicd-observability.md

# Technology Stack: CI/CD Testing, ArgoCD GitOps, and Observability

**Project:** TaskPlanner v2.0 Production Operations
**Researched:** 2026-02-03
**Scope:** Stack additions for existing k3s-deployed SvelteKit app

## Executive Summary

This research covers three areas: (1) adding tests to the existing Gitea Actions pipeline, (2) ArgoCD for GitOps deployment automation, and (3) Prometheus/Grafana/Loki observability. The existing setup already has ArgoCD configured; research focuses on validating that configuration and adding the observability stack.

**Key finding:** Promtail is EOL on 2026-03-02. Use Grafana Alloy instead for log collection.

---

## 1. CI/CD Testing Stack

### Recommended Stack

| Component | Version | Purpose | Rationale |
|-----------|---------|---------|-----------|
| Playwright | ^1.58.1 (existing) | E2E testing | Already configured, comprehensive browser automation |
| Vitest | ^3.0.0 | Unit/component tests | Official Svelte recommendation for Vite-based projects |
| @testing-library/svelte | ^5.0.0 | Component testing utilities | Streamlined component assertions |
| mcr.microsoft.com/playwright | v1.58.1 | CI browser execution | Pre-installed browsers, eliminates install step |

### Why This Stack

**Playwright (keep existing):** Already configured with `playwright.config.ts` and `tests/docker-deployment.spec.ts`. The existing tests cover critical paths: health endpoint, CSRF-protected form submissions, and data persistence. Extend rather than replace.

**Vitest (add):** Svelte officially recommends Vitest for unit and component testing when using Vite (which SvelteKit uses). Vitest shares Vite's config, eliminating configuration overhead. Jest muscle memory transfers directly.

**NOT recommended:**
- Jest: Requires separate configuration, slower than Vitest, no Vite integration
- Cypress: Overlaps with Playwright; adding both creates maintenance burden
- @vitest/browser with Playwright: Adds complexity; save for later if jsdom proves insufficient

### Gitea Actions Workflow Updates

The existing workflow at `.gitea/workflows/build.yaml` needs a test stage. Gitea Actions uses GitHub Actions syntax.

**Recommended workflow structure:**

```yaml
name: Build and Push

on:
  push:
    branches: [master, main]
  pull_request:
    branches: [master, main]

env:
  REGISTRY: git.kube2.tricnet.de
  IMAGE_NAME: tho/taskplaner

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.58.1-noble
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm ci

      - name: Run type check
        run: npm run check

      - name: Run unit tests
        run: npm run test:unit

      - name: Run E2E tests
        run: npm run test:e2e
        env:
          CI: true

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    steps:
      # ... existing build steps ...
```

**Key decisions:**
- Use Playwright Docker image to avoid browser installation (saves 2-3 minutes)
- Run tests before build to fail fast
- Only build/push on push to master, not PRs
- Type checking (`svelte-check`) catches errors before runtime

### Package.json Scripts to Add

```json
{
  "scripts": {
    "test": "npm run test:unit && npm run test:e2e",
    "test:unit": "vitest run",
    "test:unit:watch": "vitest",
    "test:e2e": "playwright test",
    "test:e2e:docker": "BASE_URL=http://localhost:3000 playwright test tests/docker-deployment.spec.ts"
  }
}
```

### Installation

```bash
# Add Vitest and testing utilities
npm install -D vitest @testing-library/svelte jsdom
```

### Vitest Configuration

Create `vitest.config.ts`:

```typescript
import { defineConfig } from 'vitest/config';
import { sveltekit } from '@sveltejs/kit/vite';

export default defineConfig({
  plugins: [sveltekit()],
  test: {
    include: ['src/**/*.{test,spec}.{js,ts}'],
    environment: 'jsdom',
    globals: true,
    setupFiles: ['./src/test-setup.ts']
  }
});
```

### Confidence: HIGH

Sources:
- [Svelte Testing Documentation](https://svelte.dev/docs/svelte/testing) - Official recommendation for Vitest
- [Playwright CI Setup](https://playwright.dev/docs/ci-intro) - Docker image and CI best practices
- Existing `playwright.config.ts` in project

---

## 2. ArgoCD GitOps Stack

### Current State

ArgoCD is already configured in `argocd/application.yaml`. The configuration is correct and follows best practices:

```yaml
syncPolicy:
  automated:
    prune: true      # Removes resources deleted from Git
    selfHeal: true   # Reverts manual changes
```

### Recommended Stack

| Component | Version | Purpose | Rationale |
|-----------|---------|---------|-----------|
| ArgoCD Helm Chart | 9.4.0 | GitOps controller | Latest stable, deploys ArgoCD v3.3.0 |

### What's Already Done (No Changes Needed)

1. **Application manifest:** `argocd/application.yaml` correctly points to `helm/taskplaner`
2. **Auto-sync enabled:** `automated.prune` and `selfHeal` are configured
3. **Git-based image tags:** Pipeline updates `values.yaml` with new image tag
4. **Namespace creation:** `CreateNamespace=true` is set

### What May Need Verification

1. **ArgoCD installation:** Verify ArgoCD is actually deployed on the k3s cluster
2. **Repository credentials:** If the Gitea repo is private, ArgoCD needs credentials
3. **Registry secret:** The `gitea-registry-secret` placeholder needs real credentials

### Installation (if ArgoCD not yet installed)

```bash
# Add ArgoCD Helm repository
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# Install ArgoCD (minimal for single-node k3s)
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --set server.service.type=ClusterIP \
  --set configs.params.server\.insecure=true  # If behind Traefik TLS termination
```

### Apply Application

```bash
kubectl apply -f argocd/application.yaml
```

### NOT Recommended

- **ArgoCD Image Updater:** Overkill for single-app deployment; the current approach of updating values.yaml in Git is simpler and provides better audit trail
- **ApplicationSets:** Unnecessary for single environment
- **App of Apps pattern:** Unnecessary complexity for one application

### Confidence: HIGH

Sources:
- [ArgoCD Helm Chart on Artifact Hub](https://artifacthub.io/packages/helm/argo/argo-cd) - Version 9.4.0 confirmed
- [ArgoCD Helm GitHub Releases](https://github.com/argoproj/argo-helm/releases) - Release notes
- Existing `argocd/application.yaml` in project

---

## 3. Observability Stack

### Recommended Stack

| Component | Chart | Version | Purpose |
|-----------|-------|---------|---------|
| kube-prometheus-stack | prometheus-community/kube-prometheus-stack | 81.4.2 | Prometheus + Grafana + Alertmanager |
| Loki | grafana/loki | 6.51.0 | Log aggregation (monolithic mode) |
| Grafana Alloy | grafana/alloy | 1.5.3 | Log collection agent |

### Why This Stack

**kube-prometheus-stack (not standalone Prometheus):** Single chart deploys Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics. Pre-configured with Kubernetes dashboards. This is the standard approach.

**Loki (not ELK/Elasticsearch):** "Like Prometheus, but for logs." Integrates natively with Grafana. Much lower resource footprint than Elasticsearch. Uses same label-based querying as Prometheus.

**Grafana Alloy (not Promtail):** CRITICAL - Promtail reaches End-of-Life on 2026-03-02 (next month). Grafana Alloy is the official replacement. It's based on OpenTelemetry Collector and supports logs, metrics, and traces in one agent.

### NOT Recommended

- **Promtail:** EOL 2026-03-02. Do not install; use Alloy
- **loki-stack Helm chart:** Deprecated, no longer maintained
- **Elasticsearch/ELK:** Resource-heavy, complex, overkill for single-user app
- **Loki microservices mode:** Requires 3+ nodes, object storage; overkill for personal app
- **Separate Prometheus + Grafana charts:** kube-prometheus-stack bundles them correctly

### Architecture

```
                                    +------------------+
                                    |     Grafana      |
                                    | (Dashboards/UI)  |
                                    +--------+---------+
                                             |
                        +--------------------+--------------------+
                        |                                         |
               +--------v---------+                    +----------v---------+
               |    Prometheus    |                    |       Loki         |
               |    (Metrics)     |                    |      (Logs)        |
               +--------+---------+                    +----------+---------+
                        |                                         |
         +--------------+---------------+                         |
         |              |               |                         |
   +-----v-----+  +-----v-----+  +------v------+         +--------v---------+
   |  node-    |  |  kube-    |  | TaskPlanner |         |   Grafana Alloy  |
   |  exporter |  |  state-   |  |   /metrics  |         |  (Log Shipper)   |
   |           |  |  metrics  |  |             |         |                  |
   +-----------+  +-----------+  +-------------+         +------------------+
```

### Installation

```bash
# Add Helm repositories
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Create monitoring namespace
kubectl create namespace monitoring

# Install kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml

# Install Loki (monolithic mode for single-node)
helm install loki grafana/loki \
  --namespace monitoring \
  --values loki-values.yaml

# Install Alloy for log collection
helm install alloy grafana/alloy \
  --namespace monitoring \
  --values alloy-values.yaml
```

### Recommended Values Files

#### prometheus-values.yaml (minimal for k3s single-node)

```yaml
# Reduce resource usage for single-node k3s
prometheus:
  prometheusSpec:
    retention: 15d
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 2Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: longhorn  # Use existing Longhorn
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

alertmanager:
  alertmanagerSpec:
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 256Mi
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: longhorn
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

grafana:
  persistence:
    enabled: true
    storageClassName: longhorn
    size: 5Gi
  # Grafana will be exposed via Traefik
  ingress:
    enabled: true
    ingressClassName: traefik
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
      - grafana.kube2.tricnet.de
    tls:
      - secretName: grafana-tls
        hosts:
          - grafana.kube2.tricnet.de

# Disable components not needed for single-node
kubeControllerManager:
  enabled: false  # k3s bundles this differently
kubeScheduler:
  enabled: false  # k3s bundles this differently
kubeProxy:
  enabled: false  # k3s uses different proxy
```

#### loki-values.yaml (monolithic mode)

```yaml
deploymentMode: SingleBinary

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

singleBinary:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi
  persistence:
    enabled: true
    storageClass: longhorn
    size: 10Gi

# Disable components not needed for monolithic
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

# Gateway not needed for internal access
gateway:
  enabled: false
```

#### alloy-values.yaml

```yaml
alloy:
  configMap:
    content: |-
      // Discover and collect logs from all pods
      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pods" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
      }

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.write.local.receiver]
      }

      loki.write "local" {
        endpoint {
          url = "http://loki.monitoring.svc:3100/loki/api/v1/push"
        }
      }

controller:
  type: daemonset

resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 256Mi
```

### TaskPlanner Metrics Endpoint

The app needs a `/metrics` endpoint for Prometheus to scrape. SvelteKit options:

1. **prom-client library** (recommended): Standard Prometheus client for Node.js
2. **Custom endpoint**: Simple counter/gauge implementation

Add to `package.json`:
```bash
npm install prom-client
```

Add ServiceMonitor for Prometheus to scrape TaskPlanner:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: taskplaner
  namespace: monitoring
  labels:
    release: prometheus  # Must match Prometheus selector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: taskplaner
  namespaceSelector:
    matchNames:
      - default
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
```

### Resource Summary

Total additional resource requirements for observability:

| Component | CPU Request | Memory Request | Storage |
|-----------|-------------|----------------|---------|
| Prometheus | 200m | 512Mi | 20Gi |
| Alertmanager | 50m | 64Mi | 5Gi |
| Grafana | 100m | 128Mi | 5Gi |
| Loki | 100m | 256Mi | 10Gi |
| Alloy (per node) | 50m | 64Mi | - |
| **Total** | ~500m | ~1Gi | 40Gi |

This fits comfortably on a single k3s node with 4+ cores and 8GB+ RAM.

### Confidence: HIGH

Sources:
- [kube-prometheus-stack on Artifact Hub](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack) - Version 81.4.2
- [Grafana Loki Helm Installation](https://grafana.com/docs/loki/latest/setup/install/helm/) - Monolithic mode guidance
- [Grafana Alloy Kubernetes Deployment](https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/) - Alloy setup
- [Promtail Deprecation Notice](https://grafana.com/docs/loki/latest/send-data/promtail/installation/) - EOL 2026-03-02
- [Migrate from Promtail to Alloy](https://grafana.com/docs/alloy/latest/set-up/migrate/from-promtail/) - Migration guide

---

## Summary: What to Install

### Immediate Actions

| Category | Add | Version | Notes |
|----------|-----|---------|-------|
| Testing | vitest | ^3.0.0 | Unit tests |
| Testing | @testing-library/svelte | ^5.0.0 | Component testing |
| Metrics | prom-client | ^15.0.0 | Prometheus metrics from app |

### Helm Charts to Deploy

| Chart | Repository | Version | Namespace |
|-------|------------|---------|-----------|
| kube-prometheus-stack | prometheus-community | 81.4.2 | monitoring |
| loki | grafana | 6.51.0 | monitoring |
| alloy | grafana | 1.5.3 | monitoring |

### Already Configured (Verify, Don't Re-install)

| Component | Status | Action |
|-----------|--------|--------|
| ArgoCD Application | Configured in `argocd/application.yaml` | Verify ArgoCD is running |
| Playwright | Configured in `playwright.config.ts` | Keep, extend tests |

### Do NOT Install

| Component | Reason |
|-----------|--------|
| Promtail | EOL 2026-03-02, use Alloy instead |
| loki-stack chart | Deprecated, unmaintained |
| Elasticsearch/ELK | Overkill, resource-heavy |
| Jest | Vitest is better for Vite projects |
| ArgoCD Image Updater | Current Git-based approach is simpler |

---

## Helm Repository Commands

```bash
# Add all needed repositories
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# Verify
helm search repo prometheus-community/kube-prometheus-stack
helm search repo grafana/loki
helm search repo grafana/alloy
helm search repo argo/argo-cd
```

---

## Sources

### Official Documentation
- [Svelte Testing](https://svelte.dev/docs/svelte/testing)
- [Playwright CI Setup](https://playwright.dev/docs/ci-intro)
- [ArgoCD Helm Chart](https://artifacthub.io/packages/helm/argo/argo-cd)
- [kube-prometheus-stack](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack)
- [Grafana Loki Helm](https://grafana.com/docs/loki/latest/setup/install/helm/)
- [Grafana Alloy](https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/)

### Critical Updates
- [Promtail EOL Notice](https://grafana.com/docs/loki/latest/send-data/promtail/installation/) - EOL 2026-03-02
- [Promtail to Alloy Migration](https://grafana.com/docs/alloy/latest/set-up/migrate/from-promtail/)