2057ccd816
- Introduced health checks for Nextcloud, Immich, and Jitsi in the .env.example file. - Implemented Prometheus metrics for HTTP requests, IMAP sync, outbox processing, and webhook executions. - Added Grafana configuration files for dashboards and data sources. - Updated Docker Compose to include Prometheus and Grafana services. - Enhanced logging middleware to include request IDs and metrics tracking. - Created health checker for monitoring database and external service statuses. - Updated README with observability setup instructions and service URLs.
159 lines
7.2 KiB
Markdown
159 lines
7.2 KiB
Markdown
# Ulti Backend
|
||
|
||
Backend monolithe Go orchestrant la Ulti Suite — alternative souveraine à Google Suite.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ nginx (reverse proxy unique, port 80) │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ ultid (Go monolithe) │
|
||
│ ├─ /api/v1/mail — Ultimail (IMAP/SMTP custom) │
|
||
│ ├─ /api/v1/drive — Ultidrive (→ Nextcloud WebDAV) │
|
||
│ ├─ /api/v1/calendar — Agenda (→ Nextcloud CalDAV) │
|
||
│ ├─ /api/v1/contacts — Contacts (→ Nextcloud CardDAV) │
|
||
│ ├─ /api/v1/meet — Ultimeet (→ Jitsi JWT) │
|
||
│ ├─ /api/v1/photos — Ultiphotos (→ Immich API) │
|
||
│ ├─ /api/v1/admin — Administration │
|
||
│ ├─ /api/v1/search — Recherche transversale │
|
||
│ └─ /ws — WebSocket realtime │
|
||
├─────────────────────────────────────────────────────────┤
|
||
│ Services tiers (Docker) │
|
||
│ ├─ PostgreSQL 16 — DB partagée │
|
||
│ ├─ KeyDB — Cache/sessions │
|
||
│ ├─ RustFS — Object storage S3 │
|
||
│ ├─ Authentik — Auth OIDC/SAML │
|
||
│ ├─ Nextcloud (nginx+FPM) — Drive/Cal/Contacts (/cloud) │
|
||
│ ├─ Jitsi — Visioconférence │
|
||
│ └─ Immich — Photos/ML │
|
||
└─────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
# 1. Copy environment file
|
||
cp .env.example .env
|
||
# Edit secrets once at the top of .env (POSTGRES_PASSWORD, RUSTFS_SECRET_KEY, etc.)
|
||
# Other variables use {{VAR}} placeholders expanded at launch.
|
||
# Toggle modules with flags:
|
||
# NEXTCLOUD_ENABLED=true|false
|
||
# JITSI_ENABLED=true|false
|
||
# IMMICH_ENABLED=true|false
|
||
|
||
# 2. Start stack (core + modules enabled by flags)
|
||
./deploy/compose-up.sh up -d
|
||
```
|
||
|
||
## Development
|
||
|
||
```bash
|
||
# Run locally (needs PG, KeyDB, RustFS running; loads .env with {{VAR}} expansion)
|
||
go run ./cmd/ultid
|
||
|
||
# Build
|
||
go build -o ultid ./cmd/ultid
|
||
|
||
# Expand .env for external tools (docker, migrate)
|
||
go run ./cmd/envexpand -in .env -out .env.resolved
|
||
source <(grep -v '^#' .env.resolved | sed 's/^/export /')
|
||
|
||
# Run migrations (use expanded ULTID_DB_URL; host may need localhost instead of postgres)
|
||
migrate -path migrations -database "$ULTID_DB_URL" up
|
||
```
|
||
|
||
### Reverse proxy (nginx)
|
||
|
||
Un seul **nginx** expose l’entrée HTTP (`:80`) et route :
|
||
|
||
| Chemin | Service |
|
||
|--------|---------|
|
||
| `/api/*` | ultid |
|
||
| `/ws` | ultid (WebSocket) |
|
||
| `/auth/*` | Authentik |
|
||
| `/meet/*` | Jitsi (si `JITSI_ENABLED=true`) |
|
||
| `/cloud/*` | Nextcloud nginx+FPM (si `NEXTCLOUD_ENABLED=true`) |
|
||
|
||
Nextcloud : FPM + nginx dédié ; ultid appelle `NEXTCLOUD_URL` en interne (`http://nextcloud:80`).
|
||
Caddy retiré : un seul proxy évite la double couche ; TLS plus tard (certbot, Traefik, ou `listen 443` nginx).
|
||
|
||
### Centralized secrets
|
||
|
||
Set passwords and keys **once** in the `Secrets` section at the top of `.env`. Derived values reference them with `{{POSTGRES_PASSWORD}}`, `{{RUSTFS_SECRET_KEY}}`, etc. Expansion runs via:
|
||
|
||
- `./deploy/compose-up.sh` — writes `.env.resolved` for Docker Compose
|
||
- `go run ./cmd/envexpand -in .env -out .env.resolved` — manual export for migrate/scripts
|
||
- `go run ./cmd/ultid` — expands `.env` in-process before reading config
|
||
|
||
Runtime secret files are also supported with `*_FILE` variables (example: `ULTID_OIDC_CLIENT_SECRET_FILE=/run/secrets/oidc_client_secret`).
|
||
|
||
Mail credentials are encrypted at rest with AES-GCM using `MAIL_CREDENTIAL_KEYS` (`key_id:base64key,...`) and `MAIL_ACTIVE_CREDENTIAL_KEY_ID`.
|
||
Secret rotation policy is enforced through:
|
||
|
||
- `SECRET_ROTATION_MAX_AGE`
|
||
- `ULTID_OIDC_CLIENT_SECRET_ROTATED_AT`
|
||
- `MAIL_CREDENTIAL_KEY_ROTATED_AT`
|
||
- `MAIL_WEBHOOK_SHARED_SECRET_ROTATED_AT`
|
||
|
||
### Observability (Prometheus / Grafana)
|
||
|
||
ultid exposes Prometheus metrics at `/metrics` (see `internal/observability/metrics.go`). The core Docker Compose stack includes Prometheus and Grafana with configs under `deploy/observability/`:
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `deploy/observability/prometheus/prometheus.yml` | Scrape ultid + self; loads alert rules |
|
||
| `deploy/observability/prometheus/alerts.yml` | Alert rules: IMAP sync stalled, outbox backlog, HTTP 5xx rate |
|
||
| `deploy/observability/grafana/ultid-baseline.json` | Baseline dashboard (HTTP latency/errors, IMAP sync, outbox, webhooks) |
|
||
| `deploy/observability/grafana/provisioning/` | Grafana datasource + dashboard auto-load |
|
||
|
||
Start with the rest of the stack (`./deploy/compose-up.sh up -d`), then open:
|
||
|
||
| Service | URL | Notes |
|
||
|---------|-----|-------|
|
||
| Prometheus | http://localhost:9090 | Targets: `ultid`, `prometheus` |
|
||
| Grafana | http://localhost:3000 | Login from `.env` (`GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD`, default `admin` / `admin`); dashboard **Ultid Baseline** under folder **Ultid** |
|
||
|
||
**Alertmanager** — not included in compose; route labels `service=ultid` and `severity` (`critical`, `warning`) to your on-call channels when you add it.
|
||
|
||
## Stack
|
||
|
||
| Component | Technology |
|
||
|-----------|-----------|
|
||
| Backend | Go 1.23+ (chi, pgx, go-imap, go-smtp) |
|
||
| Database | PostgreSQL 16 |
|
||
| Cache | KeyDB (Redis-compatible, multi-threaded) |
|
||
| Object Storage | RustFS (S3-compatible, Apache 2.0) |
|
||
| Auth | Authentik (OIDC/SAML) |
|
||
| Files/Cal/Contacts | Nextcloud headless (WebDAV/CalDAV/CardDAV) |
|
||
| Video | Jitsi Meet (JWT auth) |
|
||
| Photos | Immich (ML classification) |
|
||
| Reverse Proxy | nginx (TLS à ajouter via certbot ou autre) |
|
||
| Search | PostgreSQL tsvector + GIN |
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
├── cmd/ultid/ — Entry point
|
||
├── internal/
|
||
│ ├── api/ — HTTP handlers (mail, admin, drive, calendar, contacts, meet, photos)
|
||
│ ├── mail/ — IMAP sync, SMTP send, rules engine, webhooks
|
||
│ ├── nextcloud/ — WebDAV/CalDAV/CardDAV client
|
||
│ ├── meet/ — Jitsi JWT generation
|
||
│ ├── photos/ — Immich API client
|
||
│ ├── auth/ — OIDC verification
|
||
│ ├── search/ — Full-text search
|
||
│ ├── realtime/ — WebSocket hub
|
||
│ └── config/ — Environment config
|
||
├── migrations/ — SQL migrations
|
||
├── deploy/ — Docker Compose configs
|
||
│ ├── docker-compose.yml — Core stack
|
||
│ ├── observability/ — Prometheus alerts + Grafana dashboard
|
||
│ ├── nginx/
|
||
│ ├── nextcloud/
|
||
│ ├── jitsi/
|
||
│ └── immich/
|
||
├── Dockerfile — Multi-stage build
|
||
└── .env.example
|
||
```
|