Complete backend for the Ulti-Suite
Go to file
R3D347HR4Y f232aaf960 Enhance Contacts API with new features and improvements
- Updated the Contacts API to support contact synchronization with incremental updates using sync tokens.
- Added functionality for merging duplicate contacts on the server side.
- Introduced new endpoints for enriching contact interactions, including mail, meetings, and files.
- Implemented ETag support for contact updates to ensure data integrity.
- Enhanced validation for sync tokens and interaction queries.
- Updated project checklist to reflect the completion of Contacts API enhancements.
2026-05-22 20:50:46 +02:00
.github/workflows Add CI workflow and unit tests for mail API 2026-05-22 17:02:37 +02:00
cmd Enhance Contacts API with new features and improvements 2026-05-22 20:50:46 +02:00
deploy Add observability features with Prometheus and Grafana integration 2026-05-22 16:17:10 +02:00
internal Enhance Contacts API with new features and improvements 2026-05-22 20:50:46 +02:00
migrations Implement rule simulation and webhook enhancements 2026-05-22 17:53:51 +02:00
project-plan Enhance Contacts API with new features and improvements 2026-05-22 20:50:46 +02:00
.dockerignore Initialize Ulti Backend project with Docker setup, environment configuration, and core services. Added .dockerignore, .env.example, Dockerfile, and docker-compose files for PostgreSQL, KeyDB, RustFS, Authentik, Nextcloud, Jitsi, and Immich. Implemented main application structure in Go with API handlers and environment variable expansion. Included README for project overview and setup instructions. 2026-05-22 16:02:53 +02:00
.env.example Enhance search functionality with multi-engine support and configuration updates 2026-05-22 19:14:27 +02:00
.gitignore Initialize Ulti Backend project with Docker setup, environment configuration, and core services. Added .dockerignore, .env.example, Dockerfile, and docker-compose files for PostgreSQL, KeyDB, RustFS, Authentik, Nextcloud, Jitsi, and Immich. Implemented main application structure in Go with API handlers and environment variable expansion. Included README for project overview and setup instructions. 2026-05-22 16:02:53 +02:00
Dockerfile Initialize Ulti Backend project with Docker setup, environment configuration, and core services. Added .dockerignore, .env.example, Dockerfile, and docker-compose files for PostgreSQL, KeyDB, RustFS, Authentik, Nextcloud, Jitsi, and Immich. Implemented main application structure in Go with API handlers and environment variable expansion. Included README for project overview and setup instructions. 2026-05-22 16:02:53 +02:00
go.mod Enhance mail API with rate limiting, idempotency, and attachment management 2026-05-22 17:19:16 +02:00
go.sum Enhance mail API with rate limiting, idempotency, and attachment management 2026-05-22 17:19:16 +02:00
README.md Implement Calendar API enhancements with new endpoints and features 2026-05-22 20:29:53 +02:00

Ulti Backend

Backend monolithe Go orchestrant la Ulti Suite — alternative souveraine à Google Suite.

Architecture

┌─────────────────────────────────────────────────────────┐
│  nginx (reverse proxy unique, port 80)                  │
├─────────────────────────────────────────────────────────┤
│  ultid (Go monolithe)                                   │
│  ├─ /api/v1/mail      — Ultimail (IMAP/SMTP custom)    │
│  ├─ /api/v1/drive     — Ultidrive (→ Nextcloud WebDAV) │
│  ├─ /api/v1/calendar  — Agenda (→ Nextcloud CalDAV)    │
│  ├─ /api/v1/contacts  — Contacts (→ Nextcloud CardDAV) │
│  ├─ /api/v1/meet      — Ultimeet (→ Jitsi JWT)        │
│  ├─ /api/v1/photos    — Ultiphotos (→ Immich API)     │
│  ├─ /api/v1/admin     — Administration                 │
│  ├─ /api/v1/search    — Recherche transversale         │
│  └─ /ws               — WebSocket realtime             │
├─────────────────────────────────────────────────────────┤
│  Services tiers (Docker)                                │
│  ├─ PostgreSQL 16     — DB partagée                    │
│  ├─ KeyDB             — Cache/sessions                  │
│  ├─ RustFS            — Object storage S3              │
│  ├─ Authentik         — Auth OIDC/SAML                 │
│  ├─ Nextcloud (nginx+FPM) — Drive/Cal/Contacts (/cloud) │
│  ├─ Jitsi             — Visioconférence                │
│  └─ Immich            — Photos/ML                      │
└─────────────────────────────────────────────────────────┘

Quick Start

# 1. Copy environment file
cp .env.example .env
# Edit secrets once at the top of .env (POSTGRES_PASSWORD, RUSTFS_SECRET_KEY, etc.)
# Other variables use {{VAR}} placeholders expanded at launch.
# Toggle modules with flags:
# NEXTCLOUD_ENABLED=true|false
# JITSI_ENABLED=true|false
# IMMICH_ENABLED=true|false

# 2. Start stack (core + modules enabled by flags)
./deploy/compose-up.sh up -d

Development

# Run locally (needs PG, KeyDB, RustFS running; loads .env with {{VAR}} expansion)
go run ./cmd/ultid

# Build
go build -o ultid ./cmd/ultid

# Expand .env for external tools (docker, migrate)
go run ./cmd/envexpand -in .env -out .env.resolved
source <(grep -v '^#' .env.resolved | sed 's/^/export /')

# Run migrations (use expanded ULTID_DB_URL; host may need localhost instead of postgres)
migrate -path migrations -database "$ULTID_DB_URL" up

Reverse proxy (nginx)

Un seul nginx expose lentrée HTTP (:80) et route :

Chemin Service
/api/* ultid
/ws ultid (WebSocket)
/auth/* Authentik
/meet/* Jitsi (si JITSI_ENABLED=true)
/cloud/* Nextcloud nginx+FPM (si NEXTCLOUD_ENABLED=true)

Nextcloud : FPM + nginx dédié ; ultid appelle NEXTCLOUD_URL en interne (http://nextcloud:80).
Caddy retiré : un seul proxy évite la double couche ; TLS plus tard (certbot, Traefik, ou listen 443 nginx).

Centralized secrets

Set passwords and keys once in the Secrets section at the top of .env. Derived values reference them with {{POSTGRES_PASSWORD}}, {{RUSTFS_SECRET_KEY}}, etc. Expansion runs via:

  • ./deploy/compose-up.sh — writes .env.resolved for Docker Compose
  • go run ./cmd/envexpand -in .env -out .env.resolved — manual export for migrate/scripts
  • go run ./cmd/ultid — expands .env in-process before reading config

Runtime secret files are also supported with *_FILE variables (example: ULTID_OIDC_CLIENT_SECRET_FILE=/run/secrets/oidc_client_secret).

Mail credentials are encrypted at rest with AES-GCM using MAIL_CREDENTIAL_KEYS (key_id:base64key,...) and MAIL_ACTIVE_CREDENTIAL_KEY_ID. Secret rotation policy is enforced through:

  • SECRET_ROTATION_MAX_AGE
  • ULTID_OIDC_CLIENT_SECRET_ROTATED_AT
  • MAIL_CREDENTIAL_KEY_ROTATED_AT
  • MAIL_WEBHOOK_SHARED_SECRET_ROTATED_AT

Observability (Prometheus / Grafana)

ultid exposes Prometheus metrics at /metrics (see internal/observability/metrics.go). The core Docker Compose stack includes Prometheus and Grafana with configs under deploy/observability/:

File Purpose
deploy/observability/prometheus/prometheus.yml Scrape ultid + self; loads alert rules
deploy/observability/prometheus/alerts.yml Alert rules: IMAP sync stalled, outbox backlog, HTTP 5xx rate
deploy/observability/grafana/ultid-baseline.json Baseline dashboard (HTTP latency/errors, IMAP sync, outbox, webhooks)
deploy/observability/grafana/provisioning/ Grafana datasource + dashboard auto-load

Start with the rest of the stack (./deploy/compose-up.sh up -d), then open:

Service URL Notes
Prometheus http://localhost:9090 Targets: ultid, prometheus
Grafana http://localhost:3000 Login from .env (GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD, default admin / admin); dashboard Ultid Baseline under folder Ultid

Alertmanager — not included in compose; route labels service=ultid and severity (critical, warning) to your on-call channels when you add it.

Stack

Component Technology
Backend Go 1.23+ (chi, pgx, go-imap, go-smtp)
Database PostgreSQL 16
Cache KeyDB (Redis-compatible, multi-threaded)
Object Storage RustFS (S3-compatible, Apache 2.0)
Auth Authentik (OIDC/SAML)
Files/Cal/Contacts Nextcloud headless (WebDAV/CalDAV/CardDAV)
Video Jitsi Meet (JWT auth)
Photos Immich (ML classification)
Reverse Proxy nginx (TLS à ajouter via certbot ou autre)
Search PostgreSQL tsvector + GIN

Calendar API (mini doc)

Base path: /api/v1/calendar (module actif seulement si NEXTCLOUD_ENABLED=true).

Endpoints principaux

Endpoint Méthode Usage
/ GET Lister calendriers
/{calID}/events GET Lister événements d'un calendrier
/{calID}/events POST Créer événement
/events/* PUT Modifier événement avec If-Match
/events/* DELETE Supprimer événement
/events/response/* POST Répondre invitation participant
/freebusy POST Obtenir disponibilités (free/busy)
/events/meet-link/* POST Générer lien Meet et l'injecter dans l'événement

* représente un chemin CalDAV absolu, ex: /remote.php/dav/calendars/{user}/{calendar}/{uid}.ics

Event payload (create / update)

{
  "uid": "team-sync-2026-05-23",
  "summary": "Team sync",
  "description": "Weekly project update",
  "location": "Paris office",
  "start": "20260523T090000Z",
  "end": "20260523T100000Z",
  "organizer": "alice@example.com",
  "attendees": [
    { "email": "bob@example.com", "name": "Bob", "status": "NEEDS-ACTION" },
    { "email": "carol@example.com", "name": "Carol", "status": "ACCEPTED" }
  ]
}

Update avec ETag / If-Match

Requête:

PUT /api/v1/calendar/events/remote.php/dav/calendars/alice/work/team-sync-2026-05-23.ics
If-Match: "68031b8f4d18f"
Content-Type: application/json

Réponse succès:

{ "etag": "\"68031b8f5f901\"" }

Si ETag obsolète, API renvoie 412 avec code etag_mismatch.

Réponse invitation participant

POST /api/v1/calendar/events/response/remote.php/dav/calendars/alice/work/team-sync-2026-05-23.ics
{
  "email": "bob@example.com",
  "response": "accepted",
  "if_match": "\"68031b8f5f901\""
}

response autorise: accepted, declined, tentative.

Free/Busy

POST /api/v1/calendar/freebusy
{
  "start": "2026-05-23T08:00:00Z",
  "end": "2026-05-23T18:00:00Z",
  "attendees": ["bob@example.com", "carol@example.com"]
}

Réponse (exemple):

{
  "attendees": [
    {
      "email": "bob@example.com",
      "periods": [
        { "start": "20260523T090000Z", "end": "20260523T100000Z", "type": "BUSY" }
      ]
    }
  ]
}

Création lien Meet depuis event

POST /api/v1/calendar/events/meet-link/remote.php/dav/calendars/alice/work/team-sync-2026-05-23.ics
{
  "if_match": "\"68031b8f5f901\""
}

Réponse:

{
  "meet_url": "https://your-domain/meet/team-sync-2026-05-23?jwt=...",
  "etag": "\"68031b8f8b102\""
}

Si module Meet désactivé (JITSI_ENABLED=false), API renvoie 409 avec code meet_disabled.

Erreurs API Calendar

HTTP code Cas
400 invalid_request_body JSON invalide, champ manquant, email participant invalide, format date invalide
400 request_body_too_large Payload dépasse limite handler (256KB)
401 auth.* Token manquant/invalide
403 auth.forbidden Permission calendrier insuffisante (read/write)
404 attendee_not_found Email donné absent des ATTENDEE de l'événement
409 meet_disabled Endpoint meet-link appelé alors que Jitsi désactivé
412 etag_mismatch If-Match obsolète sur update event/response/meet-link
500 internal_error Erreur Nextcloud/CalDAV ou erreur interne backend

Format standard:

{
  "code": "etag_mismatch",
  "message": "etag does not match current resource version",
  "trace_id": "req_01HZ..."
}

Project Structure

├── cmd/ultid/          — Entry point
├── internal/
│   ├── api/            — HTTP handlers (mail, admin, drive, calendar, contacts, meet, photos)
│   ├── mail/           — IMAP sync, SMTP send, rules engine, webhooks
│   ├── nextcloud/      — WebDAV/CalDAV/CardDAV client
│   ├── meet/           — Jitsi JWT generation
│   ├── photos/         — Immich API client
│   ├── auth/           — OIDC verification
│   ├── search/         — Full-text search
│   ├── realtime/       — WebSocket hub
│   └── config/         — Environment config
├── migrations/         — SQL migrations
├── deploy/             — Docker Compose configs
│   ├── docker-compose.yml  — Core stack
│   ├── observability/      — Prometheus alerts + Grafana dashboard
│   ├── nginx/
│   ├── nextcloud/
│   ├── jitsi/
│   └── immich/
├── Dockerfile          — Multi-stage build
└── .env.example