Phase 6: private AI, applications, monitoring, recovery and regional gateways

Update: completion work adds account/team controls, owned DNS, private GitHub, Git automation and scheduled cold backups. Where earlier limits below differ, that guide describes the current implementation and required migrations.

Application catalog and private AI

The marketplace has twelve compiled recipes: PostgreSQL, Redis, MariaDB, MongoDB, WordPress, Nextcloud, Ghost, n8n, Ollama, Open WebUI, static websites and Node.js apps. Recipes declare required credentials, resource minima, ports, fixed volume slots, container users and private-only status. Catalog rows start disabled with no image. An operator must review and test an immutable image, then set application_templates.image='REPOSITORY@sha256:DIGEST', enabled=true for the matching ID using an operator database connection. The API and agent validate the compiled repository and digest. Do not enable floating tags or unreviewed images. The Node recipe requires an operator-built app image; ghcr.io/ownserva/node-template is a designated repository, not a claim that an image has been published.

Templates are single containers, not Compose stacks. WordPress/Ghost need separately provisioned, reachable databases. The PostgreSQL volume mount /var/lib/postgresql requires an image version compatible with that layout; approve an appropriate version and test upgrade behavior. HTTP recipes must pass their declared endpoint with 2xx before promotion; database recipes use TCP readiness and receive no HTTP route. Initial setup redirects can require an operator-adjusted/purpose-built image. Persistent templates can run their trusted image entrypoint as root with a limited capability set; arbitrary application images retain the generic non-root policy. Volume ownership initialization and every recipe need real Docker acceptance.

Redis uses append-only persistence and REDIS_PASSWORD. Open WebUI forces authentication on and signup off; supply WEBUI_ADMIN_EMAIL and WEBUI_ADMIN_PASSWORD to create the first administrator at startup, alongside WEBUI_SECRET_KEY and an Ollama address reachable from its container. Pin a version supporting these variables; see Open WebUI configuration. Host loopback addresses are not container-to-container addresses. Provision that private connectivity explicitly; no shared application network is created automatically.

Ollama, Open WebUI and database templates cannot receive public Ownserva tunnels. Use SSH port forwarding to the host loopback port (inspect Docker's managed container binding locally). The agent controls models only through its assigned Ollama loopback port; there is no public inference proxy or remote shell endpoint. AI data stays on the customer host apart from model operation metadata. Disk/model downloads contact their upstream registries; this is not an offline install.

Model controls cover download, load, unload and remove for the compiled Qwen 2.5 0.5B / Llama 3.2 1B / Llama 3.2 3B catalog. The UI displays streamed bytes and verified runtime inventory, with stale observations marked unknown. RAM and free-disk checks reject unsuitable jobs. Operations are serialized against local deployment/lifecycle/backup actions and journaled; interrupted jobs become unknown instead of rerunning. Review model licenses. CPU inference is supported; GPU passthrough is not configured by this implementation.

Monitoring

Signed heartbeats feed current server status and one-minute samples with seven-day retention; the dashboard shows up to 240 samples over six hours. Missing observations and stale agents are not shown as healthy. Host network counters are cumulative host traffic, not edge bandwidth billing. /metrics is an operator-wide Prometheus endpoint requiring Authorization: Bearer PROMETHEUS_TOKEN, where the token is 64 random hex characters. Keep it private; it includes organization/server UUID labels. Edge routing has independent release-specific health probes.

Encrypted cold backups and recovery

Install a verified restic binary as root-owned /usr/local/bin/restic. Create root-owned /etc/ownserva/backup.json from examples/backup.json, and a separate /etc/ownserva/backup-password; both must exclude access by other users and all three files must not be group/world writable. Grant only the service's necessary read access. Keep credentials and password outside the repo. Initialize the repository separately with restic using those same files and the documented AWS environment; the agent does not initialize repositories automatically. Local absolute paths and HTTPS S3 repositories are supported. Keep an offline copy of the password; loss prevents recovery. Restic encrypts repository contents.

Stop the application before a backup; every container mounting that volume must remain stopped throughout the operation. This is a cold managed-volume backup, not a database-consistent online dump. Do not run out-of-band writers during it. Select a compiled template volume slot in Backups. The agent uses stopped helper containers and Docker's archive API, validates archive paths/types/symlinks and byte/file limits, streams the normalized archive to restic, then verifies the immutable snapshot's host/app/slot/job scope. Only verified snapshots enter the catalog with a content manifest hash.

Restore creates a new ownserva-recovery-JOB_UUID volume. It validates snapshot scope, archive safety, the saved manifest and the extracted volume's read-back manifest. It never overwrites or mounts the active application volume. The UI reports the recovery volume for operator inspection. Activating recovered data is an explicit operator recovery runbook step, with the application stopped and a fresh backup of existing data. An unknown result can leave a recovery volume for inspection; do not automatically delete it. Archive root directory ownership, xattrs/ACLs and arbitrary host paths are not captured. Verify application-specific ownership and functionality before activation.

This release provides manual backup and isolated recovery jobs. Scheduling, retention/prune policy, offsite replication, automatic recovery activation and database-native online backups remain follow-up work. Configure repository durability, periodic restic check, retention and recovery drills operationally. The private configuration is local to each customer server; credentials never transit the dashboard.

Regional routing

A public app can select one primary and up to two fallback gateways. Provision each gateway using the Phase 4 guide, separate keys, TLS trust and a reachable DNS zone. The agent opens separate credential-scoped outbound connections for the same hostname. Each edge probes its own loopback FRPS route and reports the exact current release. The worker prefers a stable healthy route, honors primary priority after recovery, reconciles DNS and independently verifies public HTTPS before reporting active. Custom domain aliases use the same selected route and require current ownership proof.

Failover is DNS based: health observation, a 30-second stability hold, reconciliation and DNS cache TTL contribute to recovery time. Existing connections may drop. The app still runs on its original customer server, so this is gateway resilience, not replicated application/database failover or geographic latency routing. No actual US/UK/ZA gateway infrastructure has been purchased or provisioned.

Acceptance

On Ubuntu, test all approved template images, first-admin bootstrap and private connections; download/load/unload/remove a small real model and interrupt a job. Verify insufficient RAM/disk and forged/foreign jobs fail. Back up a stopped disposable volume with directories, files and safe symlinks, recover into a new volume, verify content/ownership and prove the original volume is unchanged. Test a wrong snapshot, manifest, app, server and archive traversal rejection. Scrape two servers with Prometheus and check family grouping/stale status. With two real gateways, stop the primary, observe DNS and HTTPS verification, restore the primary and revoke the route. None of those live Docker/restic/model/gateway drills were performed on this Windows development host.