Certificates & mTLS

Portoser ships its own internal CA and a CLI to issue, distribute, and rotate certificates for service-to-service mTLS. Public TLS (Let's Encrypt) is handled by Caddy; internal TLS is handled here, in lib/certificates.sh plus the repo-root install_ca_on_hosts.sh helper.

What this layer is for

  • mTLS between services inside the cluster. Service A presents a cert; service B verifies against the cluster CA before responding.
  • HTTPS server certs for services that terminate TLS themselves rather than letting Caddy do it.
  • CA distribution to new hosts so they trust the certs everyone else is presenting.

It is not for public-facing TLS (use Caddy + ACME for that), and not for end-user authentication (that's Keycloak's job).

The CA

The first thing to set up:

./portoser certs init-ca

Creates ~/.portoser/ca/ containing ca-cert.pem, ca-key.pem, and a serial counter. The CA private key never leaves this machine (the one you run the CLI from). All other hosts only ever see ca-cert.pem for verification.

init-ca is idempotent — running it on an already-initialized CA is a no-op. To rotate the CA itself, you delete ~/.portoser/ca/ and re-init, then re-issue every certificate. Rare, painful, plan it.

Issuing service certificates

Two flavors:

Client certs (service A → service B)

./portoser certs generate <service>            # one
./portoser certs generate-all                   # every service in registry

Output: ~/.portoser/certs/<service>/client-cert.pem + client-key.pem. The cert's CN is the service hostname; the SAN list includes <service>.internal and any aliases declared in the registry.

Server certs (HTTPS / TLS-terminating services)

./portoser certs generate-server <service>
./portoser certs generate-all-servers

Output: ~/.portoser/certs/<service>/server-cert.pem + server-key.pem. Used by services that terminate TLS themselves — Vault, Postgres-with-SSL, Neo4j Bolt, custom services that don't sit behind Caddy.

After generating, push them to the right hosts:

./portoser certs deploy <service> <machine>     # one cert to one host
./portoser certs deploy-servers                  # all server certs to their respective hosts

The deploy step scps certs to the service's path on the target machine and runs whatever owner / permission fixup the service requires (configured in the service's service.yml).

Linking certs into the registry

Once certs exist, the registry tells Portoser (and Caddy, downstream) which files to use:

services:
  api:
    hostname: api.internal
    current_host: mini1
    deployment_type: docker
    docker_compose: /api/docker-compose.yml
    port: 8400
    tls_cert: /api/certs/api-cert.pem        # paths inside the service repo
    tls_key:  /api/certs/api-key.pem
    ca_cert:  /api/certs/ca-cert.pem

./portoser certs update-registry walks the cert directories and writes the right tls_cert / tls_key / ca_cert paths into the registry for every service that has certs. Useful as a one-shot after a bulk rotation.

Distributing the CA to hosts

For mTLS to verify, every host needs to trust the cluster CA. The install_ca_on_hosts.sh script at the repo root walks the registry, copies ca-cert.pem to each host, and adds it to the host's trust store:

./install_ca_on_hosts.sh

What it does per host:

  • macOS: imports the CA into the System keychain via security add-trusted-cert
  • Debian/Ubuntu: drops it into /usr/local/share/ca-certificates/ and runs update-ca-certificates
  • Other Linux: installs to /etc/pki/ca-trust/source/anchors/ and runs update-ca-trust

This requires sudo on each host once. The script is idempotent — running it again with a host that already trusts the CA is a no-op.

Listing and inspecting

./portoser certs list
# Prints every cert under ~/.portoser/certs/, with subject, expiry, and issuer

You'll get output like:

service          type    expiry              cn
api              client  2027-03-15 14:30   api.internal
api              server  2027-03-15 14:30   api.internal
postgres-prod    server  2027-03-15 14:30   postgres-prod.internal

For more detail, fall back to openssl x509 -in <path> -noout -text.

Rotation

Certs expire. The default lifetime is configurable in the cert generator; expect to rotate annually or biannually depending on your policy.

# Rotate one service
./portoser certs generate <service>
./portoser certs deploy <service> <machine>

# Rotate everything
./portoser certs generate-all
./portoser certs generate-all-servers
./portoser certs deploy-servers
./portoser caddy sync   # re-emit Caddyfile so Caddy reloads with new certs

For services that read certs at startup (most of them), trigger a redeploy after the new cert is on the host:

./portoser deploy <machine> <service>

For services that watch their cert files for changes (less common), the new cert gets picked up on its next read.

What about Vault?

Vault stores secrets, not the CA itself. The cluster CA is on the machine where you run the CLI. If you want long-term cert lifecycle management with audit trails and short-lived certs, look at HashiCorp Vault's PKI engine — Portoser's Vault integration can hold its config, but the rotation flow is yours to wire up. The shipped path is "longer-lived certs, manual rotation." It's deliberately simpler than Vault PKI; if you outgrow it, the upgrade path is well-trodden.

Common gotchas

  • CA not trusted on a new host. You added a Pi to the cluster but never ran install_ca_on_hosts.sh. The new host's services don't trust other services' certs. Symptom: TLS handshake failures with unknown certificate authority in logs.
  • Cert deployed to wrong path. The registry's tls_cert: path is relative to the service's repo on the target machine, not absolute on your CLI host. If you wrote /Users/me/dev/api/certs/api-cert.pem, the target machine has no idea what that means.
  • Permissions. Many services require their TLS key to be 0600 and owned by the service user. Configure this in the service's own setup; Portoser doesn't enforce it (the cert just won't get read).
  • Expired CA. The CA itself expires too. ./portoser certs list shows CA expiry. Re-init CA + re-issue + re-distribute. Plan for a maintenance window.

Where this layer falls down

  • No automatic short-lived cert rotation. Certs are valid until they expire; you rotate on a schedule, not continuously.
  • No revocation infrastructure (CRL, OCSP). Compromised cert = re-init the CA or accept residual exposure until natural expiry.
  • The CA private key on one machine is a single point of failure and a single point of compromise. Keep it on a host you trust, ideally not on a laptop that travels.

Next