Securing the Worker–Governor Channel with mTLS

Enable mutual TLS on the worker–Governor gRPC connection using your own certificate files, so only workers holding a certificate signed by a trusted CA can register.

Workers connect to Governor over gRPC. In Kubernetes, the Helm chart can secure this channel in-cluster with Envoy sidecars (the tlsInternal option). This guide covers deployments that instead manage their own certificates — Docker Compose, bare VMs, or Kubernetes without the sidecar mesh — where you enable application-level mutual TLS by pointing both sides at PEM certificate files.

When enabled, Governor requires every worker to present a client certificate signed by the configured CA before any gRPC call is accepted; the TLS handshake is rejected otherwise. Each worker likewise verifies Governor’s server certificate against the same CA. There is no separate approval step — trust is established entirely by the shared CA.

Configuration

Both sides read the same properties under worker.grpc.tls. Each maps to an environment variable through Spring relaxed binding. The certificate paths must be readable by the process; when enabled=true and any path is missing or unreadable, startup fails with an error naming the offending property.

Property Environment variable Meaning

worker.grpc.tls.enabled

WORKER_GRPC_TLS_ENABLED

Turn mutual TLS on. true / false (default false)

worker.grpc.tls.cert-chain

WORKER_GRPC_TLS_CERT_CHAIN

Path to this side’s certificate chain (PEM). Governor: the server cert; worker: the client cert

worker.grpc.tls.private-key

WORKER_GRPC_TLS_PRIVATE_KEY

Path to this side’s private key. Must be PKCS#8 PEM

worker.grpc.tls.trust-cert

WORKER_GRPC_TLS_TRUST_CERT

Path to the CA certificate (PEM) used to verify the peer

worker.grpc.tls.key-password

WORKER_GRPC_TLS_KEY_PASSWORD

Optional. Password if the private key is encrypted

gRPC/Netty accepts only PKCS#8 private keys (-----BEGIN PRIVATE KEY-----). A traditional RSA key (-----BEGIN RSA PRIVATE KEY-----) must be converted first — see the openssl pkcs8 step below.

Generating certificates

The example below creates a self-signed CA that signs one server certificate (Governor) and one client certificate (worker). In production, issue these from your organization’s PKI instead of a throwaway CA.

# 1. Certificate authority
openssl req -x509 -newkey rsa:4096 -days 3650 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=tdk-worker-ca"

# 2. Governor server certificate — CN/SAN must match the host workers dial.
# The SAN is applied to the signed certificate by -extfile on the x509 step below.
openssl req -newkey rsa:4096 -nodes -keyout governor.key -out governor.csr \
  -subj "/CN=governor"
openssl x509 -req -in governor.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 825 -out governor.crt \
  -extfile <(printf "subjectAltName=DNS:governor,DNS:localhost")

# 3. Worker client certificate
openssl req -newkey rsa:4096 -nodes -keyout worker.key -out worker.csr \
  -subj "/CN=worker"
openssl x509 -req -in worker.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 825 -out worker.crt

# 4. Convert both private keys to PKCS#8 (skip if already -----BEGIN PRIVATE KEY-----)
openssl pkcs8 -topk8 -nocrypt -in governor.key -out governor.pk8.key
openssl pkcs8 -topk8 -nocrypt -in worker.key   -out worker.pk8.key

This yields, per side:

Side Files

Governor

cert-chain=governor.crt, private-key=governor.pk8.key, trust-cert=ca.crt

Worker

cert-chain=worker.crt, private-key=worker.pk8.key, trust-cert=ca.crt

The CA certificate (ca.crt) goes to both sides. Each side keeps only its own private key. The CA private key (ca.key) stays offline on the machine that issues certificates — it never ships to Governor or a worker.

Docker Compose

Mount the certificate directory into both containers and set the env vars. The worker must also disable plaintext so it negotiates TLS.

services:
  backend:
    volumes:
      - ./certs:/certs:ro
    environment:
      WORKER_GRPC_TLS_ENABLED: "true"
      WORKER_GRPC_TLS_CERT_CHAIN: /certs/governor.crt
      WORKER_GRPC_TLS_PRIVATE_KEY: /certs/governor.pk8.key
      WORKER_GRPC_TLS_TRUST_CERT: /certs/ca.crt

  worker:
    volumes:
      - ./certs:/certs:ro
    environment:
      WORKER_USEPLAINTEXT: "false"
      WORKER_GRPC_TLS_ENABLED: "true"
      WORKER_GRPC_TLS_CERT_CHAIN: /certs/worker.crt
      WORKER_GRPC_TLS_PRIVATE_KEY: /certs/worker.pk8.key
      WORKER_GRPC_TLS_TRUST_CERT: /certs/ca.crt

Kubernetes (Helm)

If you use the chart’s built-in mTLS (tlsInternal.enabled: true), you do not need this — the Envoy sidecars already provide mutual TLS between components. Reach for file-based mTLS only when tlsInternal.enabled: false and you supply your own certificates: mount them (for example from a Secret) into the api and worker containers and set the WORKER_GRPC_TLS_* env vars on both.

Notes

  • Set worker.use-plaintext=false (WORKER_USEPLAINTEXT=false) on the worker when TLS is on. Otherwise the client opens a plaintext connection and never performs the handshake.

  • The Governor server certificate’s SAN must match the hostname workers dial. If workers connect under a different name (a Kubernetes service name, a public DNS record), regenerate the Governor certificate with that name added to the SAN.

  • File-based mTLS and the SPIRE identity source are mutually exclusive — enable only one. Configuring both on the same worker (spire.socket-path set and worker.grpc.tls.enabled=true) fails fast at startup with a clear error.

Related Topics