This documentation is available to agents as Markdown at https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/operations.md. For the complete documentation index, see llms.txt.

Backups, Logs and Monitoring

Governor keeps its state in its PostgreSQL metadata database. Back up that database, and the secrets you set at installation. This page also covers the platform’s storage, its logs and how to monitor it.

Back up the metadata database

The metadata database holds users, roles, projects, data sources with their encrypted credentials, workflows, their configuration versions and run history. It doesn’t hold the contents of your databases.

Back it up with pg_dump, in the custom format, so that pg_restore can restore it:

pg_dump -U apiuser -h <db-host> -F c -b -v -f /path/to/backup/db-backup.full.dump governor

apiuser and governor are the user and database you set in SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_URL.

  • Back up on a schedule, and always before you update.

  • Keep copies of JWT_SECRET and CREDENTIALS_ENCRYPTION_KEY with the backups, in your secret store. Without the same CREDENTIALS_ENCRYPTION_KEY, a restored Governor can’t read the data source passwords it stores.

  • Keep your docker-compose.yml or values.yaml too, so you can recreate the installation.

  • Restrict access to the backups as you would to the database: they hold the encrypted data source passwords.

Restore the metadata database

  1. Stop Governor, so nothing writes to the database while you restore it:

    • Docker Compose: docker compose down

    • Kubernetes: kubectl scale deployment governor-api --replicas=0 -n <namespace>

  2. Drop the database and create it again, empty. The PostgreSQL user must be allowed to drop and create databases. If apiuser isn’t, run dropdb and createdb as a PostgreSQL superuser.

    dropdb -U apiuser -h <db-host> governor
createdb -U apiuser -h <db-host> -O apiuser governor

  3. Restore the backup. --exit-on-error stops at the first error, so a failed restore doesn’t go unnoticed:

    pg_restore --exit-on-error -U apiuser -h <db-host> -d governor /path/to/backup/db-backup.full.dump

  4. Start Governor again with the same JWT_SECRET and CREDENTIALS_ENCRYPTION_KEY: docker compose up -d, or scale governor-api back to its previous number of replicas.

Restore a backup on the same Governor version that made it. To go back to an earlier version, follow the rollback steps in Update the Platform.

Storage

Besides its database, the platform uses local storage for working data and logs:

Storage Path in the container Notes

Working storage

/app/storage

DuckDB data, used for CSV data sources. Mount it on the host in Docker Compose.

RocksDB

/app/rocksdb

A key-value store that speeds up processing. Keeping it inside the container can cause performance and disk-space problems. In Helm, mount it with the worker’s persistence values.

Logs

/app/logs

Application log files. Mount it on the host to keep the logs, or to forward them.

Allow at least 100 GB for this storage in production, and watch its free space. The backend and worker containers run as user 1001 and group 0, so give that user ownership of the host folders you mount:

sudo chown -R 1001:0 <host folder>

Logs

Logs Where to find them

Governor and workers

docker compose logs backend and docker compose logs worker, or kubectl logs deployment/governor-api -n <namespace> and kubectl logs deployment/governor-worker -n <namespace>. With the logs folder mounted, the same logs are in files your log forwarder can collect.

Each run

The run log in the web UI, and from the REST API. Run logs record what each step did, without values from your databases.

Changes to a workflow

The workflow’s Activity logs tab, and its version history.

User actions

Governor’s audit log, which records every authenticated API call with the user, the time and the action.

Monitoring

  • Health: docker compose ps shows whether each service is running and healthy. On Kubernetes, kubectl get pods -n <namespace> shows the governor-api, governor-front and governor-worker pods.

  • Metrics: Governor exposes health, metrics and Prometheus endpoints for your monitoring to read. Choose which with MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE (default health,metrics,prometheus), and set MANAGEMENT_HEALTH_LDAP_ENABLED=true to include the LDAP connection in the health check. Allow only your monitoring systems to reach them.

  • Workers: Administration > Workers lists the connected workers and their last heartbeat.

  • Errors: to send error reports to a Sentry project, set SENTRY_DSN. It’s off by default.

See Governor and Worker Settings for these settings.

Related pages