# Backups, Logs and Monitoring

> Source: https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/operations
> For the complete documentation index, see [llms.txt](https://docs.synthesized.io/llms.txt).

Governor keeps its state in its PostgreSQL metadata database. Back up that database, and the secrets you set at installation. This page also covers the platform’s storage, its logs and how to monitor it.

## Back up the metadata database

The metadata database holds users, roles, projects, data sources with their encrypted credentials, workflows, their configuration versions and run history. It doesn’t hold the contents of your databases.

Back it up with `pg_dump`, in the custom format, so that `pg_restore` can restore it:

```shell
pg_dump -U apiuser -h <db-host> -F c -b -v -f /path/to/backup/db-backup.full.dump governor
```

`apiuser` and `governor` are the user and database you set in `SPRING_DATASOURCE_USERNAME` and `SPRING_DATASOURCE_URL`.

- Back up on a schedule, and always before you [update](https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/updating_tdk/).
- Keep copies of `JWT_SECRET` and `CREDENTIALS_ENCRYPTION_KEY` with the backups, in your secret store. Without the same `CREDENTIALS_ENCRYPTION_KEY`, a restored Governor can’t read the data source passwords it stores.
- Keep your `docker-compose.yml` or `values.yaml` too, so you can recreate the installation.
- Restrict access to the backups as you would to the database: they hold the encrypted data source passwords.

## Restore the metadata database

1. Stop Governor, so nothing writes to the database while you restore it:
   
   - Docker Compose: `docker compose down`
   - Kubernetes: `kubectl scale deployment governor-api --replicas=0 -n <namespace>`
2. Drop the database and create it again, empty. The PostgreSQL user must be allowed to drop and create databases. If `apiuser` isn’t, run `dropdb` and `createdb` as a PostgreSQL superuser.
   
   ```shell
   dropdb -U apiuser -h <db-host> governor
   createdb -U apiuser -h <db-host> -O apiuser governor
   ```
3. Restore the backup. `--exit-on-error` stops at the first error, so a failed restore doesn’t go unnoticed:
   
   ```shell
   pg_restore --exit-on-error -U apiuser -h <db-host> -d governor /path/to/backup/db-backup.full.dump
   ```
4. Start Governor again with the same `JWT_SECRET` and `CREDENTIALS_ENCRYPTION_KEY`: `docker compose up -d`, or scale `governor-api` back to its previous number of replicas.

Restore a backup on the same Governor version that made it. To go back to an earlier version, follow the rollback steps in [Update the Platform](https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/updating_tdk/).

## Storage

Besides its database, the platform uses local storage for working data and logs:

  
| Storage | Path in the container | Notes |
| --- | --- | --- |
| 
Working storage

 | 

`/app/storage`

 | 

DuckDB data, used for CSV data sources. Mount it on the host in Docker Compose.

 |
| 

RocksDB

 | 

`/app/rocksdb`

 | 

A key-value store that speeds up processing. Keeping it inside the container can cause performance and disk-space problems. In Helm, mount it with the worker’s `persistence` values.

 |
| 

Logs

 | 

`/app/logs`

 | 

Application log files. Mount it on the host to keep the logs, or to forward them.

 |

Allow at least 100 GB for this storage in production, and watch its free space. The `backend` and `worker` containers run as user `1001` and group `0`, so give that user ownership of the host folders you mount:

```shell
sudo chown -R 1001:0 <host folder>
```

## Logs

 
| Logs | Where to find them |
| --- | --- |
| 
Governor and workers

 | 

`docker compose logs backend` and `docker compose logs worker`, or `kubectl logs deployment/governor-api -n <namespace>` and `kubectl logs deployment/governor-worker -n <namespace>`. With the logs folder mounted, the same logs are in files your log forwarder can collect.

 |
| 

Each run

 | 

The run log in the web UI, and from the [REST API](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/advanced_workflows/external_api_reference). Run logs record what each step did, without values from your databases.

 |
| 

Changes to a workflow

 | 

The workflow’s **Activity logs** tab, and its [version history](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/version_history).

 |
| 

User actions

 | 

Governor’s audit log, which records every authenticated API call with the user, the time and the action.

 |

## Monitoring

- **Health**: `docker compose ps` shows whether each service is running and healthy. On Kubernetes, `kubectl get pods -n <namespace>` shows the `governor-api`, `governor-front` and `governor-worker` pods.
- **Metrics**: Governor exposes health, metrics and Prometheus endpoints for your monitoring to read. Choose which with `MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE` (default `health,metrics,prometheus`), and set `MANAGEMENT_HEALTH_LDAP_ENABLED=true` to include the LDAP connection in the health check. Allow only your monitoring systems to reach them.
- **Workers**: **Administration > Workers** lists the connected workers and their last heartbeat.
- **Errors**: to send error reports to a Sentry project, set `SENTRY_DSN`. It’s off by default.

See [Governor and Worker Settings](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/advanced_configuration) for these settings.

## Related pages

- [Update the Platform](https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/updating_tdk/)
- [Troubleshooting](https://docs.synthesized.io/tdk/latest/user_guide/080_troubleshooting/)
- [Security & Compliance](https://docs.synthesized.io/tdk/latest/user_guide/060_security_compliance/)
