Helm-based Kubernetes Deployment - Upgrade/Rollback

Prerequisites

Before performing the upgrade, ensure the following steps are completed: * Helm is installed and configured to interact with your Kubernetes cluster. * The Governor Helm chart archive (e.g., governor-helm-chart-{new-version}.tgz) is downloaded from the appropriate repository or file storage.

If you have a network-limited environment (no internet access), transfer the archive manually to the target server using a USB drive, SCP, or another method.

Upgrade Process

Backup Database Always back up both the data and schema together before performing any upgrades to ensure a consistent state for rollback.
- Backup Data and Schema: Create a full backup of the database (both data and schema) using the pg_dump command:
  pg_dump -U apiuser -h localhost -F c -b -v -f /path/to/backup/db-backup.full.dump governor
Download the New Version of Governor Helm Chart Download the new version of the Governor Helm chart. This typically contains the updated values.yaml configuration file and other necessary Kubernetes setup files.
- Download the new Governor Helm chart archive from the repository:
  - Example: governor-helm-chart-{new-version}.tgz
Upgrade the Helm Release Use Helm to upgrade the existing release with the new version of the chart:
```
helm upgrade <release-name> /path/to/governor-helm-chart-{new-version}.tgz --values /path/to/values.yaml
```
Ensure that the necessary environment variables such as database credentials and JWT_SECRET are correctly set in the values.yaml. In case you have no access to existing values.yaml file, you can get actual values with 'helm get values <release-name>' command and save them to a new file.
Verify the Update Ensure the update is successful by checking the status of the release and Kubernetes pods:
```
kubectl get pods
kubectl describe deployment <release-name>
kubectl logs <pod-name>
```
Access the application in the browser and confirm that the upgrade has been applied correctly.

Rollback Process (Without Dropping Schema)

If the upgrade does not go as planned and you need to roll back, follow these steps to restore the previous state:

Restore the Database from Backup If data corruption or issues occurred during the upgrade, restore the database from the backup you created earlier.
- Restore Full Backup: Use the pg_restore command to restore both data and schema to their previous state:
  pg_restore -U apiuser -h db_host -d governor /path/to/backup/db-backup.full.dump
Rollback the Helm Release Rollback the Helm release with the previous version.
```
helm rollback <release-name>
```
Verify the Rollback Check the Kubernetes logs to ensure that the rollback was successful.
```
kubectl logs <pod-name>
```
Access the application in the browser and confirm that the system is functioning as expected with the rolled-back data and schema.

Upgrading from `agent` to `worker` naming

Starting with chart version 1.153.0, the agent component has been renamed to worker. The upgrade is handled automatically, but depending on your setup you may need to take action.

Default behavior (no action needed for most users)

On upgrade, the chart renames Kubernetes resources from governor-agent to governor-worker. Helm’s own three-way merge deletes the old Deployment, ConfigMap, and Secret automatically. The new governor-worker Deployment starts in their place.

Legacy agent: and tdkAgents: values in your values.yaml still work — the chart merges them on top of the new worker: / tdkWorkers: defaults. You will see deprecation warnings in the Helm output until you migrate your values file.

Preserving the old resource name

If you have monitoring, NetworkPolicies, ServiceMonitors, or scripts that reference governor-agent by name, add this to your values.yaml before upgrading:

agent:
  name: governor-agent

This keeps the existing Kubernetes resource names unchanged. No PVC annotation or cleanup is needed in this case.

Worker persistence (PVC)

If you had agent.persistence.enabled: true, the pre-upgrade hook automatically annotates agent-pvc with helm.sh/resource-policy: keep so that Helm does not delete it. Your data is preserved.

However, the renamed governor-worker Deployment will not mount the legacy PVC unless you explicitly opt in:

worker:
  persistence:
    enabled: true
    claimName: agent-pvc
    name: worker-pv
    mountPath: /app/rocksdb

Alternatively, keep agent.name: governor-agent along with your existing agent.persistence block — the old names stay, and everything mounts as before.

If the hook cannot annotate the PVC (e.g. insufficient RBAC or an admission webhook), the upgrade will abort to prevent data loss. The Job logs will show the exact error and remediation steps.

If you do not need the existing worker data, set migrationCleanup.deleteLegacyPvc: true in your values. The hook will then skip the annotation and Helm will delete the PVC during the upgrade.

Migrating your values file

When you are ready to adopt the new naming:

Rename agent: to worker: (the structure is identical).
Rename tdkAgents: to tdkWorkers: if present.
Do not keep both agent: and worker: blocks — agent: overrides take precedence on conflict, which can be confusing.

Important Notes

Testing Rollback: It’s always a good practice to test the rollback process in a staging environment before performing it in production. Ensure that the data and schema are restored correctly, and there are no issues with the application after rollback.
Database Migrations: If you applied schema migrations during the upgrade, this process will restore the schema as it was at the time of the backup. If you prefer not to revert migrations, you will need to manually undo any schema changes.
Backup Frequency: Ensure you take regular backups of both your data and schema, especially before any significant update. You can automate this process to prevent human error.

Troubleshooting

If you encounter issues during the upgrade or rollback process, refer to the Kubernetes pod logs for detailed error messages.

kubectl logs <pod-name>