Hashicorp Vault as a Secret Manager

Hashicorp Vault is a widely used secrets and encryption management system. In the case of TDK, secrets refer to database connection credentials, such as usernames and passwords.

Without Vault

Without secret managers, such as Vault, we have to use usernames and passwords in their raw and unsafe format:

tdk \
    --input-url ...
    --input-username USER_FOO \
    --input-password fxORRC-IYcavW5CoYX9Z \
    --output-url ...
    --output-username USER_BAR \
    --output-password -Z3aS2JUnZUjeyCd4bGu \
    ...

Using Hashicorp Vault

But using Hashicorp Vault, we can store and manage our database credentials in a special secure storage.

To enable Vault as a secret manager for TDK, you only need to add some properties, such as URI and authentication parameters, to the Application properties file:

secrets.vault.secret.manager.enabled = true
vault.uri=http://localhost:9200
vault.token=1234

Please refer to the documentation for information on other Vault authentication methods that are supported.

Types of Vault database secrets

TDK supports at least 3 methods for storing and managing database secrets. Let’s consider each of them.

1. Simple secrets

In the simplest case, we can store usernames and passwords like separate, unrelated Vault secrets without validation on the database side, rotation, TTL options, and so on.

Add simple secrets with the database username and password in the Vault key-value storage:

vault kv put secret/postgres-source-simple-user username=simple_pg_user password=SecrETPassw0rD!
vault kv put secret/postgres-target-simple-user username=simple_pg_user password=SecrETPassw0rD!

Prepare TDK inventory file inventory_simple_secret.yaml

data_sources:
  default_source:
    jdbc_url: jdbc:postgresql://localhost:6000/postgres
    user:
      type: vault
      storage: secret
      path: postgres-source-simple-user
      versioned: true
      secret: username
    password:
      type: vault
      storage: secret
      path: postgres-source-simple-user
      versioned: true
      secret: password
  default_target:
    jdbc_url: jdbc:postgresql://localhost:6001/postgres
    user:
      type: vault
      storage: secret
      path: postgres-target-simple-user
      versioned: true
      secret: username
    password:
      type: vault
      storage: secret
      path: postgres-target-simple-user
      versioned: true
      secret: password

Run TDK using the inventory_simple_secret.yaml file:

tdk \
    --inventory-file inventory_simple_secret.yaml \
    --config-file config.tdk.yaml

2. Static database roles

Vault offers more than just ordinary secrets. It also provides static roles for storing and managing database-specific users and connections.

Enable Vault database plugin:

vault secrets enable database

Create a Vault connection for the source database:

vault write database/config/postgres_source \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@input_db:5432/postgres" \
    allowed_roles="postgres-source-static-role,postgres-source-dynamic-role" \
    username="postgres" \
    password="postgres"

Create a Vault connection for the target database in the same manner

vault write database/config/postgres_target \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@output_db:5432/postgres" \
    allowed_roles="postgres-target-static-role,postgres-target-dynamic-role" \
    username="postgres" \
    password="postgres"

Create a Vault static role for the source database:

vault write database/static-roles/postgres-source-static-role \
    db_name=postgres_source \
    username="static_vault_pg_user" \
    rotation_period=86400

Create a Vault static role for the target database in the same manner

vault write database/static-roles/postgres-target-static-role \
    db_name=postgres_target \
    username="static_vault_pg_user" \
    rotation_period=86400

Prepare TDK inventory file inventory_static_users.yaml

data_sources:
  default_source:
    jdbc_url: jdbc:postgresql://localhost:6000/postgres
    user:
      type: vault
      storage: database
      path: static-creds/postgres-source-static-role
      secret: username
    password:
      type: vault
      storage: database
      path: static-creds/postgres-source-static-role
      secret: password
  default_target:
    jdbc_url: jdbc:postgresql://localhost:6001/postgres
    user:
      type: vault
      storage: database
      path: static-creds/postgres-target-static-role
      secret: username
    password:
      type: vault
      storage: database
      path: static-creds/postgres-target-static-role
      secret: password

Run TDK using the inventory_static_users.yaml file:

tdk \
    --inventory-file inventory_static_users.yaml \
    --config-file config.tdk.yaml

3. Dynamic database roles

In addition to static roles, Vault also provides dynamic roles. In this case, Vault creates database users with random passwords for each request.

Create a Vault dynamic role for the source database:

vault write database/roles/postgres-source-dynamic-role \
    db_name="postgres_source" \
    creation_statements=" \
      create role \"{{name}}\" with login password '{{password}}' valid until '{{expiration}}'; \
      grant connect on database postgres to \"{{name}}\"; \
      grant usage on schema foo TO \"{{name}}\"; \
      grant select on all tables in schema \"foo\" to \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Create a Vault dynamic role for the target database in the same manner

vault write database/roles/postgres-target-dynamic-role \
    db_name="postgres_target" \
    creation_statements=" \
      create role \"{{name}}\" with login password '{{password}}' valid until '{{expiration}}'; \
      grant create on database postgres to \"{{name}}\"; \
      grant usage, create on schema foo TO \"{{name}}\"; \
      grant truncate, select, insert on all tables in schema \"foo\" to \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Prepare TDK inventory file inventory_dynamic_users.yaml

data_sources:
  default_source:
    jdbc_url: jdbc:postgresql://localhost:6000/postgres
    user:
      type: vault
      storage: database
      path: creds/postgres-source-dynamic-role
      secret: username
    password:
      type: vault
      storage: database
      path: creds/postgres-source-dynamic-role
      secret: password
  default_target:
    jdbc_url: jdbc:postgresql://localhost:6001/postgres
    user:
      type: vault
      storage: database
      path: creds/postgres-target-dynamic-role
      secret: username
    password:
      type: vault
      storage: database
      path: creds/postgres-target-dynamic-role
      secret: password

Run TDK using the inventory_dynamic_users.yaml file:

tdk \
    --inventory-file inventory_dynamic_users.yaml \
    --config-file config.tdk.yaml