Multiple Databases

If you want to use TDK with multiple databases, you should provide inventory and transformation configurations.

Let’s start with the inventory file. Here we describe the connection details for two input and two output databases:

#inventory.yaml
data_sources:
  sourceFirstDb:
    url: jdbc:mysql://localhost:3307/source_db_1
    user:
      type: raw
      value: db-user
    password:
      type: raw
      value: db-password
  targetFirstDb:
    url: jdbc:mysql://localhost:3308/target_db_1
    user:
      type: raw
      value: db-user
    password:
      type: raw
      value: db-password
  sourceSecondDb:
    url: jdbc:mysql://localhost:3307/source_db_2
    user:
      type: raw
      value: db-user
    password:
      type: raw
      value: db-password
  targetSecondDb:
    url: jdbc:mysql://localhost:3308/target_db_2
    user:
      type: raw
      value: db-user
    password:
      type: raw
      value: db-password

More information about the inventory.yaml file structure can be found in inventory file reference.

Next, let’s describe workflow transformations in transformation_configs and data_source_mapping using input and output databases defined in inventory.yaml.

For example:

#config.yaml
transformation_configs:
  testFirst:
    default_config:
      mode: "KEEP"
      target_ratio: 0.5
    tables:
        - table_name_with_schema: public.address
          target_ratio: 1.0
  testSecond:
    default_config:
      mode: "KEEP"
      target_ratio: 1.0
synchronised_transformations:
  - []
data_source_mapping:
  testFirst:
    source: sourceFirstDb
    targets:
      - targetFirstDb
  testSecond:
    source: sourceSecondDb
    targets:
      - targetSecondDb

More information about the config.yaml file structure can be found in configuration reference.

Now you can start the transformation using the following parameters of the command-line interface:

--inventory-file=inventory.yaml --config-file=config.yaml