Data Append mode

Starting from version 1.58, Synthesized TDK supports not only provisioning data into an empty database, but also appending data to an existing database.

This feature is experimental and is subject to change in the future as more feedback is gathered and more cases are covered.

Limitations

Smart data append is only supported for the GENERATION mode. If a table is being transformed in the MASKING mode, the resulting data will be appended to the existing output data without any reconciliation.

Configuration

The append mode is only in effect for operations modes that do not purge the output database before the transformation but at the same time allow the execution to proceed:

schema_create_mode: CREATE_IF_NOT_EXISTS or schema_create_mode: DO_NOT_CREATE
table_truncation_mode: IGNORE

The append mode does not require any additional configuration. It automatically kicks in when the above condition is met.

The behaviour with regard to the data in the parent table is controlled by the parent_data_mode configuration option. The following values are supported:

OLD - only the data that existed in the parent table before the transformation is used to create the output data in the child table.
NEW - only the data that was created by the transformation is used to create the output data in the child table.
ALL (default) - both pre-existing and newly created data are used to create the output data in the child table.

The parent_data_mode setting only has effect in combination with the GENERATION transformation mode. In other modes, it is ignored.

To get hold of how the parent_data_mode options differ, consider the example of two tables linked with a foreign key.

CREATE TABLE public.customer (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL
);

CREATE TABLE public.transaction (
    id SERIAL PRIMARY KEY,
    customer_id INTEGER NOT NULL,
    amount INTEGER NOT NULL,
    date DATE NOT NULL,
    FOREIGN KEY (customer_id) REFERENCES public.customer(id)
);

Let’s say we have the following data in the customer table:

id	name	address_id
1	Alice	11
2	Bob	22

And the following data in the address table:

street

Elm Street

Maple Street

Now, let’s run a transformation with the following configuration:

  - table_name_with_schema: "public.customer"
    transformations:
      - columns: [ address_id ]
        params:
          type: foreign_key_generator
          parent_data_mode: ALL
          distribution: ROUND_ROBIN

This may result in the following data in the address table, with the new data highlighted in bold and color:

id	street
11	Elm Street
22	Maple Street
33	Oak Street

The customer table will have address_id drawn from both pre-existing and newly created data:

id	name	address_id
1	Alice	11
2	Bob	22
3	Charlie	33
4	David	11

The same setup, but with the parent_data_mode set to OLD, would result in the following customer table:

id	name	address_id
1	Alice	11
2	Bob	22
3	Charlie	11
4	David	22

Note that the address_id for the new data is taken from the pre-existing data in the address table for Charlie and David.

If we replace the parent_data_mode with NEW, the customer table will have the data like shown below:

id	name	address_id
1	Alice	11
2	Bob	22
3	Charlie	33
4	David	33

Now, both Charlie and David have the address_id taken from the newly created data in the address table.

The OLD data mode can only work if the parent table is not empty. Otherwise, there would be no data to draw from. If the parent table is empty, an error will be thrown during the configuration validation phase.

Note that parent_data_mode is set per foreign-key transformation, so a table with several foreign keys can use a different mode for each. Not all combinations of parent_data_mode are supported in the current version.

Current TDK release might not support some combinations of parent_data_mode for complex schemas. In such cases, an error will be thrown during the transformation planning phase. As the feature evolves, more corner cases will be covered.

Same-table appending mode

Add generated rows directly to the tables you already have, in the same database, instead of writing to a separate output.

Available from version 1.169.

Overview

By default TDK writes appended data to a separate output data source. With the APPENDING_MODE flag, new rows are inserted beside the existing rows of the source tables themselves. The existing rows are left untouched; only the rows you ask for are added.

This is useful for topping up a test database with more data without rebuilding it, and for generating extra records into a schema that other systems already point at.

Prerequisites

The input and output connections point at the same database, and each schema is mapped to itself. Schema mapping to a different output schema is not supported in this mode.
The database user can create objects in the target schema. TDK creates a temporary lock object there to claim the target for the duration of the run.
Every primary key and unique column has a transformation that produces new values, for example int_sequence_generator or uuid_generator. Passing existing keys through unchanged fails with unique-constraint errors.
A maintenance window for the first run. See Caveats before running against a shared database.

Configuration

The example below adds 1,000 rows to public.customer and leaves every other table as it is:

schemas: ["public"]
metadata:
  schemas:
    - input: "public"
      output: "public"

flags:
  - SAME_DATABASE_TRANSFORMATION
  - APPENDING_MODE

schema_creation_mode: DO_NOT_CREATE
table_truncation_mode: IGNORE
safety_mode: RELAXED

default_config:
  mode: GENERATION
  target_ratio: 0.0

tables:
  - table_name_with_schema: "public.customer"
    target_row_number: 1000
    transformations:
      - columns: ["id"]
        params:
          type: int_sequence_generator

SAME_DATABASE_TRANSFORMATION: Required alongside APPENDING_MODE; tells TDK that source and target are the same database.
schema_creation_mode and table_truncation_mode: Must leave the existing data in place. DROP_AND_CREATE and TRUNCATE are rejected at configuration validation, with a message naming the modes to use instead.
target_ratio: 0.0 in default_config: Tables you do not list get no new rows.
target_row_number: The number of rows added to that table on each run.

Keeping repeat runs scoped

Each run reads the whole source table, including rows that earlier runs appended, so runs get slower as the table grows. To keep the read scoped to your original data, add a filter on the table, for example on a creation timestamp or on a marker column you populate for generated rows:

tables:
  - table_name_with_schema: "public.customer"
    filter: "created_at < '2026-01-01'"
    target_row_number: 1000

The output size does not change, because target_row_number is absolute; the filter only controls how much data each run reads.

Confirm success

After the run, the listed table has grown by target_row_number rows and the rows that existed before are unchanged. Tables you did not list have the same row count as before.

Caveats

Rows are committed progressively as the run proceeds, not in a single transaction. A run that fails part-way leaves the rows it had already written. Give generated rows an identifiable value, for example a constant marker column, so they can be found and removed if needed.
Anything that watches the table sees the new rows: change data capture, replication and downstream ETL. Check the consumers before the first run.
Generated key values are inserted directly, so database sequences behind those keys are not necessarily advanced. Check that your application’s next insert cannot collide with a generated key.
Child tables can link generated rows to your existing parent rows as well as to newly generated ones. Use parent_data_mode on the foreign_key_generator to control this:
```
tables:
  - table_name_with_schema: "public.transaction"
    transformations:
      - columns: ["customer_id"]
        params:
          type: foreign_key_generator
          parent_data_mode: NEW
```
NEW links new child rows only to newly generated parents; ALL, the default, also lets them reference existing parents.
Tables in MASKING mode are appended without any reconciliation with the rows already there, as described at the top of this page.