# YAML configuration
> Source: https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/configuration_reference
> For the complete documentation index, see [llms.txt](https://docs.synthesized.io/llms.txt).
## YAML configuration
`Object.`
To execute a transformation user provides a YAML configuration file with properties of a specific transformation. `UserConfig` is the root object of this configuration file. This object consists of [default table configuration](#DefaultConfig) and a list of `[user-defined table config overrides](#UserTableConfig)` and a set of global settings. Example:
```yaml
default_config:
mode: KEEP
target_ratio: 1.0
tables:
- table_name_with_schema: "public.orders"
mode: MASKING
target_ratio: 0.1
cycle_breaker_references: [ "public.orders" ]
insert_batch_size: 1000
use_working_directory: true
transformations:
- columns: [ "status" ]
params:
type: "categorical_generator"
categories:
value_source: PROVIDED
values:
"Shipped": 8
"Cancelled": 2
cycle_resolution_strategy: FAIL
schema_creation_mode: CREATE_IF_NOT_EXISTS
table_truncation_mode: TRUNCATE
global_seed: 42
safety_mode: "RELAXED"
```
### Properties
- `default_config`: `[Default configuration](#DefaultConfig)`.
- `matchers`: array of `[Matchers](#MatchConfig)`.
- `tables`: array of `[User-defined table configuration](#UserTableConfig)`.
- `cycle_resolution_strategy`: `[Cycle resolution strategy](#CycleResolutionStrategy)`.
- `schema_creation_mode`: `[Schema Creation Mode](#SchemaCreationMode)`.
- `table_truncation_mode`: `[Table truncation mode](#TableTruncationMode)`.
- `safety_mode`: `[Safety Mode](#SafetyMode)`.
- `global_seed`: `String.`
A value used a seed for random number generators. Should be string in base64 encoded format. Basic alphabet (please refer to [https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/Base64.html](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/Base64.html)) The result of generation must be the same each time the generation is being run with the same seed and workflow configuration. By default, this is set to an empty string, which corresponds to an empty byte array.
- `schemas`: array of `String.`
List of schemas to process. If not set or null, all available schemas will be processed.
- `metadata`: `[Additional Metadata](#AdditionalMetadata)`.
- `use_text_column_heuristics`: `Boolean.`
Enables column and table name heuristics for selecting transformations for text fields. If enabled, the `address_generator`, `person_generator` and `finance_generator` can be chosen for text fields in GENERATION mode.
- `flags`: array of `[Configuration Flag](#ConfigFlag)`.
- `scripts`: `[Pre and post execution scripts](#PrePostExecutionScripts)`.
## Default configuration
Used in: `[default_config](#UserConfig_default_config)`
optional `Object.`
Consists of the default table configuration parameters that are applied to all tables by default and a list of [rules](#DefaultConfigItem) that are conditionally applied to the table.
### Properties
- `mode`: `[Transformation mode](#UserTransformationMode)`.
- `target_ratio`: optional `Number (double).`
The relative size of each table of the output database with respect to the input. The number of rows of each output table will be computed by multiplying this parameter by the input table size. If not provided, this parameter will be target\_ratio = 1, resulting on same size for input and output databases. Can be overridden by `target_row_number`
- `target_row_number`: optional `Integer (int64).`
The absolute size of each table of the output database in rows. This parameter is applicable only for GENERATION mode. If not provided, `target_ratio` will be used.
- `insert_batch_size`: optional `Integer.`
Indicates how many table rows are inserted to the database per each batch operation. The default value for this parameter is set at 1000 rows.
- `use_working_directory`: `Boolean.`
Whether to use the configured local working directory to perform certain I/O operations more efficiently. In some cases, using working directory can speed up table insertion speed by several times. However, the more tables use the working directory, the more space is needed for it. Note that this setting is only effective when the working directory is configured globally (see [Working directory properties](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_working_directory)).
- `locale`: `String.`
Default locale for specific transformations, such as `person_generator` and `address_generator`. This value can be overridden at the column transformation level.
Supported locales:
ar
bg
ca
ca-CAT
cs
da-DK
de
de-AT
de-CH
en
en-AU
en-CA
en-GB
en-IN
en-MS
en-NEP
en-NG
en-NZ
en-PAK
en-SG
en-UG
en-US
en-ZA
en-PH
es
es-MX
fa
fi-FI
fr
he
hu
in-ID
it
ja
ko
nb-NO
nl
pl
pt
pt-BR
ru
sk
sv
sv-SE
th
tr
uk
vi
zh-CN
zh-TW
- `in_memory_filter_threshold`: optional `Integer.`
Advanced performance tuning parameter. This parameter determines whether values the parent table are loaded into memory during filtering operations, depending on the table size. If the size of the table does not exceed this specified value, the values will be loaded into memory for filtering for child tables. The higher this threshold, the faster the filter can potentially run, provided that all values fit in memory. But if there is not enough memory, it may cause an error. The default value for this parameter is set at 100\_000 rows.
- `parent_data_mode`: `[Parent data mode](#ParentDataMode)`.
- `checkpoint`: `[Checkpoint](#Checkpoint)`.
- `fetch`: `[Fetch configuration](#FetchConfig)`.
- `items`: array of `[Default configuration item](#DefaultConfigItem)`.
The rules used to select the appropriate transformation depending on the database metadata if one is not specified in the user’s configuration.
- `additional_items`: array of `[Default configuration item](#DefaultConfigItem)`.
Additional rules used to select the appropriate transformation depending on the database metadata if one is not specified in the user’s configuration. These rules take precedence over the rules in `items`: when a field matches both an `additional_items` rule and an `items` rule, the `additional_items` rule wins. Rules are evaluated in order and the first matching rule per field is applied. Use this to layer environment-specific overrides on top of the shared `items` rules without editing them. For example, an `items` rules by default in `MASKING` mode mask every column, while an `additional_items` rule tells the engine to put nulls in all nullable columns. The result of the execution is that all nullable columns will be nullified, while the rest of the columns will be masked. Example:
```yaml
default_config:
mode: MASKING
additional_items:
- id: Nullify nullable
conditions:
- type: is_nullable
transformation:
type: null_generator
```
## Matchers
Used in: `[matchers](#UserConfig_matchers)`
`Object.`
Parameters of a matcher.
### Properties
- `transformation`: `[MatcherTransformation](#MatcherTransformation)`.
Depending on `match_by` property value, can be one of the following:
## User-defined table configuration
Used in: `[tables](#UserConfig_tables)`
`Object.`
The parameters defined in the default configuration are applied to all tables in the database, so there’s no need to configure each table individually. But if needed, the user can override default configuration for any specific table present in the database. For each table, the user can create a `UserTableConfig` and add it to the list `tables` of `[YAML configuration](#UserConfig)`.
### Properties
- `table_name_with_schema`: `String.`
The name of the table affected by this `UserTableConfig`. This value must contain both the schema name and table name, separated by a dot. In other words, this value must follow the format `$schema.$table`. In most cases, this value is case-insensitive (except when a database object has been created with a double-quoted name). Here are some examples of valid values: `HR.employees`, `SCOT.SALGRADE`, `sakila.film`, `public.Actor`, `db2inst1.SALES`. In these examples, `HR`, `SCOT`, `sakila`, `public` and `db2inst1` are schema names, while `employees`, `SALGRADE`, `film`, `Actor` and `SALES` are table names.
> **NOTE**
> In MySQL, a schema is synonymous with a database, as defined in the [MySQL Glossary](https://dev.mysql.com/doc/refman/8.1/en/glossary.html#glos_schema). Therefore, in MySQL, you should always place the database name as the first component of this property.
- `mode`: `[Transformation mode](#UserTransformationMode)`.
- `transformations`: array of `[Column transformation parameters](#ColumnTransformationParams)`.
- `target_ratio`: optional `Number (double).`
The relative size of the output table with respect to the input. The number of rows of each output table will be computed by multiplying this parameter by the input table size. If not provided, this parameter will be `target_ratio = 1`, resulting on same size for input and output databases. Can be overridden by `target_row_number`
> **NOTE**
> When setting target\_ratio at a table level, the result may end up being smaller than the given value due to relationships with parent table. For example, if a customer table is set to `target_ratio = 0.5`, and its child table transactions has `target_ratio = 1.0`, the output transaction table will also end with half it’s samples due to its downstream dependency to the reduced table customer.
- `target_row_number`: optional `Integer (int64).`
The absolute size of the output table in rows. This parameter is applicable only for GENERATION mode. If not provided, `target_ratio` will be used.
- `cycle_breaker_references`: array of `String.`
When `[Cycle resolution strategy](#CycleResolutionStrategy)` is `FAIL`, this list may contain a list of table names, references to which will be ignored during the data generation.
- `insert_batch_size`: optional `Integer.`
Indicates how many table rows are inserted to the database per each batch operation. The default value for this parameter is set at 1000 rows.
- `use_working_directory`: optional `Boolean.`
- `filter`: `String.`
An expression to filter a source table data. Any SQL expression that can be used as the search condition in WHERE clause. For example:
```sql
org_id != 120 and custom_function() = 'foo'
```
- `checkpoint`: `[Checkpoint](#Checkpoint)`.
- `in_memory_filter_threshold`: optional `Integer.`
Advanced performance tuning parameter. This parameter determines whether values the parent table are loaded into memory during filtering operations, depending on the table size. If the size of the table does not exceed this specified value, the values will be loaded into memory for filtering for child tables. The higher this threshold, the faster the filter can potentially run, provided that all values fit in memory. But if there is not enough memory, it may cause an error. The default value for this parameter is set at 100\_000 rows.
- `parent_data_mode`: `[Parent data mode](#ParentDataMode)`.
- `fetch`: `[Fetch configuration](#FetchConfig)`.
## Cycle resolution strategy
Used in: `[cycle_resolution_strategy](#UserConfig_cycle_resolution_strategy)`
optional `String.`
Defines how to deal with cycles in table relations via foreign keys.
`FAIL`
if this mode is selected, `cycle_breaker_references` should be provided in the configuration file. Otherwise, execution will fail if a circular reference is detected in the schema.
`DELETE_NOT_REQUIRED`
if this mode is selected, cyclic references will be resolved automatically by removing the last nullable reference that leads to the cycle. As a result, the cyclic reference columns will be masked to nulls.
`DEFER_FOREIGN_KEY`
Oracle, MSSQL, MySQL and DB2 only. If this mode is selected, cyclic references will be automatically resolved by disabling foreign keys that lead to cycles during insertion. These references are re-enabled after the data has been inserted. As a result, the columns with cyclic references will be processed like any other keys in the schema. This strategy is only available when using `MASKING` or `KEEP` mode, and only with a `target_ratio` of 1.
Example for a cycle breaker reference:
```yaml
schema_creation_mode: CREATE_IF_NOT_EXISTS
cycle_resolution_strategy: FAIL
table_truncation_mode: TRUNCATE
safety_mode: "RELAXED"
default_config:
mode: GENERATION
target_ratio: 1.0
tables:
- table_name_with_schema: "public.employees"
cycle_breaker_references: ["public.employees"]
```
Where the employees table contains a cycle reference.
Enum values
- `FAIL`
- `DELETE_NOT_REQUIRED`
- `DEFER_FOREIGN_KEY`
## Schema Creation Mode
Used in: `[schema_creation_mode](#UserConfig_schema_creation_mode)`
optional `String.`
Defines the mode of schema creation.
> **NOTE**
> The schema creation feature only restores parts of the original schema required for Synthesized execution, such as tables, constraints, and indexes. Other database objects such as triggers, views, procedures, functions, and any vendor-specific properties are not supported.
`CREATE_IF_NOT_EXISTS`
if this mode is selected, DDL schema will be copied from the source database to the target one if it does not exist, existing schema will be used otherwise.
`DO_NOT_CREATE`
if this mode is selected, the existing schema will be used without any validations. Please use this mode carefully: run-time errors may occur if the input and output schema do not match.
`CREATE`
if this mode is selected, DDL schema will be copied from the source database to the target one. The target database should be empty.
`DROP_AND_CREATE`
if this mode is selected, DDL schema will be copied from the source database to the target one. Existing schema in the target database will be dropped. Please use this mode carefully.
Enum values
- `DO_NOT_CREATE`
- `CREATE`
- `CREATE_IF_NOT_EXISTS`
- `DROP_AND_CREATE`
## Table truncation mode
Used in: `[table_truncation_mode](#UserConfig_table_truncation_mode)`
optional `String.`
Defines the mode of table truncation.
`DO_NOT_TRUNCATE`
(default) if this mode is selected, tables in the target database won’t be truncated. An empty target database required.
`TRUNCATE`
if this mode is selected, tables in the target database will be truncated.
`IGNORE`
if this mode is selected, the status of the target database is ignored.
Enum values
- `DO_NOT_TRUNCATE`
- `TRUNCATE`
- `IGNORE`
## Safety Mode
Used in: `[safety_mode](#UserConfig_safety_mode)`
optional `String.`
The mode of choosing transformations.
`STRICT`
if this mode is selected and no matching transformation found, then no `passthrough`, `categorical_generator`, `null_generator`, `constant_generator` transformations will be chosen by default. If no matching transformation found, an error occurs. If `DEFER_FOREIGN_KEY` cycle resolution strategy fails to re-enable a foreign key after transformation, an error occurs.
`RELAXED`
if this mode is selected and no matching transformation found, then the `passthrough` transformation will be applied for `MASKING` mode, and `null_generator` or `categorical_generator` for `GENERATION` mode. Constant generators `constant_numeric`, `constant_string`, `constant_date`, `constant_boolean` will be chosen by default where the source column contains the same value in all rows. If `DEFER_FOREIGN_KEY` cycle resolution strategy fails to re-enable a foreign key after transformation, a warning is shown in logs and transformation continues.
This property has no meaning for `KEEP` mode. Default: `STRICT`.
Enum values
- `STRICT`
- `RELAXED`
## Additional Metadata
Used in: `[metadata](#UserConfig_metadata)`
optional `Object.`
Metadata that is not defined at the database level but must be taken into account when performing transformations.
### Properties
- `tables`: array of `[Table Additional Metadata](#TableAdditionalMetadata)`.
- `schemas`: optional array of `[Schema Mapping](#SchemaMapping)`.
Schema mappings for transformations. Each mapping specifies an input schema to read from and an output schema to write to. Used for single-database transformations or cross-database schema renaming.
## Configuration Flag
Used in: `[flags](#UserConfig_flags)`
optional `String.`
Options for TDK. Available flags:
`DISABLE_DB_INDEXES_ON_WRITE`
If this option enabled, TDK will try to temporarily disable non-unique database indexes before inserting data, subsequently re-enabling them afterward. If the target database does not support re-enabling indexes, TDK will try to drop and re-create them instead of disabling and re-enabling respectively. This should enhance performance during write operations. Presently, this feature is available for SQL Server, MySQL, Oracle.
NOTE: Indexes that are already disabled will be also enabled after the processing is complete.
`DISABLE_DB_TRIGGERS_ON_WRITE`
If this option enabled, TDK will try to temporarily disable triggers before inserting data, subsequently re-enabling them afterward. This should enhance performance during write operations and prevent side-effects from triggers. Presently, this feature is available only for SQL Server.
NOTE: If TDK failed during execution part of disabled triggers might be not enable back.
`USE_FILE_BATCH_INSERT`
if this flag is selected, TDK will try to use file batch insert. Works only for SQL Server. Doesn’t work with direct subsetting (without a working directory). [Value mapping](#ValueMapping) doesn’t work when this flag is enabled. Might be high CPU efficient.
`NOT_FALLBACK_TO_ONE_BY_ONE_INSERTS`
By default, insertion is performed in batches of configurable sizes (see `insert_batch_size` in [Default configuration](#DefaultConfig)). When an error is encountered during the insertion, an attempt may be made to fall back to inserting single records instead of batch. By default, for DB2 and Oracle, TDK falls back to inserting single records instead of batch. For SQL Server, Postgres, MySQL and others TDK falls back to one-by-one inserts only when the SQL error is `INTEGRITY_CONSTRAINT_VIOLATION` or `DATA_EXCEPTION`. The `NOT_FALLBACK_TO_ONE_BY_ONE_INSERTS` flag: Turn off fallback to one-by-one inserts. If one of the batches gets an error all TDK execution will fail. Conflicts with the `SKIP_FAILED_BATCHES` and `FORCE_FALLBACK_TO_ONE_BY_ONE_INSERTS` flag.
`SKIP_FAILED_BATCHES`
By default, insertion is performed in batches of configurable sizes (see `insert_batch_size` in [Default configuration](#DefaultConfig)). When an error is encountered during the insertion, an attempt may be made to fall back to inserting single records instead of batch. By default, for DB2 and Oracle, TDK falls back to inserting single records instead of batch. For SQL Server, Postgres, MySQL and others TDK falls back to one-by-one inserts only when the SQL error is `INTEGRITY_CONSTRAINT_VIOLATION` or `DATA_EXCEPTION`. The `SKIP_FAILED_BATCHES` flag: Skip failed batches without resorting to one-by-one inserts. Conflicts with the `NOT_FALLBACK_TO_ONE_BY_ONE_INSERTS` and `FORCE_FALLBACK_TO_ONE_BY_ONE_INSERTS` flag.
`FORCE_FALLBACK_TO_ONE_BY_ONE_INSERTS`
By default, insertion is performed in batches of configurable sizes (see `insert_batch_size` in [Default configuration](#DefaultConfig)). When an error is encountered during the insertion, an attempt may be made to fall back to inserting single records instead of batch. By default, for DB2 and Oracle, TDK falls back to inserting single records instead of batch. For SQL Server, Postgres, MySQL and others TDK falls back to one-by-one inserts only when the SQL error is `INTEGRITY_CONSTRAINT_VIOLATION` or `DATA_EXCEPTION`. The `FORCE_FALLBACK_TO_ONE_BY_ONE_INSERTS` flag: Force one-by-one inserts in case of failed batches and ignore the type of SQL error. Conflicts with the `NOT_FALLBACK_TO_ONE_BY_ONE_INSERTS` and `SKIP_FAILED_BATCHES` flag.
`FORCE_CLAIM_SINK`
If this option enabled, TDK will treat the output as its own, reclaiming it even if another instance has already claimed it. Should be used by caution, as it may lead to undefined behaviour if multiple instances are writing to the same output.
`START_INT_SEQUENCE_FROM_ZERO`
If this option is enabled, TDK will start generating integer sequences from 0 instead of 1. This flag is added to support backwards compatibility with older TDK versions.
`DO_NOT_CONTINUE_FROM_CHECKPOINT`
If this option enabled, TDK will ignore checkpoints during the incremental transformation process, but the new checkpoints will still be created. Setting this flag is useful when the user wants to perform a full refresh of transformation with previously created checkpoints.
`DO_NOT_PRELOAD_METADATA_SNAPSHOT`
If this option is enabled, metadata snapshot will not be preloaded. This is particularly useful for Oracle and SAP HANA databases with large schemas to reduce memory consumption and improve performance. In this case, metadata snapshot will be loaded from the database on demand if needed during transformation.
`PROPAGATE_FILTER_TO_CHILDREN`
If this option is enabled, filters defined on parent tables will be automatically propagated to child tables via EXISTS subqueries. This optimizes performance by filtering child rows at the database level instead of reading all rows and filtering in-memory. Presently, this feature is available for PostgreSQL and MySQL only.
`DO_NOT_SELECT_COUNT_FOR_MASKING`
If this option is enabled, TDK will avoid execution of nonessential `select count(*)` queries on input tables in MASKING mode. May lead to logging messages with incorrect numbers.
`SAME_DATABASE_TRANSFORMATION`
Enables transformation within a single database instance by allowing the source and target connections to point to the same database. Must be used together with schema mappings in `metadata.schemas` to specify which input schemas map to which output schemas. This allows reading from one schema (e.g., `foo`) and writing to another schema (e.g., `foo_test`) within the same database.
`DISABLE_EAGER_LOAD_TABLES`
If this option is enabled, TDK will not eagerly load small tables. This is particularly useful when the schema contains wide tables where loading small tables eagerly could lead to high memory consumption.
`DO_NOT_USE_ASSISTED_JOINS`
If this option is enabled, TDK will not use assisted joins to generate child tables and instead it will fetch records from the target database. This might be used when there is an issue with assisted storage size. Disabling assisted joins will also lead to higher rate of constraint violations.
`DISABLE_BULK_INSERT`
If this option is enabled, TDK will disable bulk (multi-row) inserts and use single-row prepared statement batching instead.
`IMPRECISE_AGGREGATION`
Uses sampled (TABLE SAMPLE) queries when computing column statistics like value frequencies. Significantly faster on large tables but may miss rare values and produce approximate counts.
`ENABLE_BULK_COLUMN_STATS`
Computes per-column numeric and string-length statistics in one consolidated query per aggregate type per table, instead of one query per column. Much faster learning on wide schemas, but scans every numeric/text column in the table. Off by default; opt in for wide-schema workloads.
`APPENDING_MODE`
Allows transformed rows to be appended to the same table when the output schema matches the input schema. Use this flag together with `SAME_DATABASE_TRANSFORMATION`. Cannot be combined with `table_truncation_mode: TRUNCATE` or `schema_creation_mode: DROP_AND_CREATE`.
Enum values
- `DISABLE_DB_INDEXES_ON_WRITE`
- `DISABLE_DB_TRIGGERS_ON_WRITE`
- `USE_FILE_BATCH_INSERT`
- `NOT_FALLBACK_TO_ONE_BY_ONE_INSERTS`
- `SKIP_FAILED_BATCHES`
- `FORCE_FALLBACK_TO_ONE_BY_ONE_INSERTS`
- `FORCE_CLAIM_SINK`
- `START_INT_SEQUENCE_FROM_ZERO`
- `DO_NOT_CONTINUE_FROM_CHECKPOINT`
- `DO_NOT_PRELOAD_METADATA_SNAPSHOT`
- `PROPAGATE_FILTER_TO_CHILDREN`
- `DO_NOT_SELECT_COUNT_FOR_MASKING`
- `SAME_DATABASE_TRANSFORMATION`
- `DISABLE_EAGER_LOAD_TABLES`
- `DO_NOT_USE_ASSISTED_JOINS`
- `DISABLE_BULK_INSERT`
- `IMPRECISE_AGGREGATION`
- `ENABLE_BULK_COLUMN_STATS`
- `APPENDING_MODE`
## Pre and post execution scripts
Used in: `[scripts](#UserConfig_scripts)`
optional `Object.`
SQL scripts to be executed on the target database before or after the transformation process.
Example:
```yaml
scripts:
pre:
source: INLINE
script: |
ALTER TABLE public.transaction DISABLE TRIGGER ALL;
post:
source: FILE
path: post_script.sql
```
The post script will be executed after the transformation process in any case, including errors during the execution of the pre-script or the transformation process.
### Properties
- `pre`: `[Pre and post execution script](#PrePostExecutionScript)`.
- `post`: `[Pre and post execution script](#PrePostExecutionScript)`.
## Transformation mode
Used in: `[mode](#DefaultConfig_mode)`, `[mode](#UserTableConfig_mode)`
optional `String.`
Defines table processing mode.
`KEEP`
if this mode is selected, the original data will be copied as it is. When this mode is selected, the output size needs to be smaller than the input, i.e. `target_ratio <= 1`.
`MASKING`
if this mode is selected, masking transformations will be applied to the original data. When this mode is selected, the output size needs to be smaller than the input, i.e. `target_ratio <= 1`.
`GENERATION`
if this mode is selected, the synthesized engine will learn the original data and generate new synthetic data. For this mode, the output database can be bigger than the input, so `target_ratio` can be greater than 1.
> **NOTE**
> Both `KEEP` and `MASKING` modes apply a transformation to original data. While KEEP uses passthrough as default transformation, while MASKING automatically assigns a privacy preserving masking transformation to all columns. See transformations list for more details. For all modes, the user can override default transformers.
Enum values
- `MASKING`
- `GENERATION`
- `KEEP`
## Parent data mode
Used in: `[parent_data_mode](#DefaultConfig_parent_data_mode)`, `[parent_data_mode](#UserTableConfig_parent_data_mode)`
`String.`
What part of parent data to consider for the child table processing. Default is `ALL`. If defined on the table level, the parent data mode is applied to all suitable transformations of the table.
Enum values
- `NEW`
- `OLD`
- `ALL`
## Checkpoint
Used in: `[checkpoint](#DefaultConfig_checkpoint)`, `[checkpoint](#UserTableConfig_checkpoint)`
optional `Object.`
Represents an entry point for handling repeated transformations.
Depending on `mode` property value, can be one of the following:
## Fetch configuration
Used in: `[fetch](#DefaultConfig_fetch)`, `[fetch](#UserTableConfig_fetch)`
optional `Object.`
Configuration for how data is read from the source database per table. When not specified, defaults to streaming with a fetch size of 1000.
Depending on `type` property value, can be one of the following:
## Default configuration item
Used in: `[items](#DefaultConfig_items)`, `[additional_items](#DefaultConfig_additional_items)`
`Object.`
The rule that applies to the tables by default, written in form "if given [conditions](#Condition) are met, the given [Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations) are applied."
### Properties
- `id`: optional `String.`
Optional identifier of the rule (generally needed for debugging).
- `conditions`: array of `[Condition](#Condition)`.
The list of conditions that must be met in order for the transformation params to be applied.
- `transformation`: `[Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations)`.
## MatcherTransformation
Used in: `[transformation](#MatchConfig_transformation)`
`Object.`
Filter for complex matching.
### Properties
- `params`: `[Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations)`.
- `mapping`: `[ValueMapping](#ValueMapping)`.
- `mode`: `[Transformation mode](#UserTransformationMode)`.
## Universal matcher
`[Matchers](#MatchConfig)`
Selects every column in the connected database, ignoring schema, table, name, and data-type constraints.
### Properties
- `match_by = all`
- `comment`: `String.`
A comment about the match configuration (optional).
## SQL type matcher
`[Matchers](#MatchConfig)`
Targets columns by their SQL data type (e.g., INTEGER, VARCHAR, TIMESTAMP).
### Properties
- `match_by = sql_type`
- `types`: array of `String.`
List of types to match.
- `comment`: `String.`
A comment about the match configuration (optional).
## Column name matcher
`[Matchers](#MatchConfig)`
Filters columns by column identifier only, independent of the containing table or schema.
### Properties
- `match_by = column_name`
- `columns`: array of `String.`
List of columns to match.
- `comment`: `String.`
A comment about the match configuration (optional).
## Table name matcher
`[Matchers](#MatchConfig)`
Limits the match scope to columns that reside in specific tables.
### Properties
- `match_by = table_name`
- `tables`: array of `String.`
List of tables to match.
- `comment`: `String.`
A comment about the match configuration (optional).
## Fully qualified name matcher
`[Matchers](#MatchConfig)`
Matches columns using the full schema.table.column path.
### Properties
- `match_by = fully_qualified_column_name`
- `names`: array of `String.`
List of names to match.
- `comment`: `String.`
A comment about the match configuration (optional).
## Composite filter matcher
`[Matchers](#MatchConfig)`
Provides an advanced filter that can combine schema, table, column, and SQL-type predicates with AND logic.
### Properties
- `match_by = complex_filter`
- `filter`: `[Filter](#Filter)`.
- `comment`: `String.`
A comment about the match configuration (optional).
## Column transformation parameters
Used in: `[transformations](#UserTableConfig_transformations)`
`Object.`
List of column names associated with [Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations).
### Properties
- `columns`: array of `String.`
List of columns that are affected by this generator.
- `params`: `[Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations)`.
- `id`: optional `String.`
- `mapping`: `[ValueMapping](#ValueMapping)`.
- `mode`: `[Transformation mode](#UserTransformationMode)`.
## Table Additional Metadata
Used in: `[tables](#AdditionalMetadata_tables)`
`Object.`
### Properties
- `table_name_with_schema`: `String.`
- `foreign_keys`: `[Additional Foreign Keys](#ForeignKeysAdditionalMetadata)`.
## Schema Mapping
Used in: `[schemas](#AdditionalMetadata_schemas)`
`Object.`
Maps an input schema to an output schema for transformations. Used when reading from one schema and writing to another.
### Properties
- `input`: `String.`
The source schema name to read from
- `output`: `String.`
The target schema name to write to
## Pre and post execution script
Used in: `[pre](#PrePostExecutionScripts_pre)`, `[post](#PrePostExecutionScripts_post)`
optional `Object.`
Depending on `source` property value, can be one of the following:
## Refresh Checkpoint
`[Checkpoint](#Checkpoint)`
Represents a checkpoint for refreshing transformations. It is equivalent to full re-execution of the table transformation. CAUTION: This checkpoint will always start from the beginning of the table, ignoring any previously processed data. This may lead to data duplication or constraints violation in the target database if the transformation has already been partially executed.
### Properties
- `mode = refresh`
## Ignore Table Checkpoint
`[Checkpoint](#Checkpoint)`
Represents a checkpoint for ignoring table on hitting checkpoint in any other table. NOTE: This checkpoint will skip the table transformation, which may lead to missing the data in the target database for this table. If you want to ensure that the table in the output database is up to date, use the `refresh` or `incremental` checkpoint instead.
### Properties
- `mode = ignore_table`
## Incremental Checkpoint
`[Checkpoint](#Checkpoint)`
Represents a checkpoint for incremental transformations. Pivots on a specific column given by `column_name`, allowing the transformation to resume from the value in that column, specified by the `source` property.
### Properties
- `mode = incremental`
- `column_name`: `String.`
The name of the column
- `source`: `[Incremental Source](#IncrementalSource)`.
## Streaming fetch configuration
`[Fetch configuration](#FetchConfig)`
Uses JDBC server-side cursors (`fetchLazy()`). This is the default and provides the lowest memory overhead, but does not survive connection drops — if the connection is lost mid-read, the entire read fails and must restart from the beginning.
### Properties
- `type = streaming`
- `fetch_size`: `Integer.`
Number of rows per fetch batch. Defaults to 1000 if not specified.
## Chunking fetch configuration
`[Fetch configuration](#FetchConfig)`
Uses keyset pagination (on primary key or unique key) to read data in independent stateless chunks. Each chunk is a separate query bounded by a checkpoint, so the read can resume from the last checkpoint after a connection drop. Not all dialects support chunking; specifying chunking for an unsupported dialect is a configuration error.
### Properties
- `type = chunking`
- `chunk_size`: `Integer.`
Number of rows per chunk. Defaults to 1000 if not specified.
- `parallelism`: `Integer.`
Maximum number of chunks for the same table read concurrently, each on its own pooled connection. Defaults to 1 (sequential reads from a single connection, the original behavior). Parallel reads do not preserve total row order across the table. Supported only for MySQL; other dialects ignore this field for now.
## Condition
Used in: `[conditions](#DefaultConfigItem_conditions)`
`Object.`
Condition on which the default parameters are being applied.
Depending on `type` property value, can be one of the following:
## ValueMapping
Used in: `[mapping](#MatcherTransformation_mapping)`, `[mapping](#ColumnTransformationParams_mapping)`
`Object.`
The pair of SQL expressions which are applied while reading and writing values from the specified column. The expression should be a valid SQL that can contain the macro `?`. This macro will be replaced with a column name on reading and transformed value on writing.
> **NOTE**
> When using plain SQL expressions, you must guarantee syntax integrity. You may create the possibility of malicious SQL injection. Be sure to properly use this functionality.
### Properties
- `read`: `String.`
The SQL expression which is applied while reading the column value from a source database. The value "?" can be used if the mapping is not required.
- `write`: `String.`
The SQL expression which is applied while writing the column value to a target database. The value "?" can be used if the mapping is not required.
## Filter
Used in: `[filter](#ComplexFilterMatchConfig_filter)`
`Object.`
Filter for complex matching.
### Properties
- `schemas`: array of `String.`
List of schemas to match.
- `tables`: array of `String.`
List of tables to match.
- `columns`: array of `String.`
List of columns to match.
- `types`: array of `String.`
List of sql types to match.
## Additional Foreign Keys
Used in: `[foreign_keys](#TableAdditionalMetadata_foreign_keys)`
map of String keys to\`[Additional Foreign Key](#ForeignKeyAdditionalMetadata)\`.
Additional foreign keys that are not defined at the database level but must be taken into account when performing transformations.
## Inline pre and post execution script
optional `[Pre and post execution script](#PrePostExecutionScript)`
The inline SQL script that will be executed before or after the transformation process.
### Properties
- `source = INLINE`
- `script`: `String.`
SQL script to be executed.
## File pre and post execution script
optional `[Pre and post execution script](#PrePostExecutionScript)`
The file with SQL script that will be executed before or after the transformation process.
### Properties
- `source = FILE`
- `path`: `String.`
Script file location. The script can be located on local file system, AWS S3 and Google Storage. In the case of a local file system, the path can be absolute or relative to the application process’s working directory (not to be confused with [working directory](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_working_directory)) To be able to load scripts from S3 the property `TDK_AWS_ENABLED==true` should be set. More details can be found [here](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_aws_enabled). The property `TDK_GCP_ENABLED==true` allows loading scripts from Google Storage. More details can be found [here](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_gcp_enabled).
## Incremental Source
Used in: `[source](#IncrementalCheckpoint_source)`
`Object.`
A source for the value of the column used for incremental transformations.
Depending on `type` property value, can be one of the following:
## IsKeyCondition
`[Condition](#Condition)`
The column is a part of either primary or a foreign key.
### Properties
- `type = is_key`
## IsPrimaryKeyCondition
`[Condition](#Condition)`
The column is a part of a primary key.
### Properties
- `type = is_primary_key`
## IsForeignKeyCondition
`[Condition](#Condition)`
The column is a part of a foreign key.
### Properties
- `type = is_foreign_key`
## IsIgnoredForeignKeyCondition
`[Condition](#Condition)`
Applied to columns that are the part of FK which is ignored due to cycles
### Properties
- `type = is_ignored_foreign_key`
## IsDetachedForeignKeyCondition
`[Condition](#Condition)`
Applied to nullable columns of a foreign key whose parent table yields no rows and starts empty, so the reference cannot be satisfied. Under MATCH SIMPLE a NULL in the tuple detaches the row from the parent.
### Properties
- `type = is_detached_foreign_key`
## IsUniqueCondition
`[Condition](#Condition)`
The column is either a part of primary key or `UNIQUE` constraint.
### Properties
- `type = is_unique`
## IsNullableCondition
`[Condition](#Condition)`
The column is nullable
### Properties
- `type = is_nullable`
## ModeInCondition
`[Condition](#Condition)`
The transformation mode is in a given array.
### Properties
- `type = mode_in`
- `modes`: array of `[Transformation mode](#UserTransformationMode)`.
## ParentModeInCondition
`[Condition](#Condition)`
If the field refers to a FK, check whether the parent table is of specific modes
### Properties
- `type = parent_mode_in`
- `modes`: array of `[Transformation mode](#UserTransformationMode)`.
## TrueCondition
`[Condition](#Condition)`
Always true, thus making the rule applicable to every column.
### Properties
- `type = true`
## DataTypeCondition
`[Condition](#Condition)`
The column has one of the given data types.
### Properties
- `type = data_type`
- `data_type`: `[Transformation data type](#TransformationDataType)`.
## UniqueValuesCondition
`[Condition](#Condition)`
Check whether the given field can be modelled as a [format preserving hashing](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations#FormatPreservingHashingParams) column instead of categorical.
### Properties
- `type = unique_values`
- `unique_ratio_threshold`: `Number (double).`
The fraction of unique values. The null values are not taken into account.
- `min_table_size_threshold`: `Integer.`
Minimum table size, as for small tables the `unique_ratio_threshold` can lead to false positives.
## IsEmptyCondition
`[Condition](#Condition)`
The column is empty.
### Properties
- `type = is_empty`
## ParentTransformationParamsCondition
`[Condition](#Condition)`
If the field refers to a FK, check whether the parent column is transformed by a specific transformer
### Properties
- `type = parent_transformation`
- `parent_transformation_params`: `[Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations)`.
## NoParentTransformationParamsCondition
`[Condition](#Condition)`
If the field refers to a FK, check that the parent transformation for specific column is not defined
### Properties
- `type = no_parent_transformation`
## IsUuidCondition
`[Condition](#Condition)`
The column is of UUID type.
### Properties
- `type = is_uuid`
## IsIdentityCondition
`[Condition](#Condition)`
The column is IDENTITY.
### Properties
- `type = is_identity`
## IsAutogeneratedCondition
`[Condition](#Condition)`
The column is AUTOGENERATED.
### Properties
- `type = is_autogenerated`
## SingleDistinctValueCondition
`[Condition](#Condition)`
The column contains only one distinct value. The null values are not taken into account.
### Properties
- `type = single_distinct_value`
## DistinctValuesCondition
`[Condition](#Condition)`
Check whether the given field distinct value number is greater than the specified threshold. The null values are not taken into account.
### Properties
- `type = distinct_values`
- `minimum_threshold`: `Integer.`
Distinct value number (non-negative)
- `maximum_threshold`: optional `Integer.`
Distinct value number (non-negative)
## StandardDeviationCondition
`[Condition](#Condition)`
Check whether the given field standard deviation is greater than the specified threshold.
### Properties
- `type = std_dev`
- `std_dev_threshold`: `Number (double).`
Standard deviation threshold (non-negative)
## NotCondition
`[Condition](#Condition)`
Negation of the specified condition
### Properties
- `type = not`
- `condition`: `[Condition](#Condition)`.
## SafetyModeCondition
`[Condition](#Condition)`
The given safety mode is used.
### Properties
- `type = safety_mode`
- `mode`: `[Safety Mode](#SafetyMode)`.
## IsIgnoredTableCondition
`[Condition](#Condition)`
True if the table is ignored, _e.g._ `target_ratio` is 0
### Properties
- `type = is_ignored_table`
## TextColumnHeuristicsEnabledCondition
`[Condition](#Condition)`
Check if the property `use_text_column_heuristics` is enabled
### Properties
- `type = text_column_heuristics`
## ColumnTableNameRegexCondition
`[Condition](#Condition)`
Check if the column and table name matches the specified regular expression patterns. During the detection process, table and column names will be preprocessed: \* they will be lower-cased \* non-alpha characters will be removed
### Properties
- `type = column_table_name_regex`
- `column_regex`: `String.`
Column selection pattern.
- `table_regex`: `String.`
Table selection pattern. Any table matches by default.
## IsComputableColumnCondition
`[Condition](#Condition)`
Applied to columns that are auto-computable.
### Properties
- `type = is_computable_column_condition`
## IsEnum
`[Condition](#Condition)`
Applied to columns that are enum.
### Properties
- `type = is_enum`
## IsColumnWithLargeValuesCondition
`[Condition](#Condition)`
The column has large values, which are bigger than the threshold
### Properties
- `type = is_large_value_column`
## HasDefaultValue
`[Condition](#Condition)`
The column type is supported in DefaultValueGenerator.
### Properties
- `type = has_default_value_generator`
## ExternalClassifierCondition
`[Condition](#Condition)`
Check if the field’s external classifiers contain the specified classifier.
### Properties
- `type = external_classifier`
- `classifier`: `String.`
The name of the external classifier to check for.
## Additional Foreign Key
Used in: `[foreign_keys](#TableAdditionalMetadata_foreign_keys)`
`Object.`
### Properties
- `referred_database`: `String.`
Slug of the datasource holding the referred table, when it is not the database being transformed. Values of the covered columns are then generated from the keys of the referred table in that datasource. Resolving a slug requires the run to be orchestrated by Governor; in masking mode the covered columns are copied over unchanged.
- `referred_schema`: `String.`
- `referred_table`: `String.`
- `columns`: array of `[Additional Foreign Key Column Mapping](#ForeignKeyColumnMappingAdditionalMetadata)`.
## Last Value Incremental Source
`[Incremental Source](#IncrementalSource)`
Represents a source for incremental transformations that uses the last processed value in the specified column. The last processed value is recorded by TDK and used to filter the data in the specified column, allowing the transformation to resume from this point.
### Properties
- `type = last_value`
## Constant Incremental Source
`[Incremental Source](#IncrementalSource)`
Represents a source for incremental transformations that uses a constant value, specified by the `value` property. This is useful for managing transformations that need to start from a specific point in the data.
### Properties
- `type = constant`
- `start_from`: `String.`
The value from which the transformation should start. This value is used to filter the data in the specified column, allowing the transformation to resume from this point. This field accepts SQL syntax.
## Transformation data type
Used in: `[data_type](#DataTypeCondition_data_type)`
`String.`
Enum values
- `TEXT`
- `NUMERIC`
- `DATE`
- `BOOLEAN`
- `ANY`
## Additional Foreign Key Column Mapping
Used in: `[columns](#ForeignKeyAdditionalMetadata_columns)`
`Object.`
### Properties
- `column`: `String.`
- `referred_column`: `String.`