# Transformations > Source: https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations > For the complete documentation index, see [llms.txt](https://docs.synthesized.io/llms.txt). ## Transformations optional `Object.` Parameters of a transformation. All parameters have a type key with the type name of the transformation, and other parameters that are transformation-specific. Depending on `type` property value, can be one of the following:

Key

Link

Modes

Data types

Multiple columns

categorical_generator

Categorical generator

GENERATION, MASKING

Yes

conditional_generator

Conditional generator

GENERATION, MASKING

Yes

continuous_generator

Continuous generator

GENERATION, MASKING

No

quantile_generator

Quantile generator

GENERATION, MASKING

No

foreign_key_generator

Foreign key generator

GENERATION, MASKING

Yes

unique_generator

Unique generator

GENERATION, MASKING

Yes

format_preserving_hashing

Format preserving hashing

MASKING

No

formatted_string_generator

Formatted string generator

GENERATION, MASKING

No

int_sequence_generator

Integers sequence generator

GENERATION, MASKING

No

string_sequence_generator

String sequence generator

GENERATION, MASKING

No

noising

Noising transformation

MASKING

No

null_generator

Null generator

GENERATION, MASKING

Yes

passthrough

Passthrough transformation

MASKING

Yes

person_generator

Person generator

GENERATION, MASKING

Yes

address_generator

Address generator

GENERATION, MASKING

Yes

finance_generator

Finance generator

GENERATION, MASKING

Yes

redaction

Redaction masker

MASKING

No

unique_hashing

Unique hashing

MASKING

No

date_time_unique_hashing

Date unique hashing

MASKING

No

enum_unique_hashing

Enum unique hashing

MASKING

No

date_generator

Date generator

GENERATION, MASKING

No

uuid_generator

UUID generator

GENERATION, MASKING

No

constant_numeric

Constant numeric generator

GENERATION, MASKING

No

constant_string

Constant string generator

GENERATION, MASKING

No

default_value

Default value generator

GENERATION, MASKING

No

constant_date

Constant date generator

GENERATION, MASKING

No

constant_boolean

Constant boolean generator

GENERATION, MASKING

No

loop_generator

Loop generator

GENERATION, MASKING

Yes

constant_xml

Constant XML generator

GENERATION, MASKING

No

json_pointer_transformer

JSON pointer transformer

MASKING

No

xpath_transformer

Xml XPath transformer

MASKING

No

json_multi_value_transformer

JSON multi-value transformer

MASKING

No

t24_id_remap

T24 id remap

MASKING

No

void_generator

Void generator

GENERATION, MASKING

Yes

scripting_transformer

Scripting transformer

GENERATION, MASKING

Yes

arithmetic_generator

Arithmetic generator

GENERATION, MASKING

No

sieve_generator

Sieve generator

GENERATION, MASKING

Yes

## Categorical generator `[Transformations](#TransformationParams)` Randomly sample from a given key / value of categories and weights. Weights and categories can be provided from various data sources or learned from data. The following example demonstrates how to accomplish this: ```yaml transformations: - columns: - transaction_type params: type: categorical_generator categories: value_source: PROVIDED values: "sent": 0.5 "received": 0.4 "skipping_value": 0.0 "null": 0.1 ``` You can also specify a CSV file as the source of categories. The following example illustrates the complete configuration example: ```yaml transformations: - columns: - transaction_type params: type: categorical_generator categories: value_source: CSV_FILE path: src/e2e/resources/data_with_header.csv null_values: ["null", ""] format: columns: column_accessor_type: NAME categories: "title" weights: "rank" encoding: "UTF-8" delimiter: "," trim: true ``` For simple scenarios, the minimal configuration may be useful, which relies on the defaults mentioned earlier. The following example illustrates this approach: ```yaml transformations: - columns: - transaction_type params: type: categorical_generator categories: value_source: CSV_FILE path: src/e2e/resources/data.csv ``` This transformation can be applied to several columns at once. This illustrates how the generator can handle multiple columns with provided tuples of categories that always appear together: ```yaml transformations: - columns: - productcode - productname params: type: categorical_generator categories: value_source: MULTIPLE_PROVIDED null_values: ["nil"] category_values: - values: productcode: "P1" productname: "Product 1" weight: 0.5 - values: productcode: "P2" productname: "Product 2" weight: 0.3 - values: productcode: "P3" productname: "Product 3" weight: 0.2 - values: productcode: "nil" productname: "nil" weight: 0.5 ``` You can also specify a CSV file as the source of categories for multiple columns. ```yaml transformations: - columns: - productcode - productname params: type: categorical_generator categories: value_source: MULTIPLE_CSV_FILE path: src/e2e/resources/data_multi.csv ``` Example of advanced configuration with multiple columns and multiple categories: ```yaml transformations: - columns: - productcode - productname params: type: categorical_generator categories: value_source: MULTIPLE_CSV_FILE path: src/e2e/resources/data_with_header_multi.csv null_values: ["null", ""] format: columns: column_accessor_type: NAME categories: productcode: "code" productname: "name" weights: "rank" encoding: "UTF-8" delimiter: "," trim: true ``` > **WARNING** > This generator normalizes weights to the probability interval. If the sum of weights exceeds the capacity of the `double` format, you need to use weights with a smaller scale factor. ### Properties - `type = categorical_generator` - `categories`: `[Categories source configuration](#Categories)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Conditional generator `[Transformations](#TransformationParams)` Uses one of two transformations/generators depending on the value of a column. If the value of the conditional column equals `conditional_value`, then the `if_true` transformation is used, otherwise the `if_false` transformation is used. Conditioning on the same table: Omit `conditional_table` (or set it to the table being transformed) to condition on the row currently being generated; no foreign key is required. Below, `tracking_reference` is only set for orders whose `status` is `DONE`: ```yaml transformations: - columns: [ "status" ] params: type: categorical_generator categories: value_source: PROVIDED values: "PENDING": 51 "DONE": 49 - columns: [ "tracking_reference" ] params: type: conditional_generator # No conditional_table, so the condition is evaluated against the # "status" column of the same row. conditional_column: "status" conditional_value: "DONE" if_true: type: formatted_string_generator pattern: "TRK-[0-9]{8}" if_false: type: null_generator ``` Conditioning on a parent table: Set `conditional_table` to a parent table to condition on the referenced parent row. Below, `order_completed` is `true` for orders whose `public.delivery` row has `status` `DONE`: ```yaml transformations: - columns: [ "order_completed" ] params: type: conditional_generator conditional_table: "public.delivery" conditional_column: "status" conditional_value: "DONE" if_true: type: constant_boolean value: true if_false: type: constant_boolean value: false ``` When conditioning on a parent table, the table being transformed must have a foreign key referencing `conditional_table`. In the above example, `public.orders` must have a foreign key to `public.delivery`. > **NOTE** > If the foreign key required by the generator is not present in your database schema, you can define a virtual foreign key as shown [here](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/virtual_foreign_keys#UserConfig_metadata) ### Properties - `type = conditional_generator` - `conditional_column`: `String.` Column the condition is evaluated on. It belongs to `conditional_table` when that is set, otherwise to the table being transformed. - `conditional_table`: optional `String.` Parent table holding the conditional column. Optional in database configs, where omitting it conditions on the same row; required in XML configs; not supported in CSV configs. - `conditional_value`: optional `String.` Value to be compared with. If the value of the conditional column is equal to `conditional_value`, then `if_true` generator is used, otherwise `if_false` generator. - `if_false`: `[Transformations](#TransformationParams)`. - `if_true`: `[Transformations](#TransformationParams)`. - `reference_columns`: optional array of `String.` Column list of the foreign key that is used to match parent and child records. This parameter is only needed when there are multiple foreign keys between the current and conditional tables. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Continuous generator `[Transformations](#TransformationParams)` Output data is sampled from a parameterized continuous distribution. If parameters are not given, they will be fitted from the original data. Example: ```yaml transformations: - columns: - "amount" params: type: continuous_generator mean: 354.21 std: 98.96 min: 0.0 ``` ### Properties - `type = continuous_generator` - `mean`: optional `Number (double).` Mean of the sampled distribution - `std`: optional `Number (double).` Standard Deviation - `min`: optional `Number (double).` Minimum value. Must be strictly smaller than `max`. - `max`: optional `Number (double).` Maximum value. Must be strictly larger than `min`. - `numeric_type`: `[Numeric type](#NumericType)`. - `round`: optional `Integer.` If given, output data will be rounded to this number of digits after the decimal point. - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## Quantile generator `[Transformations](#TransformationParams)` Given a list of probabilities and bin edges, the output data is sampled from a mixture of uniform distributions, where each uniform distribution `i` is chosen with probability `probabilities[i]` and its edges are given by `bin_edges[i]` and `bin_edges[i + 1]`. If parameters are not given, they will be fitted from the original data. Example: ```yaml transformations: - columns: ["amount"] params: type: quantile_generator hist: [0.5, 0.2, 0.3] bin_edges: [0.1, 0.15, 0.3, 0.45] numeric_type: DOUBLE ``` ### Properties - `type = quantile_generator` - `hist`: optional array of `Number (double).` Probabilities of each uniform distribution. - `bin_edges`: optional array of `Number (double).` Bin edges of each uniform distribution. - `numeric_type`: `[Numeric type](#NumericType)`. - `consistent_with_column`: optional `String.` When set, produced values will be consistent with the specified column. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## Foreign key generator `[Transformations](#TransformationParams)` Fills columns with the parent table’s primary key values of a random row. > **WARNING** > Normally this generator is being created implicitly wherever data generation for tables related with foreign keys is needed, so it should not be explicitly set up by the user. ### Properties - `type = foreign_key_generator` - `distribution`: `[Foreign key distribution](#ForeignKeyDistribution)`. - `parent_data_mode`: `[Parent data mode](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/configuration_reference#ParentDataMode)`. - `referred_schema`: optional `String.` - `referred_table`: optional `String.` - `referred_fields`: optional array of `String.` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Unique generator `[Transformations](#TransformationParams)` This generator is intended for the case where primary key values are part of the foreign key. > **WARNING** > Normally this generator is being created implicitly wherever data generation for tables related with foreign keys is needed, so it should not be explicitly set up by the user. ### Properties - `type = unique_generator` - `transformations`: optional array of `[Column transformation parameters](#UniqueGeneratorColumnTransformationParams)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Format preserving hashing `[Transformations](#TransformationParams)` A hash transformation is applied to each character, which included into the configured group, in a given text so that the output preserves the format but contains different characters. This transformation is secure and non-reversible. Examples: Default configuration: ```yaml transformations: - columns: ["registration_number"] params: type: format_preserving_hashing groups: - selector: type: digits alphabets: - type: digits - selector: type: lower_letters alphabets: - type: lower_letters - selector: type: upper_letters alphabets: - type: upper_letters ``` Mask only last 5 characters: ```yaml transformations: - columns: ["registration_number"] params: type: format_preserving_hashing filter: type: "last" n: 5 ``` Mask only substring ignoring case: ```yaml transformations: - columns: ["registration_number"] params: type: format_preserving_hashing filter: type: substring substring: sub ignore_case: true ``` Mask only a set of characters ignoring case: ```yaml transformations: - columns: ["registration_number"] params: type: format_preserving_hashing filter: type: characters characters: "abc" ignore_case: true ``` Mask characters selected by regex with a custom alphabet: ```yaml transformations: - columns: ["phone_number"] params: type: format_preserving_hashing groups: - selector: type: regex pattern: "[123]" alphabets: - type: custom parts: - type: characters characters: "456" - type: characters characters: "789" - type: unicode_block name: LATIN_EXTENDED_D - type: unicode_block name: "Latin Extended-A" - type: unicode_range from: 0x0D00 to: 0x0D7F ``` ### Properties - `type = format_preserving_hashing` - `groups`: array of `[Hashing group](#FormatPreservingHashingGroup)`. Hashing groups to apply on top of the specified filter. There can be multiple groups configured. In that case the groups will be tried to match a region within the filtered value in the order they are specified in configuration. If a match is successfully found, the corresponding group’s alphabet will be used for transformation, and no other groups will be tried for that region. This implies that most specific hashing groups must be specified first in the configuration. Unspecified parameter or `null` is equivalent to the following: ```yaml transformations: - columns: ["registration_number"] params: type: format_preserving_hashing groups: - selector: type: digits alphabets: - type: digits - selector: type: lower_letters alphabets: - type: lower_letters - selector: type: upper_letters alphabets: - type: upper_letters - selector: type: word_characters alphabets: - type: digits - type: lower_letters - type: upper_letters ``` - `locale`: optional `String.` To generate a string sequence with letters from different alphabets, the user can change this parameter when a hashing group is not explicitly specified. The default value is `en-GB`, representing the Latin alphabet. Supported locales: - `en`, `ca`, `cs`, `da`, `de`, `es`, `fi`, `fr`, `hu`, `in`, `it`, `nb`, `nl`,`pl`, `pt`, `sk`, `sv`, `tr`, `vi` - The Latin alphabet characters - `ar`, `fa` - The `ARABIC` Unicode block - `bg`, `ru`, `uk` - The `CYRILLIC` Unicode block - `he` - The `HEBREW` Unicode block - `ja` - The `HIRAGANA`, `KATAKANA`, `CJK_UNIFIED_IDEOGRAPHS` Unicode block - `ko` - The `HANGUL_JAMO` Unicode block - `th` - The `THAI` Unicode block - `zh` - The `CJK_UNIFIED_IDEOGRAPHS` Unicode block - `filter`: `[Format preserving hashing filter](#FormatPreservingHashingFilter)`. - `length_threshold`: optional `Integer (int32).` The length threshold at which the masker is applicable. If any value in the column exceeds this length, execution will not start and will raise a validation error. Compatible modes: **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Formatted string generator `[Transformations](#TransformationParams)` Generate a string column based on a given pattern. If the pattern is not given, will generate random characters with similar length as original column. Example: ```yaml transformations: - columns: - "phone_number" params: type: formatted_string_generator pattern: "\\+44[0-9]{10}" ``` ### Properties - `type = formatted_string_generator` - `pattern`: optional `String.` Regular expression pattern used to sample data from - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## Integers sequence generator `[Transformations](#TransformationParams)` Generate a sequence of integers that represent a unique id column that contain unique values. ### Properties - `type = int_sequence_generator` - `start_from`: optional `Integer.` Where to start the sequence from, default to 0. If the generator is used on existing data, this should be used as the maximum of the existing data plus 1. Example: ```yaml transformations: - columns: - "user_id" params: type: int_sequence_generator ``` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## String sequence generator `[Transformations](#TransformationParams)` Generate a sequence of strings that represent a unique id column that contain unique values, including uppercase alphabetic and numeric values. Example: ```yaml transformations: - columns: ["country_id"] params: type: string_sequence_generator ``` ### Properties - `type = string_sequence_generator` - `length`: optional `Integer.` Maximum length of the column, extracted from the database DDL if not given - `start_from`: optional `String.` Where to start the sequence from, default empty string. If the generator is used on existing data, this should be used as the maximum of the existing data with shift. - `alphabets`: array of `[Alphabet](#Alphabet)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Noising transformation `[Transformations](#TransformationParams)` Add laplacian noise to the input column in order to protect the privacy but output similar values. Example: ```yaml transformations: - columns: ["product_price"] params: type: noising sensitivity: 23.47 min: 0 ``` ### Properties - `type = noising` - `sensitivity`: optional `Number (double).` Amount of noise to be added - `min`: optional `Number (double).` If there’s a hard minimum, transformation will truncate output values there if smaller - `max`: optional `Number (double).` If there’s a hard maximum, transformation will truncate output values there if greater Compatible modes: **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## Null generator `[Transformations](#TransformationParams)` The output column is filled with null values Example: ```yaml transformations: - columns: ["empty_column"] params: type: null_generator ``` ### Properties - `type = null_generator` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Passthrough transformation `[Transformations](#TransformationParams)` The output data is equal to the input, no transformation is applied. Example: ```yaml transformations: - columns: ["customer_number", "plate"] params: type: passthrough ``` ### Properties - `type = passthrough` Compatible modes: **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Person generator `[Transformations](#TransformationParams)` Generate personal fields (_e.g.,_ name, surname, title) and keep them consistent across columns. Available templates are: - `${email}` - `${title}` - `${first_name}` - `${male_first_name}` - `${female_first_name}` - `${last_name}` - `${full_name}` - `${username}` - `${company}` - `${phone_national}` - `${phone_international}` - `${ssn}` Supported locales:

ar

bg

ca

ca-CAT

cs

da-DK

de

de-AT

de-CH

en

en-AU

en-CA

en-GB

en-IN

en-MS

en-NEP

en-NG

en-NZ

en-PAK

en-SG

en-UG

en-US

en-ZA

en-PH

es

es-MX

fa

fi-FI

fr

he

hu

in-ID

it

ja

ko

nb-NO

nl

pl

pt

pt-BR

ru

sk

sv

sv-SE

th

tr

uk

vi

zh-CN

zh-TW

Example for several columns: ```yaml transformations: - columns: ["first_name", "last_name"] params: type: person_generator column_templates: ["${first_name}", "${last_name}"] ``` Example for a single column: ```yaml transformations: - columns: ["full_name"] params: type: person_generator column_templates: ["${first_name} ${last_name}"] ``` ### Properties - `type = person_generator` - `column_templates`: array of `String.` For each column, the template to be used to generate personal data - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). - `locale`: optional `String.` To generate names from different geographical areas, the user can change this parameter. Default to 'en-GB', which corresponds to British names. - `length_exceeded_mode`: `[Value length exceeded mode](#ValueLengthExceededMode)`. - `column_lengths`: optional array of `Integer.` Max lengths allowable for the column. Ignored when length\_exceeded\_mode: "IGNORE"\` - `discriminator_digits`: `Integer.` Number of random digits appended to email templates to ensure uniqueness. Rule of thumb: Set it to at least 2 \* log10(N) - 6, where N is the number of unique emails to generate, but not less than 4. Default is 4. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `TEXT` Supports multiple columns: Yes ## Address generator `[Transformations](#TransformationParams)` Generate address fields (_e.g.,_ street, zip code) and keep them consistent across columns. Available templates are: - `${zip_code}` - `${country}` - `${city}` - `${street_name}` - `${house_number}` - `${flat_number}` - `${full_address}` - `${street_address}` - `${region}` - `${latitude}` - `${longitude}` - `${coordinates}` - `${time_zone}` Supported locales:

ar

bg

ca

ca-CAT

cs

da-DK

de

de-AT

de-CH

en

en-AU

en-CA

en-GB

en-IN

en-MS

en-NEP

en-NG

en-NZ

en-PAK

en-SG

en-UG

en-US

en-ZA

en-PH

es

es-MX

fa

fi-FI

fr

he

hu

in-ID

it

ja

ko

nb-NO

nl

pl

pt

pt-BR

ru

sk

sv

sv-SE

th

tr

uk

vi

zh-CN

zh-TW

Example for several columns: ```yaml transformations: - columns: [ "street_name", "zip_code" ] params: type: address_generator column_templates: [ "${street_name}", "${zip_code}" ] ``` Example for a single column: ```yaml transformations: - columns: ["address"] params: type: address_generator column_templates: ["${country}, ${city}, ${street_name}, ${house_number}, ${flat_number}, ${zip_code}"] ``` ### Properties - `type = address_generator` - `column_templates`: array of `String.` For each column, the template to be used to generate address data - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same street. The "self" value means consistency with the source value (masking only). - `locale`: optional `String.` To generate addresses from different geographical areas, the user can change this parameter. Default to 'en-GB', which corresponds to Great Britain addresses. - `length_exceeded_mode`: `[Value length exceeded mode](#ValueLengthExceededMode)`. - `column_lengths`: optional array of `Integer.` Max lengths allowable for the column. Ignored when length\_exceeded\_mode: "IGNORE"\` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `TEXT` Supports multiple columns: Yes ## Finance generator `[Transformations](#TransformationParams)` Generate financial data. Available templates: - `${credit_card}` - `${bic}` - `${iban}` - `${nasdaq_ticker}` - `${nyse_ticker}` - `${stock_market}` - `${us_routing_number}` The template `credit_card` (without the card type qualification) will result in a random type being picked. The credit card type can be configured using the following templates: - `${credit_card.visa}` - `${credit_card.mastercard}` - `${credit_card.discover}` - `${credit_card.american_express}` - `${credit_card.diners_club}` - `${credit_card.jcb}` - `${credit_card.switch}` - `${credit_card.solo}` - `${credit_card.dankort}` - `${credit_card.forbrugsforeningen}` - `${credit_card.laser}` Example: ```yaml transformations: - columns: [ "credit_card" ] params: type: finance_generator column_templates: [ "${credit_card.visa}" ] ``` ### Properties - `type = finance_generator` - `column_templates`: array of `String.` For each column, the template to be used to generate financial data - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. The "self" value means consistency with the source value (masking only). - `length_exceeded_mode`: `[Value length exceeded mode](#ValueLengthExceededMode)`. - `column_lengths`: optional array of `Integer.` Max lengths allowable for the column. Ignored when length\_exceeded\_mode: "IGNORE"\` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `TEXT` Supports multiple columns: Yes ## Redaction masker `[Transformations](#TransformationParams)` Some values in the input string are substituted by the same value, obtaining partially masked text in the output. Example: ```yaml transformations: - columns: ["credit_card"] params: type: redaction action: MASK which: FIRST count: 4 mask_with: "#" ``` ### Properties - `type = redaction` - `action`: `[Action](#Action)`. - `which`: `[Position](#Position)`. - `count`: `Integer.` amount of characters to be masked or kept, default to 4 - `mask_with`: `String.` character used to mask values, default to `*` Compatible modes: **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Unique hashing `[Transformations](#TransformationParams)` Apply a hash transformation to a given value so that the output is encrypted but structural coherence is preserved (same input hashed with same key is always going to produce same output). Output values are unique. This transformation is applied to primary and foreign keys by default in `MASKING` mode. Example: ```yaml transformations: - columns: ["card_id"] params: type: unique_hashing ``` ### Properties - `type = unique_hashing` - `max_value`: `Number (double).` Max value to generate, null means absence of limit - `precision`: `Integer.` Max precision to generate (e.g. if the value is 3, the maximal value is 999), null means absence of limit. Minimal value is applied if both max\_value and precision are specified Compatible modes: **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## Date unique hashing `[Transformations](#TransformationParams)` Apply a hash transformation to a date time format value so that the output is encrypted but structural coherence is preserved (same input hashed with same key is always going to produce same output). Output values are unique. This transformation is applied to primary and foreign keys by default in `MASKING` mode. Example: ```yaml transformations: - columns: ["create_date"] params: type: date_time_unique_hashing min: 2000-01-01T12:00:00Z max: 2022-01-01T12:00:00Z ``` ### Properties - `type = date_time_unique_hashing` - `min`: optional `String (date-time).` Minimum value - `max`: optional `String (date-time).` Maximum value Compatible modes: **MASKING** Compatible column data types: `DATE` Supports multiple columns: No ## Enum unique hashing `[Transformations](#TransformationParams)` Apply a hash transformation to a enum column value so that the output is encrypted but structural coherence is preserved (same input hashed with same key is always going to produce same output). Output values are unique. Example: ```yaml transformations: - columns: ["rating"] params: type: enum_unique_hashing ``` ### Properties - `type = enum_unique_hashing` - `values`: optional array of optional `String.` List of possible enum values - `is_enum_set`: `Boolean.` Flag indicating that the target column is enum set. In this case, the input and the output are expected to be a comma separated lists of enum values. Compatible modes: **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Date generator `[Transformations](#TransformationParams)` Output data is sampled from a parameterized continuous distribution, and transformed into dates. If parameters are not given, they will be extracted from the original data Example: ```yaml transformations: - columns: - "date_of_birth" params: type: date_generator mean: 2018-02-01T12:00:00Z std: 2d 4h 45m 12s 434ms min: 2000-01-01T12:00:00Z max: 2022-01-01T12:00:00Z ``` ### Properties - `type = date_generator` - `mean`: optional `String (date-time).` Average date of the sampled distribution - `std`: optional `String.` Standard deviation. The following formats are accepted: - ISO-8601 Duration format, _e.g.,_ `P1DT2H3M4.058S`. - The concise format described [here](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.time/-duration/to-string.html), _e.g.,_ `10s`, `1h 30m` or `-(1h 30m)` - Milliseconds without the specific unit, _e.g.,_ `12534`. - `min`: optional `String (date-time).` Minimum value. Must be strictly smaller than `max`. - `max`: optional `String (date-time).` Maximum value. Must be strictly larger than `min`. - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `DATE` Supports multiple columns: No ## UUID generator `[Transformations](#TransformationParams)` The output column is filled with UUIDs. Example: ```yaml transformations: - columns: ["unique_id"] params: type: uuid_generator ``` ### Properties - `type = uuid_generator` - `consistent_with_column`: optional `String.` If given, the column that need to be consistent on during both masking and generation. For example, if `consistent_with_column="user_id"` all people with same `user_id` will have the same name. The "self" value means consistency with the source value (masking only). Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## Constant numeric generator `[Transformations](#TransformationParams)` Generates a single numeric value for the entire column Example: ```yaml transformations: - columns: [ "balance" ] params: type: constant_numeric value: 0.0 ``` Example (range): ```yaml transformations: - columns: [ "balance" ] params: type: constant_numeric min: 0.0 max: 10000.0 ``` ### Properties - `type = constant_numeric` - `value`: optional `Number.` numeric value to generate - `min`: optional `Number.` The lower boundary for the value (inclusive) - `max`: optional `Number.` The upper boundary for the value (exclusive) - `numeric_type`: `[Numeric type](#NumericType)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC` Supports multiple columns: No ## Constant string generator `[Transformations](#TransformationParams)` Generates a single string value for the entire column Example: ```yaml transformations: - columns: [ "status" ] params: type: constant_string value: "ACTIVE" ``` ### Properties - `type = constant_string` - `value`: optional `String.` string value to generate Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Default value generator `[Transformations](#TransformationParams)` Generates a single default value for the entire column based on its type. It will generate default values as follows: \+ **`JSON`, `JSONB`: an empty document like `{}`** `XML`: an empty root document like `` ### Properties - `type = default_value` - `default_value`: optional `String.` Default value Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## Constant date generator `[Transformations](#TransformationParams)` Generates a single date value for the entire column Example: ```yaml transformations: - columns: [ "creation_date" ] params: type: constant_date value: 2022-07-28T12:21:00Z ``` Example (range): ```yaml transformations: - columns: [ "creation_date" ] params: type: constant_date min: 2022-07-01T00:00:00Z max: 2022-07-31T23:59:59Z ``` ### Properties - `type = constant_date` - `value`: optional `String (date-time).` date value to generate - `min`: optional `String (date-time).` The lower boundary for the value (inclusive) - `max`: optional `String (date-time).` The upper boundary for the value (exclusive) Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `DATE` Supports multiple columns: No ## Constant boolean generator `[Transformations](#TransformationParams)` Generates a single boolean value for the entire column Example: ```yaml transformations: - columns: [ "is_active" ] params: type: constant_boolean value: true ``` ### Properties - `type = constant_boolean` - `value`: optional `Boolean.` boolean value to generate Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `BOOLEAN` Supports multiple columns: No ## Loop generator `[Transformations](#TransformationParams)` Generates a sequence of elements in a loop that can be repeated. Data can be obtained from various sources or generated based on existing data. The following example demonstrates how to accomplish this: ```yaml transformations: - columns: - transaction_type params: type: "loop_generator" repeatable: true source: value_source: "PROVIDED" values: - "sent" - "skipping_value" - "received" - null ``` This transformation can be applied to several columns at once. This illustrates how the generator can handle multiple columns with provided tuples of values that always appear together: ```yaml transformations: - columns: - productcode - productname params: type: "loop_generator" repeatable: true source: value_source: "MULTIPLE_PROVIDED" values: - productcode: "P1" productname: "Product 1" - productcode: "P2" productname: "Product 2" - productcode: "P3" productname: "Product 3" - productcode: null productname: null ``` You can also specify a CSV file as the source of elements. The following example illustrates the complete configuration example: ```yaml transformations: - columns: - productcode - productname params: type: "loop_generator" repeatable: true source: value_source: "CSV_FILE" path: src/e2e/resources/data_with_header_multi.csv null_values: null format: encoding: "UTF-8" delimiter: "," trim: true columns: column_accessor_type: NAME names: ["code", "name"] ``` ### Properties - `type = loop_generator` - `repeatable`: `Boolean.` The list will repeat itself in a loop if necessary. Otherwise, an exception will be thrown when the record number exceeds the size of the list. - `source`: `[Source of elements](#ElementSource)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Constant XML generator `[Transformations](#TransformationParams)` Generates a single XML value for the entire column. Example: ```yaml transformations: - columns: [ "xml_column" ] params: type: constant_xml value: "test" ``` ### Properties - `type = constant_xml` - `value`: `String.` XML string value to generate Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## JSON pointer transformer `[Transformations](#TransformationParams)` Transforms JSON value nodes indicated by JSON pointers. The rest of the values are kept as is. Example: ```yaml transformations: - columns: ["productspec"] params: type: "json_pointer_transformer" specifications: - pointers: [ "/sku" ] transformation: type: "format_preserving_hashing" - pointers: [ "/tags/0" ] transformation: type: "format_preserving_hashing" ignore_errors: true ``` ### Properties - `type = json_pointer_transformer` - `specifications`: array of `[JSON Pointer Transformer Specification](#JsonPointersSpecification)`. - `locale`: optional `String.` To generate data from different geographical areas. Default to 'en-GB'. - `auto_detect`: `Boolean.` When a JsonPointer is assigned to a particular key, it is automatically constructed based on the type of the column. This works for simple data types such as strings, numbers, and Boolean values. However, this method does not guarantee data consistency and may reduce the length of strings. If column or key consistency is required, the pointers must be set manually. Compatible modes: **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## Xml XPath transformer `[Transformations](#TransformationParams)` Transforms XML value nodes indicated by XML XPath. The rest of the values are kept as is. Example: ```yaml transformations: - columns: ["productspec"] params: type: "xpath_transformer" specifications: - queries: [ "/sku" ] transformation: type: "format_preserving_hashing" ``` ### Properties - `type = xpath_transformer` - `specifications`: array of `[Xml XPath Transformer Specification](#XPathSpecification)`. - `encoding`: `String.` This property enables to specify an encoding format. - `omit_xml_declaration`: `Boolean.` When true, the re-serialized XML omits the declaration. Use for XML values stored without a declaration, so masking does not introduce one. Compatible modes: **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## JSON multi-value transformer `[Transformations](#TransformationParams)` Masks a T24 multi-value column stored in the JSON format. The column holds either a single bare value or a JSON array (a nested array carries sub-values); each non-empty string leaf is masked independently with the inner transformation and the structure is rebuilt. Nulls and empty leaves are preserved. Use this for multi-value PII (names, addresses, phones, emails) on the `json` format. Example: ```yaml transformations: - columns: ["NAME.1"] params: type: "json_multi_value_transformer" transformation: type: "person_generator" column_templates: ["${full_name}"] mask_sub_values: true ``` ### Properties - `type = json_multi_value_transformer` - `transformation`: `[Transformations](#TransformationParams)`. - `mask_sub_values`: `Boolean.` When true (default), each sub-value of a nested array is also masked. Set to false to mask only top-level array elements and leave nested arrays untouched. Compatible modes: **MASKING** Compatible column data types: `ANY` Supports multiple columns: No ## T24 id remap `[Transformations](#TransformationParams)` Deterministically remaps a T24 RECID (or a reference to one) to a new id for the T24 reseed feature. The body of the id is produced by format-preserving encryption, which is value-keyed (same input always yields the same output, so a reference column stays consistent with its remapped primary key) and injective (distinct inputs yield distinct outputs, so distinct source ids remain distinct after remapping). A constant reserved `prefix` is prepended so that minted ids sit in a namespace that is disjoint from the real ids they are appended beside. With a numeric prefix, a numeric input stays numeric. Null or blank values pass through unchanged. Example: ```yaml transformations: - columns: ["RECID"] params: type: "t24_id_remap" prefix: "9" ``` ### Properties - `type = t24_id_remap` - `prefix`: `String.` Reserved leading marker prepended to every minted id so the output lands in a namespace disjoint from the real ids it is appended beside (read-side complement filter `RECID NOT LIKE '%'`). Required and non-blank UNLESS preserve\_structure or preserve\_prefix is set. Suits free-numeric ids (e.g. CUSTOMER); unsafe for ids whose real values span the prefix (e.g. T24 account numbers, which start with any digit or currency code). - `preserve_structure`: `Boolean.` When true, non-digit characters are preserved verbatim and each maximal run of digits is remapped in place (same width, value-keyed, injective) — so a currency-coded account like `USD160050001` stays `USD`. No prefix is added; disjointness and re-run filtering for these ids come from the reseed pipeline’s synthetic-id side table. - `preserve_prefix`: `String.` Literal application prefix kept verbatim (for ids that start with it) while the rest of the id is remapped in place at the same width (value-keyed, injective); an id that does not start with the prefix has its whole value remapped. For T24 Arrangement Architecture loan ids like `AA`, set `AA` so the marker stays and only the date/sequence body is remapped; the body keeps its character classes (digits stay digits, letters stay letters) so it remains a valid base-36 arrangement id. No reserved namespace prefix is added; disjointness and re-run filtering come from the reseed pipeline’s synthetic-id side table. Compatible modes: **MASKING** Compatible column data types: `TEXT` Supports multiple columns: No ## Void generator `[Transformations](#TransformationParams)` An auxiliary transformer that throws an error when called. It is used only when it is necessary to ignore the processing of the entire table. ### Properties - `type = void_generator` Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Scripting transformer `[Transformations](#TransformationParams)` The scripting transformer allows you to implement own logic for both the `GENERATION` mode and the `MASKING` mode. Currently only Javascript implementation is supported. The script for GENERATION mode must define a lambda function that returns a dictionary where keys are column names, and the values are the desired values of the record. If a transformer is applied to a single column, the value may be returned instead of a dictionary. The script for the MASKING mode must define a lambda function with the two arguments `ctx` and `originalRecord`. The following example shows how to use a custom script in MASKING mode. ```yaml transformations: - columns: - title params: type: scripting_transformer language: "JAVASCRIPT" script: code: | (ctx, originalRecord) => { const title = originalRecord.get("title"); return title.split(' ').reverse().join(' '); } ``` The script for the GENERATION mode should define a lambda function with the single argument `ctx`. The following example shows how to use a custom script for GENERATION: ```yaml transformations: - columns: - credit_card params: type: scripting_transformer language: "JAVASCRIPT" init_script: code: | /** * @returns {String} */ function generateRandomCreditCardNumber() { let creditCardNumber = 4 for (let i = 1; i < 16; i++) { const digit = Math.floor(Math.random() * 10); creditCardNumber += digit.toString(); } return creditCardNumber; } script: code: | /** * @typedef { Object. | * } Result * * @param {GenerationContext} ctx * @returns {Result} */ (ctx) => generateRandomCreditCardNumber(); ``` ### Properties - `type = scripting_transformer` - `language`: `[Scripting Language](#ScriptingLanguage)`. - `script`: `[Script](#Script)`. - `init_script`: `[Script](#Script)`. - `additional_properties`: `[ScriptingAdditionalProperties](#ScriptingAdditionalProperties)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `ANY` Supports multiple columns: Yes ## Arithmetic generator `[Transformations](#TransformationParams)` The output column is filled with values that are calculated from other columns based on the given expression. A basic use case would be converting one unit to another, _e.g.,_ converting Celsius to Fahrenheit: ```yaml transformations: - columns: ["temperature_fahrenheit"] params: type: arithmetic_generator expression: "temperature_celsius * 9 / 5 + 32" ``` The generator can also include more complex expressions that incorporate other generators. Additionally, it supports basic date arithmetic based on epoch milliseconds: ```yaml transformations: - columns: ["card_expiry_date"] params: type: arithmetic_generator expression: "card_issue_date + card_validity_period + 1d" variables: "card_validity_period": type: categorical_generator categories: value_source: PROVIDED values: "730d": 0.5 "1461d": 0.5 ``` Expressions in the `arithmetic_generator` can refer to columns within the current table, or columns in parent tables. To refer to a column in the current table, supply the column name. To refer to another table, use the syntax `schema_name.table_name.column_name`. ```yaml table_name_with_schema: public.transaction transformations: - columns: [ "creation_date" ] params: type: arithmetic_generator variables: "days_between_card_issue_and_transaction": type: continuous_generator max: 365 mean: 180 std: 100 min: 1 numeric_type: INT expression: public.card.card_issue_date + 1d * days_between_card_issue_and_transaction ``` The current table must have a foreign key relationship with the parent table and a `foreign_key_generator` must be configured correctly. > **NOTE** > If the foreign key required by the generator is not present in your database schema, you can define a virtual foreign key as shown [here](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/virtual_foreign_keys#UserConfig_metadata) In addition, expressions can reference the original (input) value of a column from the current table by using the `ORIGINAL` prefix. To refer to the original value of a column, use the syntax `ORIGINAL.column_name`. > **NOTE** > The `ORIGINAL` prefix is not supported in **GENERATION** mode. ### Properties - `type = arithmetic_generator` - `variables`: `[List of named variables](#Variables)`. - `constants`: `[List of named constants](#Constants)`. - `is_transforming_dates`: optional `Boolean.` Indicates if this transformation will perform arithmetic on one or more dates. If true, the generator will be able to operate on time units (`ms`, `s`, `m`, `h`, `d`). If the output column type is a date, this will be on by default. - `expression`: `String.` The expression to be evaluated. It can contain arithmetic combinations of `variables`, `constants`, and (numeric) table columns, as well as numbers. Scientific notation is supported, _e.g.,_ `1.23e-4`. - `references`: optional array of `[Referred foreign key](#ExternalColumnDependency)`. A list of references explaining which parent row should be used in `expression`. This is only necessary in situations where the table has multiple foreign keys pointing to the same parent table. For example, a table might include both `employee_id` and `manager_id` referencing a parent table `Employees.employee_id`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC`, `DATE` Supports multiple columns: No ## Sieve generator `[Transformations](#TransformationParams)` The output column is filled with values that are sampled from a given distribution and rejected if they do not satisfy a given condition. The process is repeated until a valid value is found. Example: ```yaml transformations: - columns: ["start"] params: type: date_generator - columns: ["finish"] params: type: sieve_generator condition: "finish > start + 1h" generator: type: date_generator ``` Conditions in the `sieve_generator` can refer to columns within the current table, or columns in parent tables. To refer to a column in the current table, supply the column name. To refer to another table, use the syntax `schema_name.table_name.column_name`. ```yaml table_name_with_schema: public.transaction transformations: - columns: ["creation_date"] params: type: sieve_generator condition: creation_date > public.card.card_issue_date && creation_date < public.card.card_expiry_date generator: type: date_generator ``` The current table must have a foreign key relationship with the parent table and a `foreign_key_generator` must be configured correctly. > **NOTE** > If the foreign key required by the generator is not present in your database schema, you can define a virtual foreign key as shown [here](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/virtual_foreign_keys#UserConfig_metadata) > **NOTE** > If the provided condition is too narrow the generator may struggle to find a valid value. In such cases, it is recommended to adjust the condition or use another generator like `arithmetic_generator`. ### Properties - `type = sieve_generator` - `generator`: `[Transformations](#TransformationParams)`. - `condition`: `String.` The condition that the generated value must satisfy. It should contain the current column name and can include other columns and numbers. Scientific notation is supported, _e.g.,_ `1.23e-4`. Complex conditions can be expressed using boolean algebra, _e.g._: `column1 > 0 && !(column2 < column1 || column3 >= column1)`. - `max_discard_percentage`: `Number (double).` If number of discarded values exceeds given percentage of expected number of records, the generator will terminate the transformation. By default, no more than 2/3 of the expected number of records can be discarded. - `is_condition_on_dates`: optional `Boolean.` Indicates if the condition is applied to date. If true, the generator will operate on time units (`ms`, `s`, `m`, `h`, `d`). If not set, the value will inferred from the output column type. - `references`: optional array of `[Referred foreign key](#ExternalColumnDependency)`. A list of references explaining which parent row should be used in `condition`. This is only necessary in situations where the table has multiple foreign keys pointing to the same parent table. For example, a table might include both `employee_id` and `manager_id` referencing a parent table `Employees.employee_id`. - `constants`: `[List of named constants](#Constants)`. Compatible modes: **GENERATION**, **MASKING** Compatible column data types: `NUMERIC`, `DATE` Supports multiple columns: Yes ## Categories source configuration Used in: `[categories](#CategoricalGeneratorParams_categories)` optional `Object.` Depending on `value_source` property value, can be one of the following:

PROVIDED

Config categories source

CSV_FILE

CSV file categories source

MULTIPLE_PROVIDED

Config categories source

MULTIPLE_CSV_FILE

CSV file categories source for multiple columns

## Numeric type Used in: `[numeric_type](#ContinuousGeneratorParams_numeric_type)`, `[numeric_type](#QuantileGeneratorParams_numeric_type)`, `[numeric_type](#ConstantNumericGeneratorParams_numeric_type)` optional `String.` Type of numbers used by a generator Enum values - `INT` - `LONG` - `DOUBLE` - `FLOAT` - `BIG_DECIMAL` - `BIG_INTEGER` - `SHORT` - `BYTE` - `UNSIGNED_BYTE` - `UNSIGNED_INTEGER` - `UNSIGNED_LONG` - `UNSIGNED_SHORT` ## Foreign key distribution Used in: `[distribution](#ForeignKeyGeneratorParams_distribution)` `String.` Distributions: `POISSON` (default) - Generates parent-child relations based on the Poisson distribution. Where lambda represents the ratio of parent to child count. `ROUND_ROBIN` - Assign children to parents using a round-robin algorithm. `ORIGINAL` - Preserves the original reference ratio (Note: This option may require more computational resources for recalculation of the model. It’s worth noting that this model may generate a distribution that does not perfectly match the original one). Enum values - `POISSON` - `ROUND_ROBIN` - `ORIGINAL` ## Column transformation parameters Used in: `[transformations](#UniqueGeneratorParams_transformations)` `Object.` List of column names associated with [Transformation parameters](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations). ### Properties - `columns`: array of `String.` List of columns that are affected by this generator. - `params`: `[Transformations](#TransformationParams)`. ## Hashing group Used in: `[groups](#FormatPreservingHashingParams_groups)` `Object.` The pair of `selector` and list of `alphabet`. `selector` is used to choose characters from the input string, `alphabet` - is a set of characters, which are used to replace source ones. ### Properties - `selector`: `[Hashing group selector](#FormatPreservingHashingGroupSelector)`. - `alphabets`: array of `[Alphabet](#Alphabet)`. ## Format preserving hashing filter Used in: `[filter](#FormatPreservingHashingParams_filter)` optional `Object.` Depending on `type` property value, can be one of the following:

first

First N characters

last

Last N characters

characters

Specified characters

substring

Specified substring

regex

Regex filter

## Alphabet Used in: `[alphabets](#StringSequenceGeneratorParams_alphabets)`, `[alphabets](#FormatPreservingHashingGroup_alphabets)` `Object.` Depending on `type` property value, can be one of the following:

digits

Digits 0-9

lower_letters

English letters in lowercase [a-z]

upper_letters

English letters in UPPERCASE [A-Z]

custom

Custom alphabet

## Value length exceeded mode Used in: `[length_exceeded_mode](#PersonGeneratorParams_length_exceeded_mode)`, `[length_exceeded_mode](#AddressGeneratorParams_length_exceeded_mode)`, `[length_exceeded_mode](#FinanceGeneratorParams_length_exceeded_mode)` optional `String.` Action, required on value length overflow. Modes: `IGNORE` - error if the value exceeds column length `TRUNCATE` (default) - truncate value to the field length Enum values - `IGNORE` - `TRUNCATE` ## Action Used in: `[action](#RedactionParams_action)` `String.` Enum values - `KEEP` - `MASK` ## Position Used in: `[which](#RedactionParams_which)` `String.` Enum values - `FIRST` - `LAST` ## Source of elements Used in: `[source](#LoopGeneratorParams_source)` optional `Object.` Depending on `value_source` property value, can be one of the following:

PROVIDED

Element’s source provided from a configuration

CSV_FILE

Element’s source provided from a CSV file

MULTIPLE_PROVIDED

Multi columns element’s source provided from a configuration

## JSON Pointer Transformer Specification Used in: `[specifications](#JsonPointerTransformerParams_specifications)` `Object.` ### Properties - `pointers`: array of `String.` JSON Pointer (specified by RFC6901) - `transformation`: `[Transformations](#TransformationParams)`. - `ignore_errors`: `Boolean.` Controls the behaviour when no JSON node is found at the pointer or the node has a type incompatible with the specified transformer. If this setting is `true`, the found JSON node, if any, will remain unchanged. If the setting is `false`, an error will be raised. Default is `false`. - `relative`: `Boolean.` The relative mode changes how JSON pointers work. Instead of always starting from the root, the pointer works from the leaf of a property. It doesn’t need to end at the root of the JSON. For example, the pointer `/user_id` will match multiple paths like: `/user_id`, `/account/user_id`, `/customers/0/user_id`. This means one pointer can match multiple locations, unlike the regular mode, which only allows a one-to-one match. ## Xml XPath Transformer Specification Used in: `[specifications](#XPathTransformerParams_specifications)` `Object.` ### Properties - `queries`: array of `String.` XPath (specified by [https://www.w3.org/TR/xpath-31/](https://www.w3.org/TR/xpath-31/)) - `transformation`: `[Transformations](#TransformationParams)`. - `ignore_errors`: `Boolean.` Controls the behaviour when no XML node is found at the xPath or the node has a type incompatible with the specified transformer. If this setting is `true`, the found XML node, if any, will remain unchanged. If the setting is `false`, an error will be raised. Default is `false`. ## Scripting Language Used in: `[language](#ScriptingTransformerParams_language)` `String.` Enum values - `JAVASCRIPT` ## Script Used in: `[script](#ScriptingTransformerParams_script)`, `[init_script](#ScriptingTransformerParams_init_script)` optional `Object.` The `script` should define a lambda function to be called on every record or row depending on the chosen method. The `init_script` is executed once on start. It can be helpful for defining variables and functions which are available to use in the main `script` ### Properties - `code`: optional `String.` Script code - `file`: optional `String.` Script file location. In the case of a local file system, the path can be absolute or relative to the application process’s working directory (not to be confused with [working directory](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_working_directory)) The script can be located on local file system, AWS S3 and Google Storage. To be able to load scripts from S3 the property `TDK_AWS_ENABLED==true` should be set. More details can be found [here](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_aws_enabled). The property `TDK_GCP_ENABLED==true` allows loading scripts from Google Storage. More details can be found [here](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/application_properties#tdk_gcp_enabled). ## ScriptingAdditionalProperties Used in: `[additional_properties](#ScriptingTransformerParams_additional_properties)` optional map of String keys to\`Object.\` Additional properties to be used in the `script` and `init_script`. The dictionary variable `additionalProperties` is available from scripts. ## List of named variables Used in: `[variables](#ArithmeticGeneratorParams_variables)` optional map of String keys to\`[Transformations](#TransformationParams)\`. ## List of named constants Used in: `[constants](#ArithmeticGeneratorParams_constants)`, `[constants](#SieveGeneratorParams_constants)` optional map of String keys to\`Number.\` ## Referred foreign key Used in: `[references](#ArithmeticGeneratorParams_references)`, `[references](#SieveGeneratorParams_references)` `Object.` ### Properties - `column_name`: `String.` Name of the column that is used in the `condition` or `expression`. - `referred_schema`: optional `String.` - `referred_table`: `String.` Name of the table that contains the column that is used in the `condition` or `expression`. - `reference_columns`: array of `String.` Column list of the foreign key that is used to match parent and child records. ## Config categories source `[Categories source configuration](#Categories)` Config source involves configuring categories and weights directly within the configuration. ### Properties - `value_source = PROVIDED` - `type`: `[Dictionary data type](#DictionaryType)`. - `values`: `[Categories dictionary](#ProvidedCategoriesValues)`. - `null_values`: optional array of `String.` The values that should be treated as `NULL` values. Default is `["null"]` ## CSV file categories source `[Categories source configuration](#Categories)` ### Properties - `value_source = CSV_FILE` - `type`: `[Dictionary data type](#DictionaryType)`. - `path`: `String.` The path to file on your file system. - `format`: `[Parser configuration](#CsvFileCategoriesFormat)`. - `null_values`: optional array of `String.` The values that should be treated as `NULL` values. Default is `["null"]` ## Config categories source `[Categories source configuration](#Categories)` Config source involves configuring categories and weights for multiple columns directly within the configuration. ### Properties - `value_source = MULTIPLE_PROVIDED` - `column_types`: `[Columns data types](#DictionaryTypes)`. - `null_values`: optional array of `String.` The values that should be treated as `NULL` values. Default is `["null"]` - `category_values`: array of `[Config categories source](#MultiProvidedCategory)`. ## CSV file categories source for multiple columns `[Categories source configuration](#Categories)` ### Properties - `value_source = MULTIPLE_CSV_FILE` - `column_types`: `[Columns data types](#DictionaryTypes)`. - `null_values`: optional array of `String.` The values that should be treated as `NULL` values. Default is `["null"]` - `path`: `String.` The path to file on your file system. - `format`: `[Parser configuration for multiple columns](#MultiCsvFileCategoriesFormat)`. ## Hashing group selector Used in: `[selector](#FormatPreservingHashingGroup_selector)` `Object.` Depending on `type` property value, can be one of the following:

digits

Digits 0-9

lower_letters

Lower-case alphabetic characters

upper_letters

Upper-case alphabetic characters

regex

Custom regex pattern

word_characters

Word characters (equivalent of regex '\w+' as described at this link)

## First N characters `[Format preserving hashing filter](#FormatPreservingHashingFilter)` Mask only first N characters of the input string ### Properties - `type = first` - `n`: `Integer (int32).` ## Last N characters `[Format preserving hashing filter](#FormatPreservingHashingFilter)` Mask only last N characters of the input string ### Properties - `type = last` - `n`: `Integer (int32).` ## Specified characters `[Format preserving hashing filter](#FormatPreservingHashingFilter)` Mask only specified characters of the input string ### Properties - `type = characters` - `characters`: `String.` - `ignore_case`: `Boolean.` ## Specified substring `[Format preserving hashing filter](#FormatPreservingHashingFilter)` Mask only specified substring of the input string ### Properties - `type = substring` - `substring`: `String.` - `ignore_case`: `Boolean.` ## Regex filter `[Format preserving hashing filter](#FormatPreservingHashingFilter)` Mask only characters filtered by regex ### Properties - `type = regex` - `pattern`: `String.` - `ignore_case`: `Boolean.` ## Digits 0-9 `[Alphabet](#Alphabet)` ### Properties - `type = digits` ## English letters in lowercase \[a-z\] `[Alphabet](#Alphabet)` ### Properties - `type = lower_letters` ## English letters in UPPERCASE \[A-Z\] `[Alphabet](#Alphabet)` ### Properties - `type = upper_letters` ## Custom alphabet `[Alphabet](#Alphabet)` Custom alphabet which can consist of characters, unicode blocks and unicode ranges. In total it can be from 1 to (2^16) characters. ### Properties - `type = custom` - `parts`: optional array of `[Custom alphabet part](#CustomAlphabetPart)`. ## Element’s source provided from a configuration `[Source of elements](#ElementSource)` ### Properties - `value_source = PROVIDED` - `column_type`: `[Dictionary data type](#DictionaryType)`. - `values`: array of optional `String.` The list of repeatable elements ## Element’s source provided from a CSV file `[Source of elements](#ElementSource)` ### Properties - `value_source = CSV_FILE` - `path`: `String.` The path to file on your file system. - `format`: `[Parser configuration](#CsvFileFormat)`. - `column_types`: `[Columns data types](#DictionaryTypes)`. - `null_values`: optional array of `String.` The values that should be treated as `NULL` values. Default is `["null"]` ## Multi columns element’s source provided from a configuration `[Source of elements](#ElementSource)` ### Properties - `value_source = MULTIPLE_PROVIDED` - `column_types`: `[Columns data types](#DictionaryTypes)`. - `values`: array of `[Elements dictionary for multiple columns](#MultiProvidedValues)`. ## Dictionary data type Used in: `[type](#ProvidedCategories_type)`, `[type](#CsvFileCategories_type)`, `[column_type](#ProvidedElementSource_column_type)` optional `String.` Data type of categories. Modes: `STRING` (default) - Interpret values as strings `BOOLEAN` - Interpret values as booleans `NUMERIC` - Interpret values as doubles Enum values - `STRING` - `BOOLEAN` - `NUMERIC` ## Categories dictionary Used in: `[values](#ProvidedCategories_values)` map of String keys to\`Number.\` The map can store keys in one of three formats: `STRING`, `BOOLEAN` or `NUMERIC`. The weights must be equal to or greater than zero. Example of formats: ```yaml ---- "23": 123 "owl": 534 "true": 532 ---- ``` To disable one of the categories, simply set its weight value to zero. ## Parser configuration Used in: `[format](#CsvFileCategories_format)` optional `Object.` ### Properties - `encoding`: `String.` This property enables to specify an encoding format. - `delimiter`: `String.` You have the option to specify a custom delimiter. - `trim`: `Boolean.` The trim option removes leading and trailing spaces from each cell. - `columns`: `[Column value extract method](#CsvFileCategoriesColumns)`. ## Columns data types Used in: `[column_types](#MultiProvidedCategories_column_types)`, `[column_types](#MultiCsvFileCategories_column_types)`, `[column_types](#CsvFileElementSource_column_types)`, `[column_types](#ProvidedMultiColumnsElementSource_column_types)` optional map of String keys to\`[Dictionary data type](#DictionaryType)\`. ## Config categories source Used in: `[category_values](#MultiProvidedCategories_category_values)` `Object.` The values for multiple columns of single category. ### Properties - `values`: `[Categories dictionary for multiple columns](#MultiProvidedCategoryValues)`. - `weight`: `Number.` ## Parser configuration for multiple columns Used in: `[format](#MultiCsvFileCategories_format)` optional `Object.` ### Properties - `encoding`: `String.` This property enables to specify an encoding format. - `delimiter`: `String.` You have the option to specify a custom delimiter. - `trim`: `Boolean.` The trim option removes leading and trailing spaces from each cell. - `columns`: `[Column value extract method for multiple columns](#MultiCsvFileCategoriesColumns)`. ## Digits 0-9 `[Hashing group selector](#FormatPreservingHashingGroupSelector)` ### Properties - `type = digits` ## Lower-case alphabetic characters `[Hashing group selector](#FormatPreservingHashingGroupSelector)` ### Properties - `type = lower_letters` ## Upper-case alphabetic characters `[Hashing group selector](#FormatPreservingHashingGroupSelector)` ### Properties - `type = upper_letters` ## Custom regex pattern `[Hashing group selector](#FormatPreservingHashingGroupSelector)` ### Properties - `type = regex` - `pattern`: `String.` ## Word characters (equivalent of regex '\\w+' as described at [this link](https://unicode.org/reports/tr18/#word)) `[Hashing group selector](#FormatPreservingHashingGroupSelector)` ### Properties - `type = word_characters` ## Custom alphabet part Used in: `[parts](#CustomAlphabet_parts)` optional `Object.` Depending on `type` property value, can be one of the following:

characters

Character set

unicode_block

Unicode block

unicode_range

Unicode range

## Parser configuration Used in: `[format](#CsvFileElementSource_format)` optional `Object.` ### Properties - `encoding`: `String.` This property enables to specify an encoding format. - `delimiter`: `String.` You have the option to specify a custom delimiter. - `trim`: `Boolean.` The trim option removes leading and trailing spaces from each cell. - `columns`: `[Column value extract method](#CsvFileColumns)`. ## Elements dictionary for multiple columns Used in: `[values](#ProvidedMultiColumnsElementSource_values)` map of String keys to\[.green\]optional `String.` ## Column value extract method Used in: `[columns](#CsvFileCategoriesFormat_columns)` optional `Object.` There are multiple ways to access categories and weights from a CSV file. By default, when the `columns` parameter is absent, the generator expects a file with more than one column, where the column with index `0` represents the category, and the column with index `1` represents the weight. In the default configuration, file headers are not accepted. However, if you need to work with specific columns, you can specify which columns will provide data using either the `NAME` or `INDEX` column parameter. Depending on `column_accessor_type` property value, can be one of the following:

NAME

String column name value extract

INDEX

Column index value extract

## Categories dictionary for multiple columns Used in: `[values](#MultiProvidedCategory_values)` map of String keys to\`String.\` ## Column value extract method for multiple columns Used in: `[columns](#MultiCsvFileCategoriesFormat_columns)` optional `Object.` There are multiple ways to access categories and weights from a CSV file. By default, when the `columns` parameter is absent, the generator expects a file with (table columns size + 1) columns, where the column with index `0` represents the first category, and the last column represents the weight. In the default configuration, file headers are not accepted. However, if you need to work with specific columns, you can specify which columns will provide data using either the `NAME` or `INDEX` column parameter. Depending on `column_accessor_type` property value, can be one of the following:

NAME

String column name value extract for multiple columns

INDEX

Column index value extract

## Character set `[Custom alphabet part](#CustomAlphabetPart)` Custom alphabet which can consist of 1 to (2^16) characters. All printable characters from Unicode Basic Multilingual Plane are supported. ### Properties - `type = characters` - `characters`: `String.` ## Unicode block `[Custom alphabet part](#CustomAlphabetPart)` Unicode block by name. Name of the Unicode block formatted according to the results described in [Java’s UnicodeBlock documentation](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Character.UnicodeBlock.html). Examples: "BASIC\_LATIN", "Basic Latin". Only the blocks from BMP (codepoints from 0x0000 to 0xFFFF) are supported. You can refer to the [Unicode specification](https://www.unicode.org/Public/UNIDATA/Blocks.txt) to find out the range for a block of interest. ### Properties - `type = unicode_block` - `name`: `String.` ## Unicode range `[Custom alphabet part](#CustomAlphabetPart)` Unicode range of characters specified by the first and last character range int codes. You can use ranges from `0x0000` to `0xFFFF`. ### Properties - `type = unicode_range` - `from`: `Integer (int32).` - `to`: `Integer (int32).` ## Column value extract method Used in: `[columns](#CsvFileFormat_columns)` optional `Object.` There are several ways to access the categories and weights from a CSV file. By default, if the `columns` parameter is not specified, the generator will map the columns from the file to the transformation columns in order. In the default configuration, file headers are not accepted. However, if you need to work with specific columns, you can specify which columns will provide data using either the `NAME` or `INDEX` column parameter. Depending on `column_accessor_type` property value, can be one of the following:

NAME

String column name value extract

INDEX

Column index value extract

## String column name value extract `[Column value extract method](#CsvFileCategoriesColumns)` ### Properties - `column_accessor_type = NAME` - `categories`: `String.` - `weights`: `String.` ## Column index value extract `[Column value extract method](#CsvFileCategoriesColumns)` ### Properties - `column_accessor_type = INDEX` - `categories`: `Integer.` - `weights`: `Integer.` ## String column name value extract for multiple columns `[Column value extract method for multiple columns](#MultiCsvFileCategoriesColumns)` ### Properties - `column_accessor_type = NAME` - `categories`: `[Categories dictionary for multiple columns](#MultiCsvFileCategoriesColumnsStringMapping)`. - `weights`: `String.` ## Column index value extract `[Column value extract method for multiple columns](#MultiCsvFileCategoriesColumns)` ### Properties - `column_accessor_type = INDEX` - `categories`: `[Categories dictionary for multiple columns](#MultiCsvFileCategoriesColumnsNumericMapping)`. - `weights`: `Integer.` ## String column name value extract `[Column value extract method](#CsvFileColumns)` ### Properties - `column_accessor_type = NAME` - `names`: array of `String.` ## Column index value extract `[Column value extract method](#CsvFileColumns)` ### Properties - `column_accessor_type = INDEX` - `indexes`: array of `Integer.` ## Categories dictionary for multiple columns Used in: `[categories](#MultiCsvFileCategoriesColumnsString_categories)` map of String keys to\`String.\` ## Categories dictionary for multiple columns Used in: `[categories](#MultiCsvFileCategoriesColumnsNumeric_categories)` map of String keys to\`Integer.\`