Changelog

To be able to read the configuration file from AWS S3 you need to enable the TDK_AWS_ENABLED property, see Application properties.

Database credentials can be requested from AWS Secrets Manager:

Where:

By default, TDK preserves referential integrity based on the foreign keys in the source database schema. In this release, added the ability to provide additional foreign keys in the yaml configuration file.

For example, if order.user_id is a foreign key referred to user.id, and it’s not defined in the database schema, then the following configuration can be provided:

More details about the additional foreign keys in the Configuration reference section.

Significant performance improvement for the subsetting mode.

To enable the performance improvement, the following application property should be set:

In the RELAXED mode, constant generators constant_numeric, constant_string, constant_date, constant_boolean will be chosen by default where the source column contains the same value in all rows.

FAQ page in the documentation.

By default, STRICT mode is enabled. If no suitable transform is found, then the passthrough, null_generator and categorical_generator will not be chosen by default.

To keep the behavior of previous versions, you can use the RELAXED mode.

Data Filtering feature.

Multiple Database support.

YAML anchors and aliases support to reduce repeated sections in the configuration.

The latest free TDK CLI version is currently available in the documentation for download.

GitHub Actions Integration page for the free TDK CLI version in the documentation.

--dry-run, --default-config, --json-schema, --license-expiration options have been turned into separate commands:

dry-run command prints the effective configuration to the console (by default) or to the file (specified as -ec or --effective-config-file parameter value).

For example tdk.jar help dry-run:

Fixed the error Error getting connection from data source when running TDK transformation on a huge database.

Fixed the behaviour where ORA-00972: identifier is too long error appeared when working with Oracle DB instances.

Default rules now recognize boolean-typed fields (e. g. TINYINT(1) in MySql or BOOL in PostgreSQL) and use Categorical generator to generate random boolean values in GENERATION and MASKING modes.

The performance of creating effective config is now improved for certain cases.

Allows transforming columns as they are being read from the input database and written to the output database.

For example, a column might need a cast to a different type on read and then a cast back to the original type on write. The following configuration might be provided to address that:

The my_binary_column will be cast to char type on read and the result of transformation will be cast back to binary type on write.

TDK is now aware of unsigned numeric types supported by some DBMS (MySQL, Oracle).

format_preserving_hashing is now collision-free even for short strings.

The --dry-run CLI option now creates a YAML config file that contains transformations for all tables and can be used for further runs.

Introduced the ability to define the list of schemas to process:

schemas: array of String. If not set or null, all schemas available to the source database user will be processed.

Example:

json_pointer_transformer transforms JSON value nodes indicated by JSON pointers, the rest of the values are kept as is:

Refer to JSON Pointer Transformer for more details.

Introduced more empirical default rules for generation from empty database. If a source table is empty and the generated field is not null and no user configuration provided for this field, then reasonable defaults for generators are chosen for the following data types:

The following yaml configuration parameters have been renamed:

Example before:

Example after:

A validation step to check that the output schema is not the source schema to prevent any source data or schema corruption.

By combining Testcontainers with Synthesized TDK, developers can populate any Testcontainers database with synthetically generated data, enabling rapid development of tests for logic which involves interaction with the database. Refer to documentation for more details.

Documentation tutorials for different modes:

A transient local storage area can now be configured to speed up TDK operations. Refer to documentation for more details.

Introduced --json-schema parameter for CLI which prints JSON Schema for the YAML Configuration. This schema can be used in an IDE to provide auto-completion for your YAML and to validate your configuration before run.

Introduced the ability to define the output alphabets in format_preserving_hashing with unicode_block and unicode_range. Refer to custom alphabets for more details.

Introduced more empirical default rules for generation from empty database.

Added the minimum database permissions required to run Synthesized TDK with Oracle database, see Oracle permissions.

Preserve null values for all masking transformers.

Fixed the behaviour when CategoricalGenerator didn’t preserve the input probabilities in masking mode.

Fixed the behaviour when CategoricalGenerator produced 'null' strings instead of null values.

For tables wider than a certain threshold (for example, 21 columns for SQL Server) insertion failed. This issue is now fixed for all supported databases.

Significant performance improvement for MASKING and GENERATION modes for all supported databases.

To use GENERATION mode for empty tables, the user should specify target_row_number at the global or table level:

target_row_number: optional Integer (int64). The absolute size of the output table in rows. This parameter is applicable only for GENERATION mode. If not provided, target_ratio will be used.

More user-friendly messages in scenarios where there is not enough data to determine the generation parameters.

Improved error handling for Microsoft SQL server IDENTITY exceptions.

MySQL database can be used as an input and output database.

Improved performance of all transformations up to 30%.

Improved data quality and reduce memory consumption of date_generator.

Fixed the behavior when the second run with schema_creation_mode: CREATE_IF_NOT_EXISTS configuration fails for Microsoft SQL server.

Fixed IDENTITY property support for not PRIMARY KEY columns. Now int_sequence_generator is used by default for any IDENTITY column.

Oracle database can be used as an input and output database.

New unique_hashing algorithm:

Microsoft SQL Server IDENTITY property support.

The default MASKING behavior has become more secure. Now format_preserving_hashing is always used by default for string columns. To passthrough, the user must configure it explicitly.

conditional_generator now works correctly both when conditional_column is in the same table as the column being generated and when they are in different tables.

New format_preserving_hashing algorithm:

Database permissions page describing the minimum database permissions required to run Synthesized TDK.

A new parameter length_exceeded_mode is added for address_generator and person_generator, it allows to truncate the generated values by the column size.

For example, if the country column is varchar(20), and the generated value is "Saint Vincent And The Grenadines":

All the examples in the documentation are now covered with tests and up-to-date with the Synthesized TDK version.

Fixed an issue with temporary columns after failed execution, now it is cleaned up for most cases.

For more information, see BigID integration.

CREATE_IF_NOT_EXISTS schema creation mode support for Microsoft SQL Server.

Transformations and YAML configuration are now autogenerated and up-to-date.

Getting Started and Installation sections in the documentation with H2 demo database.

Columns with a single value do not fail with an error, but are kept as-is for MASKING mode and filled with nulls for GENERATION.

Subsetting is repeatable with the same global_seed value.

Huge performance improvement for address_generator and person_generator.

XML data type support for Microsoft SQL Server.

Fixed constant_date, now it works for both value and range modes.

Fixed categorical_generator validation, now it fails with not provided probabilities.

Microsoft SQL Server can be used as input and output database.

A hash transformation is applied to each character, which included into the configured group, in a given text so that the output preserves the format but contains different characters. This transformation is secure and non-reversible.

Parameters:

Available character selectors:

Available alphabets:

Available filters:

For more information, see Transformations.

Added new constant generators for numeric, date, string, boolean data types.

For more information, see Transformations.

categorical_generator supports numeric columns.

For more information, see Transformations.

Default value for insert_batch_size increased from 30 to 1000.

In addition to the nice appearance, many pages and yaml examples are now generated automatically from source code and tests, which reduces the number of mistakes and allows the documentation to be up-to-date with the product version.

Enjoy!

This release includes significant performance improvement for MASKING mode with target_ratio: 1.0.

Fixed issues with processing empty tables in MASKING and GENERATION modes.

Fixed issues with processing columns with single values in MASKING mode.

Fixed issue with missing relations in data generation, when GENERATION tables have Foreign Keys to KEEP tables.

To support custom database types:

For example, for the following custom ENUM type:

Use a configuration like this:

For more information, see Custom database types.

Generate a single numeric value for the entire column

Parameters:

Compatible modes: GENERATION,badge-primary MASKING,badge-secondary

Compatible column data types: NUMERIC

Supports multiple columns: No,badge-danger

Example:

For more information, see Transformations.

UUID data type support for MASKING mode.

BIGINT and SMALLINT data type support for GENERATION, MASKING, and KEEP modes.

global_seed to set the seed for random number generators.

An integer 32-bit value between -2147483648 and 2147483647, used a seed for random number generators. The result of generation must be the same each time the generation is being run with the same seed and workflow configuration. By default global_seed is 0.

Example:

For more information, see YAML configuration.

This release includes significant rework of transformation execution internals, bringing the following benefits to end users:

H2 database can be used as input and output database.

SQLite database can be used as input and output database.

The license expiration can be requested via API:

Where:

If the service is up and running correctly, you should receive a 200 status with the body containing information like:

UUID data type support for GENERATION and KEEP modes.

BOOLEAN data type support for GENERATION, MASKING, and KEEP modes.

YAML configuration can be uploaded as a file via API.

The database credentials can be provided from Google Secret Manager:

A new table_truncation_mode:

It allows not to delete existing data from the output database, but to generate additional and append above.

For more information, see YAML configuration.

Supported locales:

For more information, see Transformations.

For currently unsupported types, such as XML datatype, null_generator will be used by default.

Added ways to stop the workflow using workflow_id and workflow_run_id. Improved error handling.

Removed comparison between input and output schema. It allows to process a subset of the input tables.

formatted_string_generator in MASKING mode generates consistent values across the schema.

If the input numeric column contains only positive values, then the generated values will also be positive by default.

There are two table truncation modes:

Usage example for table_truncation_mode:

MASKING mode for tables with CHAR primary keys can be used without any additional configuration. In the previous versions passthrough transformation was used as a workaround.

Composite primary and foreign keys can be automatically handled without any additional configuration. In the previous versions foreign_key_generator was used as a workaround.

Advanced subsetting implementation for KEEP and MASKING modes. In the previous versions some of the tables after subsetting were empty.

Changed CLI parameters from camelCase to kebab-case:

person_generator and address_generator in MASKING mode will generate consistent values across the schema.

For example, all mentions of James Bond with UK address will be masked as Jon Snow with Seven Kingdoms address for any mentions in the schema.

There are four schema creation modes:

Note: If CREATE_IF_NOT_EXISTS, DO_NOT_CREATE modes are used, the target schema should be equal to the source one.

Generate address fields (e.g. street, zip code) and keep them consistent across columns.

Parameters:

Available templates are:

Compatible modes: GENERATION,badge-primary MASKING,badge-secondary KEEP,badge-warning

Compatible column data types: STRING

Supports multiple columns: Yes,badge-success

Example for multiple columns:

Example for a single column:

There are two cycle resolution strategies:

Example for FAIL mode:

Where the employees table contains a cycle reference.

Example for DELETE_NOT_REQUIRED mode:

We have been working hard to combine our products into a single product with enhanced architecture that will enable us to add exciting new features and optimizations!