Changelog

The categorical_generator now includes support for multiple columns, enabling users to specify multi-column categories. This enhancement facilitates the preservation of relationships between values across different columns within a single category.

For detailed implementation, refer to the following example:

You can also specify a CSV file as the source of categories for multiple columns:

The example above works with the following CSV file data:

Example of advanced configuration with multiple columns and multiple categories:

The example above works with the following CSV file data:

Alternatively, you can leave the categorical_generator configuration empty, in which case TDK will automatically generate categories from the source table.

The nullable_weight parameter in the categorical_generator has been replaced with null_values. This change allows users to specify the number of values to be treated as special NULL value. By default, null_values is set to ["null"].

Fixed the freeze when processing too many tables concurrently.

In certain scenarios, a foreign_key_generator with default distribution could generate an empty output, leading to error occurrences.

Support has been added to set the transformation mode parameter (such as MASKING, GENERATION, KEEP) at the column level, in addition to the global and table levels. This feature allows us to handle a common scenario - setting the KEEP mode as a global mode and specifying which columns to mask. In this case, TDK automatically determines the necessary transformation for each column (such as format_preserving_hashing, noising, and so on). More information can be found in this example:

Ability to specify the mapping value without providing any transformation parameters (the params parameter):

Enhanced the schema copy support for MSSQL. Now, it can handle the following data types: timestamp, numeric, decimal, money, smalldatetime, datetime, and sysname.

Added support of cyclically referenced table truncation for: DB2, H2, MSSQL and ORACLE databases.

Resolved an issue where schema copying for SQL Server would attempt to access other databases, leading to permission access errors.

Significantly enhance the performance of the TDK in MASKING mode when the working directory enabled. This enhancement particularly benefits large tables by accelerating the rows filtering process.

Corrects a bug where certain rows were erroneously filtered out in MASKING mode. This issue primarily affected composite keys containing Null values. With this fix, Null values are now properly handled, ensuring the retention of all rows during the filtering process.

Enhanced DROP_AND_CREATE mode to include support for dropping schema-related objects such as stored functions and procedures, domains, and triggers within the DROP_AND_CREATE mode.

This advanced performance tuning property, the In-Memory Filter Threshold, is designed to optimize filtering operations.

This property determines whether values the parent table are loaded into memory during filtering operations, depending on the table size. If the size of the table does not exceed this specified value, the values will be loaded into memory for filtering for child tables.

The property can be set globally and be overridden at the table-level configuration.

Certain datatypes previously had a different handling method that resulted in exceptions for some columns with names containing spaces, such as [Column Name with Spaces].

This bugfix addresses an issue encountered when copying tables that include columns with varbinary(max) data type in SQL Server.

Introducing the new Constant XML Generator.

Implemented a fix for work with XML fields on SQL Server for runs with working directory.

Implemented a fix to enhance the stability of schema transfer functionality by disabling trigger copying for Postgres.

Addressed a bug in the data_generator where automatic parameter calculation led to an arithmetic overflow error for DB2.

The view copying functionality was not fully implemented, leading to inconveniences during schema copying. To maintain consistency with previous versions, view copying during schema transfer has been disabled.

This release introduces two new configuration flags, providing users with greater control over batch handling:

This release includes a fix to mitigate the rare bug that could cause interference from previous runs affecting the current execution.

This strategy optimally performs bulk loading for SQL Server, significantly enhancing the efficiency of writing large files into tables.

This enhancement boosts performance by reducing processing time in schemas with numerous search indexes.

Enhanced support for category names in various cases and encodings.

Enhanced support for generating data from an empty table with alias types.

Implemented support for an ignore filter in table processing and introduced a caching strategy for improved performance in table size calculation.

The timestamp (Transact-SQL) data type is just an incrementing number and does not preserve a date or a time.

Introducing the new Finance Generator.

Choose from a variety of templates, including:

Configure specific card types with templates such as:

Example:

TDK now intelligently applies heuristics based on column and table names, allowing for more suitable default transformation selections.

Therefore, Person Generator, Address Generator and Finance Generator can be chosen by default for GENERATION mode.

This behavior is enabled by default but can be disabled using the use_text_column_heuristics property.

The foreign key generator now enables the generation of foreign key relationships with links that follow a Poisson distribution, offering a more accurate representation of real-world data structures.

This feature is now set as the default behavior, replacing the previous default of ROUND_ROBIN. However, ROUND_ROBIN remains available for use if preferred.

New templates are now supported for Person Generator:

New templates for Address Generator:

Make length_exceeded_mode: TRUNCATE default for person_generator, address_generator and finance_generator.

This ensures generated values are truncated by the column’s max length instead of causing overflow errors.

The Default Locale property can be set globally for person_generator, address_generator and finance_generator.

This setting can be overridden at the table-level configuration.

The expanded list of supported locales for person_generator, address_generator and finance_generator.:

Enhanced support for user-defined data type aliases, allowing these types to be processed by the TDK.

JSON Pointer transformer now can process PostgreSQL JSONB type.

TDK requires at least JVM 17 to be run.

It’s needed to update JDK to be able using latest versions of TDK.

No changes required for Docker and Kubernetes users.

Despite text and ntext are deprecated, they may be present in production schemas.

Now these types can be processed by the TDK.

With the introduction of the new Scripting Transformer, it’s allowed to write custom scripts using the Javascript programming language.

This makes it possible to extend the TDK and add any specific logic.

The Scripting Transformer can be applied to any column for the GENERATION and MASKING modes.

The foreign key generator now allows you to preserve the original relationship of links, which can significantly improve data quality.

This feature can be enabled by specifying the distribution ORIGINAL for the foreign key columns:

New and more convenient format for configuring categorical_generator, including the ability to specify a CSV file as the source of categories.

Fix issues in SQLite and PostgreSQL support.

Optimize Oracle batch inserts.

Snowflake support using a JDBC-compatible driver.

Increased efficiency in foreign keys metadata collecting.

The TDK is now available on the GCP Marketplace. Getting started page https://console.cloud.google.com/marketplace/product/synthesized-marketplace-public/synthesized-tdk

All transformations now use Java-agnostic hashes; there is no more Objects.hash() or similar unportable calls.

Fixed a number of corner cases where unique and referential integrity constraints were processed incorrectly.

Added support for Hashicorp Vault secret manager.

Before this release, for GENERATION mode for date generator we had to set std option in milliseconds.

Now the following formats are supported:

Starting from release v1.17, the UUID transformer worked as a pure generator, it never took into account input values. Therefore it had no utility when processing FK-connected columns. In this release, the correct behaviour for the masking mode was restored.

New DEFER_FOREIGN_KEY Cycle Resolution Strategy: when selected - all FK references will be preserved, but the ones that lead to cycles will be disabled during masking and then re-enabled after data is inserted.

This strategy is suitable for databases with cyclic schema and works only with MASKING mode without subsetting.

Extend TDK CLI Helm chart with the property ttlSecondsAfterFinished. This allows Kubernetes pods to be removed in specified number of seconds after they have completed.

Copy DDL for views was previously disabled for MySQL.

The process of creating an efficient configuration is no longer aborted after the first detected error.

Previously, these columns were copied with the other data type.

New tutorials for Masking, Generation, Subsetting and Data Filtering.

For more information, refer to java.sql.SQLException: Zero date value prohibited page.

Starting from this version, the Subsetting mode is available in the free version of TDK. For more details about the Subsetting mode, please see here.

TDK can operate with Postgres partitioned tables.

The TDK is now available on the AWS Marketplace with Docker, ECS Fargate, and Helm charts delivery options for . See more details here: AWS Marketplace.

TDK can now be launched not only via command line interface but as a Docker image, see details here: Docker.

To be able to read the configuration file from AWS S3 you need to enable the TDK_AWS_ENABLED property, see Application properties.

Database credentials can be requested from AWS Secrets Manager:

Where:

By default, TDK preserves referential integrity based on the foreign keys in the source database schema. In this release, added the ability to provide additional foreign keys in the yaml configuration file.

For example, if order.user_id is a foreign key referred to user.id, and it’s not defined in the database schema, then the following configuration can be provided:

More details about the additional foreign keys in the Configuration reference section.

Significant performance improvement for the subsetting mode.

To enable the performance improvement, the following application property should be set:

In the RELAXED mode, constant generators constant_numeric, constant_string, constant_date, constant_boolean will be chosen by default where the source column contains the same value in all rows.

FAQ page in the documentation.

By default, STRICT mode is enabled. If no suitable transform is found, then the passthrough, null_generator and categorical_generator will not be chosen by default.

To keep the behavior of previous versions, you can use the RELAXED mode.

Data Filtering feature.

Multiple Database support.

YAML anchors and aliases support to reduce repeated sections in the configuration.

The latest free TDK CLI version is currently available in the documentation for download.

GitHub Actions Integration page for the free TDK CLI version in the documentation.

--dry-run, --default-config, --json-schema, --license-expiration options have been turned into separate commands:

dry-run command prints the effective configuration to the console (by default) or to the file (specified as -ec or --effective-config-file parameter value).

For example tdk.jar help dry-run:

Fixed the error Error getting connection from data source when running TDK transformation on a huge database.

Fixed the behaviour where ORA-00972: identifier is too long error appeared when working with Oracle DB instances.

Default rules now recognize boolean-typed fields (e. g. TINYINT(1) in MySql or BOOL in PostgreSQL) and use Categorical generator to generate random boolean values in GENERATION and MASKING modes.

The performance of creating effective config is now improved for certain cases.

Allows transforming columns as they are being read from the input database and written to the output database.

For example, a column might need a cast to a different type on read and then a cast back to the original type on write. The following configuration might be provided to address that:

The my_binary_column will be cast to char type on read and the result of transformation will be cast back to binary type on write.

TDK is now aware of unsigned numeric types supported by some DBMS (MySQL, Oracle).

format_preserving_hashing is now collision-free even for short strings.

The --dry-run CLI option now creates a YAML config file that contains transformations for all tables and can be used for further runs.

Introduced the ability to define the list of schemas to process:

schemas: array of String. If not set or null, all schemas available to the source database user will be processed.

Example:

json_pointer_transformer transforms JSON value nodes indicated by JSON pointers, the rest of the values are kept as is:

Refer to JSON Pointer Transformer for more details.

Introduced more empirical default rules for generation from empty database. If a source table is empty and the generated field is not null and no user configuration provided for this field, then reasonable defaults for generators are chosen for the following data types:

The following yaml configuration parameters have been renamed:

Example before:

Example after:

A validation step to check that the output schema is not the source schema to prevent any source data or schema corruption.

By combining Testcontainers with Synthesized TDK, developers can populate any Testcontainers database with synthetically generated data, enabling rapid development of tests for logic which involves interaction with the database. Refer to documentation for more details.

Documentation tutorials for different modes:

A transient local storage area can now be configured to speed up TDK operations. Refer to documentation for more details.

Introduced --json-schema parameter for CLI which prints JSON Schema for the YAML Configuration. This schema can be used in an IDE to provide auto-completion for your YAML and to validate your configuration before run.

Introduced the ability to define the output alphabets in format_preserving_hashing with unicode_block and unicode_range. Refer to custom alphabets for more details.

Introduced more empirical default rules for generation from empty database.

Added the minimum database permissions required to run Synthesized TDK with Oracle database, see Oracle permissions.

Preserve null values for all masking transformers.

Fixed the behaviour when CategoricalGenerator didn’t preserve the input probabilities in masking mode.

Fixed the behaviour when CategoricalGenerator produced 'null' strings instead of null values.

For tables wider than a certain threshold (for example, 21 columns for SQL Server) insertion failed. This issue is now fixed for all supported databases.

Significant performance improvement for MASKING and GENERATION modes for all supported databases.

To use GENERATION mode for empty tables, the user should specify target_row_number at the global or table level:

target_row_number: optional Integer (int64). The absolute size of the output table in rows. This parameter is applicable only for GENERATION mode. If not provided, target_ratio will be used.

More user-friendly messages in scenarios where there is not enough data to determine the generation parameters.

Improved error handling for Microsoft SQL server IDENTITY exceptions.

MySQL database can be used as an input and output database.

Improved performance of all transformations up to 30%.

Improved data quality and reduce memory consumption of date_generator.

Fixed the behavior when the second run with schema_creation_mode: CREATE_IF_NOT_EXISTS configuration fails for Microsoft SQL server.

Fixed IDENTITY property support for not PRIMARY KEY columns. Now int_sequence_generator is used by default for any IDENTITY column.

Oracle database can be used as an input and output database.

New unique_hashing algorithm:

Microsoft SQL Server IDENTITY property support.

The default MASKING behavior has become more secure. Now format_preserving_hashing is always used by default for string columns. To passthrough, the user must configure it explicitly.

conditional_generator now works correctly both when conditional_column is in the same table as the column being generated and when they are in different tables.

New format_preserving_hashing algorithm:

Database permissions page describing the minimum database permissions required to run Synthesized TDK.

A new parameter length_exceeded_mode is added for address_generator and person_generator, it allows to truncate the generated values by the column size.

For example, if the country column is varchar(20), and the generated value is "Saint Vincent And The Grenadines":

All the examples in the documentation are now covered with tests and up-to-date with the Synthesized TDK version.

Fixed an issue with temporary columns after failed execution, now it is cleaned up for most cases.

For more information, see BigID integration.

CREATE_IF_NOT_EXISTS schema creation mode support for Microsoft SQL Server.

Transformations and YAML configuration are now autogenerated and up-to-date.

Getting Started and Installation sections in the documentation with H2 demo database.

Columns with a single value do not fail with an error, but are kept as-is for MASKING mode and filled with nulls for GENERATION.

Subsetting is repeatable with the same global_seed value.

Huge performance improvement for address_generator and person_generator.

XML data type support for Microsoft SQL Server.

Fixed constant_date, now it works for both value and range modes.

Fixed categorical_generator validation, now it fails with not provided probabilities.

Microsoft SQL Server can be used as input and output database.

A hash transformation is applied to each character, which included into the configured group, in a given text so that the output preserves the format but contains different characters. This transformation is secure and non-reversible.

Parameters:

Available character selectors:

Available alphabets:

Available filters:

For more information, see Transformations.

Added new constant generators for numeric, date, string, boolean data types.

For more information, see Transformations.

categorical_generator supports numeric columns.

For more information, see Transformations.

Default value for insert_batch_size increased from 30 to 1000.

In addition to the nice appearance, many pages and yaml examples are now generated automatically from source code and tests, which reduces the number of mistakes and allows the documentation to be up-to-date with the product version.

Enjoy!

This release includes significant performance improvement for MASKING mode with target_ratio: 1.0.

Fixed issues with processing empty tables in MASKING and GENERATION modes.

Fixed issues with processing columns with single values in MASKING mode.

Fixed issue with missing relations in data generation, when GENERATION tables have Foreign Keys to KEEP tables.

To support custom database types:

For example, for the following custom ENUM type:

Use a configuration like this:

For more information, see Custom database types.

Generate a single numeric value for the entire column

Parameters:

Compatible modes: GENERATION,badge-primary MASKING,badge-secondary

Compatible column data types: NUMERIC

Supports multiple columns: No,badge-danger

Example:

For more information, see Transformations.

UUID data type support for MASKING mode.

BIGINT and SMALLINT data type support for GENERATION, MASKING, and KEEP modes.

global_seed to set the seed for random number generators.

An integer 32-bit value between -2147483648 and 2147483647, used a seed for random number generators. The result of generation must be the same each time the generation is being run with the same seed and workflow configuration. By default global_seed is 0.

Example:

For more information, see YAML configuration.

This release includes significant rework of transformation execution internals, bringing the following benefits to end users:

H2 database can be used as input and output database.

SQLite database can be used as input and output database.

The license expiration can be requested via API:

Where:

If the service is up and running correctly, you should receive a 200 status with the body containing information like:

UUID data type support for GENERATION and KEEP modes.

BOOLEAN data type support for GENERATION, MASKING, and KEEP modes.

YAML configuration can be uploaded as a file via API.

The database credentials can be provided from Google Secret Manager:

A new table_truncation_mode:

It allows not to delete existing data from the output database, but to generate additional and append above.

For more information, see YAML configuration.

Supported locales:

For more information, see Transformations.

For currently unsupported types, such as XML datatype, null_generator will be used by default.

Added ways to stop the workflow using workflow_id and workflow_run_id. Improved error handling.

Removed comparison between input and output schema. It allows to process a subset of the input tables.

formatted_string_generator in MASKING mode generates consistent values across the schema.

If the input numeric column contains only positive values, then the generated values will also be positive by default.

There are two table truncation modes:

Usage example for table_truncation_mode:

MASKING mode for tables with CHAR primary keys can be used without any additional configuration. In the previous versions passthrough transformation was used as a workaround.

Composite primary and foreign keys can be automatically handled without any additional configuration. In the previous versions foreign_key_generator was used as a workaround.

Advanced subsetting implementation for KEEP and MASKING modes. In the previous versions some of the tables after subsetting were empty.

Changed CLI parameters from camelCase to kebab-case:

person_generator and address_generator in MASKING mode will generate consistent values across the schema.

For example, all mentions of James Bond with UK address will be masked as Jon Snow with Seven Kingdoms address for any mentions in the schema.

There are four schema creation modes:

Note: If CREATE_IF_NOT_EXISTS, DO_NOT_CREATE modes are used, the target schema should be equal to the source one.

Generate address fields (e.g. street, zip code) and keep them consistent across columns.

Parameters:

Available templates are:

Compatible modes: GENERATION,badge-primary MASKING,badge-secondary KEEP,badge-warning

Compatible column data types: STRING

Supports multiple columns: Yes,badge-success

Example for multiple columns:

Example for a single column:

There are two cycle resolution strategies:

Example for FAIL mode:

Where the employees table contains a cycle reference.

Example for DELETE_NOT_REQUIRED mode:

We have been working hard to combine our products into a single product with enhanced architecture that will enable us to add exciting new features and optimizations!