[experimental] Incremental Masking

If other tables are updated, you can still use the incremental masking. With the previous configuration, TDK will only process the rows in the payment table that have a payment_date greater than the maximum payment_date processed in the previous run, and it will ignore the customer table as specified by the checkpoint.mode: ignore_table option. This will result in ignoring any new rows in the payment table that reference new rows in the customer table.

To address this, you can set the checkpoint in mode incremental for the customer table, similar to the payment table:

In this case, TDK will track both the maximum payment_date in the payment table and the maximum id in the customer table. If new rows are added to the customer table, TDK will process them in the next run, ensuring that all related data is masked correctly.

If the customer table does not have a monotonically increasing column, you can use the checkpoint.mode: refresh option for the customer table, which will cause TDK to reprocess the entire table in each run, while keeping incremental processing for payment table. This is useful if the customer table is updated frequently, and you want to ensure that all data is masked correctly.

If you want to start from scratch and reprocess all the data in the tables, you can set both flag DO_NOT_CONTINUE_FROM_CHECKPOINT and either table_truncation_mode to TRUNCATE or schema_creation_mode to DROP_AND_CREATE in the configuration file. This will cause TDK to ignore the checkpoint, get rid of the existing data in the output tables, and remember the maximum values in the checkpoint columns.

First, you need to execute a clear TDK run using the settings

To continue with the incremental runs, you can remove the DO_NOT_CONTINUE_FROM_CHECKPOINT flag and use the same configuration as before:

If you want to have more control over the checkpoint values, you can use the constant source type in the checkpoint configuration. This allows you to specify a fixed value start_from as the starting point for incremental processing. In this case, TDK will not remember the last processed value, and you will need to update the value manually for each run.

The configuration would look like this:

In this example, TDK will process rows in the payment table where payment_date is greater than 2025-08-08 00:00:00. You will need to update the start_from field in the configuration file for each run to ensure that only new rows are processed.

It is possible to combine incrementality with subsetting. The most straightforward way is to use filter field in the table configuration to limit the rows processed.

If you want to maintain a subset of the data while using incremental masking, you can set up a filter that works in conjunction with the incrementality. For example, you can filter rows based on a date range and use the checkpoint to process only new rows within that range.

The following configuration will maintain a subset of the payment table where payment_date is greater than 2024-01-01 and process only new rows added since the last run:

It is also possible to use subsetting with target_ratio in combination with incremental masking. However, this approach is more complex and might not be efficient for large datasets.