This documentation is available to agents as Markdown at https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/how_masking_works.md. For the complete documentation index, see llms.txt.

How Masking Works

Masking writes a copy of the input to the output with the sensitive values replaced. The tables, rows and relationships stay the same, so applications and tests work with the result. The input database is only read.

What MASKING mode does

In a table in MASKING mode, every column gets a transformer:

  • the one you set for that column in the configuration, if any;

  • otherwise, one chosen by the engine’s default rules from the column’s type and role;

  • otherwise, the safety mode decides. With STRICT, the default, the run stops with an error rather than copy the column unmasked.

The output has the same rows as the input, unless you also subset the table.

How keys stay valid

By default, numeric and date keys are hashed: primary and foreign keys get unique_hashing if they’re numeric, and date_time_unique_hashing if they’re dates. A hashing transformer gives the same output for the same input, and different outputs for different inputs. For keys of other types, check which transformer the engine chooses, in the workflow’s Database view, or with dry-run in the CLI. So a customer’s ID gets a new value, every foreign key that refers to that customer gets the same new value, and joins still work.

In this example, the keys use the default, and the name and email columns use a person_generator:

Column Input Output (illustrative)

customer.customer_id

148

730215

rental.customer_id, in each of the customer’s rentals

148

730215

customer.first_name

Eleanor

Sarah

customer.email

eleanor.hunt@sakila.org

sarah.mills2741@example.com

To keep the original key values instead, set passthrough on the key and on the foreign keys that refer to it.

Choosing transformers

The default rules mask every column, but the values they produce may not look realistic or pass your application’s checks. For those columns, choose a transformer. The most common choices:

You want Transformer

Realistic names, emails, usernames and phone numbers, consistent with each other

person_generator

Realistic addresses

address_generator

Card numbers, IBAN and BIC codes, and other financial values

finance_generator

The same format with different characters, for IDs and codes

format_preserving_hashing

Part of a value hidden, such as all but the last digits of a card

redaction

Numbers close to the real ones

noising

Values that follow a pattern

formatted_string_generator

A field inside JSON or XML

json_pointer_transformer, xpath_transformer

Your own logic

scripting_transformer

The original value

passthrough

See Masking Transformers for the full list. The sensitive data scan finds the columns that hold personal data and suggests a transformer for each.

Consistent values

  • Hashing transformers are consistent by design: the same input always gives the same output.

  • Generators such as person_generator produce a new value for each row. Set consistent_with_column to tie them to another column: with consistent_with_column: "customer_id", all rows with the same customer_id get the same name. The value self ties the output to the input value, so the same real name always becomes the same fake name.

  • Across columns, one person_generator on several columns keeps them consistent with each other: the generated email matches the generated name.

  • Across runs, hashing transformers keep giving the same output for the same input. For generators, see Repeatable runs.

KEEP and MASKING

Both copy rows from the input. KEEP passes every column through unchanged unless you set a transformer; MASKING masks every column unless you set otherwise. A common pattern is MASKING by default, with reference tables such as countries or currencies in KEEP:

default_config:
  mode: MASKING
tables:
  - table_name_with_schema: "public.country"
    mode: KEEP
  - table_name_with_schema: "public.language"
    mode: KEEP

What masking doesn’t do

  • It doesn’t change the input database, unless you set the workflow to write into the same database.

  • It doesn’t mask a column that has passthrough, or that safety_mode: RELAXED passes through.

  • It doesn’t search free text for personal data. Transformers work on column values, or on the JSON or XML fields you select. For documents and other files, see Unstructured Data.

Will masking break my application?

The output keeps the schema, data types, constraints and keys. What can break is a value your application checks: a check digit, a format, a length or a business rule. For those columns, choose a transformer that produces valid values, such as finance_generator for card numbers and IBAN codes, or write the rule with the scripting_transformer, then test your application with the output.

Related pages