Data Protection

Whether an output is safe to share depends on the workflow’s configuration, not on the platform alone. This page explains which configurations can put real values in the output, and what to check before you share one.

What the output can contain

Output databases receive masked, subsetted or generated data, and unmasked sensitive values don’t land downstream unless the workflow’s configuration keeps them. These are the setups that keep real values:

Setup Real values? Why

Setup	Real values?	Why
A table in `KEEP` mode	Yes	KEEP copies rows as they are: columns get `passthrough` unless you set a transformer.
A subset (`target_ratio` below 1) in KEEP mode	Yes	Subsetting picks fewer rows; it doesn’t change their values. Combine it with MASKING to mask the rows you keep.
A column with `passthrough`	Yes	`passthrough` copies the value unchanged.
MASKING with `safety_mode: RELAXED`	Possibly	A column that no rule matches gets `passthrough`, instead of stopping the run as `STRICT` does.
A `categorical_generator` with categories learned from the input	Yes, for that column	It outputs values that occur in the input. That’s what you want for a status or country code, not for a name or account number.
GENERATION with `safety_mode: RELAXED`	Possibly	A column that holds one value in every row gets a constant generator with that value.
A transformer that changes only part of a value: `redaction`, or `json_pointer_transformer` and `xpath_transformer` on some fields	Yes, in the parts it keeps	These transformers keep the rest of the value. Transformers don’t search free text for personal data.
MASKING with `safety_mode: STRICT` and no `passthrough`	Only where a transformer you chose keeps them	Every column gets a masking transformer, and a column no rule matches stops the run.
GENERATION	Only through the exceptions above	Generated values are sampled from what the engine learned, not copied.

A table in KEEP mode

Yes

KEEP copies rows as they are: columns get passthrough unless you set a transformer.

A subset (target_ratio below 1) in KEEP mode

Yes

Subsetting picks fewer rows; it doesn’t change their values. Combine it with MASKING to mask the rows you keep.

A column with passthrough

Yes

passthrough copies the value unchanged.

MASKING with safety_mode: RELAXED

Possibly

A column that no rule matches gets passthrough, instead of stopping the run as STRICT does.

A categorical_generator with categories learned from the input

Yes, for that column

It outputs values that occur in the input. That’s what you want for a status or country code, not for a name or account number.

GENERATION with safety_mode: RELAXED

Possibly

A column that holds one value in every row gets a constant generator with that value.

A transformer that changes only part of a value: redaction, or json_pointer_transformer and xpath_transformer on some fields

Yes, in the parts it keeps

These transformers keep the rest of the value. Transformers don’t search free text for personal data.

MASKING with safety_mode: STRICT and no passthrough

Only where a transformer you chose keeps them

Every column gets a masking transformer, and a column no rule matches stops the run.

GENERATION

Only through the exceptions above

Generated values are sampled from what the engine learned, not copied.

See How Masking Works and How Generation Works.

Keyed masking

Masking is keyed: masked values can’t be reversed without your masking key, which stays in your environment.

In MASKING mode, primary and foreign keys get a hashing transformer, such as unique_hashing for numeric keys. Under one masking key, the same input always gives the same output, in every table and data source, so joins and referential integrity hold.

That consistency keeps the data usable, but it also means that someone who knows an input value and its masked value can recognise it again in other outputs made with the same key. Treat masked data as pseudonymised rather than anonymous, keep the masking key apart from the masked data, and protect both.

Finding sensitive columns

A PII scan checks column values against patterns for types of sensitive data, such as emails, card numbers and national identifiers, and marks the columns that match. Use it as a starting point:

Review the columns it marks before you choose transformers for them.
Look for sensitive columns it can’t recognise by pattern, such as free text, notes and comments.
Run it again when the schema changes.

Documents and other files

For documents, spreadsheets, emails and images, docanon finds and masks sensitive values in the files. It runs on your own machines and doesn’t connect to Governor. A value it doesn’t detect stays in the file, so check its results before you use them. See Unstructured Data.

Before you share an output

Print the effective configuration with tdk dry-run, and look for passthrough, learned categorical_generator columns and KEEP tables.
Keep safety_mode: STRICT, so the run stops at any column no rule matches, instead of copying it.
Check partial transformers: where redaction keeps characters, or JSON and XML transformers change only some fields, make sure the parts they keep hold no personal data.
Spot-check the output: look for values you know are in the input, such as your own name or a known account number.
Restrict access to the output until you’ve checked it, as you would to the input.
Keep the configuration under review: a new column or table in the input gets a transformer from the rules, so check new columns after schema changes.

The input database

The input is only read. A workflow writes to its input only with the SAME_DATABASE_TRANSFORMATION flag. To make sure, give the platform a read-only database user on input databases and mark them Read-only in Governor. See Database Permissions and Data Sources.