# Workflows and Runs

> Source: https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/workflows
> For the complete documentation index, see [llms.txt](https://docs.synthesized.io/llms.txt).

A workflow says what to do with each table of the input. This page explains how its configuration is structured, how every column gets a transformer, and what the engine does when the workflow runs.

## What a workflow is

A workflow joins three things:

- an **input** data source, which is read;
- an **output** data source, where the result is written;
- a YAML **configuration**, which says what to do with each table.

In Governor, a workflow belongs to a [project](https://docs.synthesized.io/tdk/latest/user_guide/010_get_started/projects), and can only use the data sources added to that project. You build it in the web UI, from a template or in the YAML editor, and saved changes are kept as [versions](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/version_history), unless versioning is turned off for the workflow. You run it from the web UI or through the [REST API](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/advanced_workflows/external_api_reference). With the [TDK CLI](https://docs.synthesized.io/tdk/latest/user_guide/050_deployment_operations/installation_methods/cli), the same configuration is a file you pass on the command line.

## The configuration

A configuration has three levels:

 
| Level | What it sets |
| --- | --- |
| 
**Run-wide settings**

 | 

How the output is prepared and how the engine behaves: `schema_creation_mode`, `table_truncation_mode`, `safety_mode`, `cycle_resolution_strategy`, `global_seed`, and optional pre- and post-scripts.

 |
| 

**`default_config`**

 | 

What happens to every table unless a table says otherwise: its `mode`, its size (`target_ratio`), and the default rules for choosing transformers.

 |
| 

**`tables`**

 | 

Overrides for particular tables: a different mode or size, a `filter`, and the transformers for particular columns.

 |

For example, this configuration masks every table, keeps half the customers, keeps only recent rentals, copies `country` unchanged, and sets how three customer columns are masked:

```yaml
default_config:
  mode: MASKING

tables:
  - table_name_with_schema: "public.customer"
    target_ratio: 0.5
    transformations:
      - columns: ["first_name", "last_name", "email"]
        params:
          type: person_generator
          column_templates: ["${first_name}", "${last_name}", "${email}"]
  - table_name_with_schema: "public.rental"
    filter: "rental_date >= '2022-07-01'"
  - table_name_with_schema: "public.country"
    mode: KEEP

schema_creation_mode: CREATE_IF_NOT_EXISTS
table_truncation_mode: TRUNCATE
safety_mode: STRICT
```

Every key is described in the [Configuration Reference](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/configuration_reference).

## Modes

Each table has one mode, set in `default_config` or per table, so one workflow can mix them.

  
| Mode | What happens to the table | Output size |
| --- | --- | --- |
| 
`KEEP`

 | 

Rows are copied as they are.

 | 

The same, or smaller with `target_ratio` below 1 or a `filter`

 |
| 

`MASKING`

 | 

Every column gets a masking transformer, unless you set another one. See [How Masking Works](https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/how_masking_works).

 | 

The same, or smaller with `target_ratio` below 1 or a `filter`

 |
| 

`GENERATION`

 | 

New rows are generated from what the engine learns from the input. See [How Generation Works](https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/how_generation_works).

 | 

Any: `target_ratio` can be above 1, or set `target_row_number`

 |

Making a table smaller is [subsetting](https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/how_subsetting_works). It isn’t a separate mode: it works with `KEEP` and `MASKING`.

## How each column gets a transformer

A transformer produces a column’s output values. For each column, the engine uses the first of these that applies:

1. **A transformer you set** for that column in `tables`, under `transformations`. You can also assign transformers to many columns at once with `matchers`, by column name, SQL type or table name.
2. **A default rule.** The engine has rules that pick a transformer from the column’s type and role. In `KEEP` mode, columns are passed through unchanged. In `MASKING` mode, for example, primary and foreign keys are hashed. You can add your own rules in `default_config`, under `additional_items`; they take precedence over the rules in `items`.
3. **The safety mode**, when no rule matches:
   
   - `STRICT`, the default: the run fails, so no column passes through unmasked by accident.
   - `RELAXED`: in `MASKING` mode the column is copied unchanged (`passthrough`); in `GENERATION` mode it gets a `null_generator` or a `categorical_generator`. A column that holds the same value in every row gets a constant generator.

> **WARNING**
> With `safety_mode: RELAXED`, columns that no rule matches are copied unmasked. Use it only when you have checked which columns those are.

The [sensitive data scan](https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/pii) can write the `transformations` for you: it finds the columns that hold personal data and suggests a transformer for each.

## What a run does

[![A run has five steps. 1: read the input’s tables, columns, keys and foreign keys, plus any virtual foreign keys from the configuration. 2: prepare the output: create the schema and empty tables as the configuration says, and run the pre-script if there is one. 3: plan the order, parents before children, resolving cycles as the cycle resolution strategy says. 4: for each table, read its rows with the table’s filter, choose the rows to keep, keeping only rows whose parents were kept, transform each column with its transformer, and write the rows in batches. 5: run the post-script, which runs even if the run failed, and report status and logs to Governor.](https://docs.synthesized.io/tdk/latest/user_guide/_images/cc-run.svg)](https://docs.synthesized.io/tdk/latest/user_guide/_images/cc-run.svg)

1. **Read the input.** The engine reads the schema: tables, columns, primary keys and foreign keys. [Virtual foreign keys](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/virtual_foreign_keys) in the configuration are added to the ones in the database.
2. **Prepare the output.** It creates the output schema according to the [schema creation mode](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/managing_data_properties/schema_creation), empties tables according to the table truncation mode, and runs the pre-script, if there is one.
3. **Plan the order.** Tables are processed parents first, so that every foreign key in the output can point to a row that’s already there. A foreign key cycle, such as an `employees` table that references itself, is resolved as `cycle_resolution_strategy` says. See [Cycle resolution strategy](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/configuration_reference#CycleResolutionStrategy).
4. **Process each table.** The engine reads its rows, applying the table’s `filter`; keeps only the rows whose parent rows were kept, and as many as `target_ratio` asks for; gives each column its output values; and writes the rows to the output in batches of `insert_batch_size`, 1,000 by default. In `GENERATION` mode, it generates new rows from what it learned about the input, instead of choosing input rows.
5. **Finish.** It runs the post-script, if there is one; the post-script runs even when the run fails. Governor records the run’s status and logs in the run history. The CLI prints them.

## Repeatable runs

`global_seed` seeds the random number generators. For generation, the reference states that the same seed and the same configuration give the same result each time. The default seed is empty; set your own base64-encoded value to get a different result.

Hashing transformers, such as `unique_hashing`, give the same output for the same input, which is how keys stay consistent across tables.

## Where a run happens

In Governor, a run is a task that a [worker](https://docs.synthesized.io/tdk/latest/user_guide/030_core_concepts/architecture#_how_workers_and_governor_connect) takes. Worker groups decide which workers can take it; see [Synthesized Workers](https://docs.synthesized.io/tdk/latest/user_guide/020_guides/advanced_workflows/workers). With the CLI, the run happens on the machine where you start it.

## Related pages

- [Create a Workflow](https://docs.synthesized.io/tdk/latest/user_guide/010_get_started/workflow): build one in the web UI.
- [Configuration Reference](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/configuration/configuration_reference)
- [All Transformers](https://docs.synthesized.io/tdk/latest/user_guide/040_reference/transformers/transformations)