This documentation is available to agents as Markdown at https://docs.synthesized.io/tdk/latest/user_guide/020_guides/generating_data/csri_graph_and_run_history.md. For the complete documentation index, see llms.txt.

Cross-System Referential Integrity

Keep generated and masked data linked across the systems in a project, and see those links on the Cross-system references screen.

Cross-system referential integrity is off by default. While it’s off, its screens are hidden and its API endpoints return 404. See Turn it on.

Overview

Cross-System Referential Integrity (CSRI) links related records across different systems in a project - for example, a customer ID that appears in a CRM database, an accounts database, and a billing system. Synthesized uses these links so that data stays connected across every system that references the same record: generated child records point at keys that exist in the parent system, and cross-system keys are kept intact when masking.

When masking, a column that refers to another system is copied from that system’s output, so it holds the value that was already masked there, not the original. To change this, set another transformer on the column in the workflow configuration.

The Cross-system references screen of a project shows these links. It has two tabs:

  • Graph - a visual map of the systems and the references between them, with a Run all button that runs the project’s workflows in dependency order.

  • Run history - the list of past Run all runs, with details for each run.

Turn it on

Set the feature flag UI_CSRI to "true" on the Governor API container, then restart that container. Setting it on the frontend or worker containers has no effect.

  • Docker Compose

  • Helm

docker-compose.yml already passes UI_CSRI to the backend service, so add this line to .env:

UI_CSRI=true

Add the flag under api.container.config in values.yaml:

api:
  container:
    config:
      UI_CSRI: "true"

Example scenario

Example: linked test data across three systems

The setup. A retail bank keeps customer records in an Oracle CRM, accounts and transactions in PostgreSQL, and daily statement extracts as CSV files. The same customer ID appears in all three.

The problem. To test a customer journey end to end, from onboarding to account opening to the monthly statement, test teams need realistic data in all three systems at once. Preparing each system separately breaks the links: a customer in the CRM test data no longer matches any account in PostgreSQL or any row in the statement file. End-to-end tests then fail for reasons unrelated to the code under test.

What’s needed. Test data that is privacy-safe in every system and still connected across them, so a record can be followed through every system that depends on it.

With Synthesized. Add the three data sources to one project and declare which fields refer to each other across systems. Synthesized then produces the data for all three systems in one coordinated run, so every customer ID in the output resolves in each system that references it. Use Graph to see how the systems are linked and to start that run, and Run history to review each run.

Cross-system links are declared in YAML only, in the workflow configuration, for example in the workflow’s Configuration tab. There’s no form for them: the Cross-system references screen shows the links that the configurations declare, but can’t create them.

In the workflow of the system that holds the referring column, add a foreign key under metadata.tables and set referred_database to the slug of the data source that holds the referred table:

metadata:
  tables:
    - table_name_with_schema: "public.accounts"
      foreign_keys:
        fk_accounts_crm_customer:
          referred_database: "crm"
          referred_schema: "CRM"
          referred_table: "CUSTOMERS"
          columns:
            - column: "customer_id"
              referred_column: "CUSTOMER_ID"

The other keys work as they do for virtual foreign keys. For every key, see Configuration Reference: Additional Foreign Key.

A data source’s slug is in the Slug field of its form, when you create or edit the data source, and in small text on its node in the Cross-system references graph. It isn’t shown in the data source list. In the REST API, it’s the data source’s slug field.

Prerequisites

  • Cross-system referential integrity is turned on. See Turn it on.

  • A project whose workflows declare cross-system links. See Declare a link.

Steps

  1. Open the project.

  2. Open the project’s actions menu (the three dots next to the project name) and select Cross-system references.

    The project’s actions menu with Cross-system references highlighted.
  3. Select the Graph tab to view the reference graph. Run all starts a run of every workflow in the graph.

    The Graph tab showing the systems and the references between them.
  4. Select the Run history tab to view past runs.

    The Run history tab listing past runs.
  5. Select a run from the list to open its run details.