This documentation is available to agents as Markdown at https://docs.synthesized.io/tdk/latest/user_guide/020_guides/unstructured_data/processing_files.md. For the complete documentation index, see llms.txt.

Process Unstructured Files

Configure the input scope and treatment policy, process representative files, and review the transformed outputs before running a larger collection.

This guide describes the operational sequence for a configured file-processing integration. For the underlying processing model, see Unstructured Data.

Use the interface supplied for the selected integration. The document-masking CLI processes PDF, DOCX, or PPTX files, with one format per batch. It is not an archive or mixed-file ingestion interface. This guide does not introduce a universal upload endpoint or a new configuration schema.

Before You Start

Confirm the following:

  • Processing environment: the required application components and any local detection models are available.

  • Input adapters: the integration can read the selected file types and relevant embedded content.

  • Policy: the sensitive categories and required treatments are defined.

  • Consistency scope: the files that must share replacements or resolved identities are identified.

  • Storage: source, working, mapping, and output locations have suitable access and retention controls.

  • Acceptance criteria: representative files and expected results are available for review.

For Governor-backed document masking, confirm the Governor connection, masking entitlement, and credentials required by the selected release. The documented v0.3.0 masking workflow requires administrator credentials; the access key used for generation is not sufficient for masking. Use HTTPS with certificate verification for the Governor connection.

1. Define the Input Set

Start with representative examples of each relevant input type. Include files that exercise the structures the integration must handle, such as message history, spreadsheet fields, document tables, and scanned pages.

For archives, identify which members will be processed. Confirm behavior for nested archives, attachments, encrypted files, and unsupported member types. Record exclusions so a successfully processed outer archive is not mistaken for complete coverage of every file inside it.

Keep the source files separate from output files. Define the expected output format for each input type, including cases where extracted or transformed text will be delivered instead of the original format.

2. Define the Treatment Policy

For each sensitive category, specify whether to tokenize, replace, redact, or retain it, using the options available in the selected workflow. Review categories in context: the same type of text can serve different roles in a document or table.

For example, a contact identifier may need a consistent reference token, a field used for format validation may need a synthetic replacement, and free-text content outside the required test scope may be redacted. These are policy choices, not executable configuration examples.

Keep the selected policy with the run configuration so reviewers can interpret the output and repeat the same evaluation.

3. Define the Consistency Boundary

Group files that need to retain recognized relationships. Where the integration supports identity resolution, provide the available identifiers or contextual sources used to establish those relationships.

Review ambiguous references separately. Matching repeated text is not the same as determining that a shortened name and an email address identify the same person.

Do not assume that a mapping persists across independent runs or that file identities automatically match database identities. Confirm the required behavior before using outputs in a workflow that depends on those links.

4. Process a Representative Sample

Run the configured extraction, detection, transformation, and reconstruction steps using the selected integration. Keep the transformation settings and input scope fixed while reviewing the first results.

Inspect detection results where available. Check whether sensitive spans or fields were located correctly and whether important information was missed. For scanned documents, assess both OCR quality and sensitive-data detection; a text recognition error can prevent a value from being found.

For the dedicated document-masking CLI, list inputs explicitly and keep each batch to one supported format. Use the supplied synthetic sample documents to verify the processing chain before evaluating sensitive source documents.

5. Inspect Outputs and Reports

Check the delivered files against the output contract:

  1. Open them in the applications that will consume them.

  2. Confirm the expected files and content were produced, and account for refusals, exclusions, or text fallbacks.

  3. Check that detected sensitive values have been transformed at each occurrence.

  4. Verify that related references receive the intended consistent treatment.

  5. Review known-value checks and compare a labeled sample against expected sensitive information.

  6. Separate approved outputs from source data, identity mappings, and sensitive working files.

Per-document reports and batch manifests have different contents. In the documented v0.3.0 masking workflow, a batch manifest can contain repeated original values. Reports and logs can also identify source filenames or paths. Review these artifacts before sharing them.

Interpret Document-Masking Results

The document-masking CLI reports the following exit codes:

Code Meaning

0

All documents passed the check for detected original values.

1

At least one document failed that check. Do not release the output without resolving the failure.

2

The input or configuration was refused. Some per-document failures can occur after earlier outputs have been written; inspect the batch results.

3

The Governor-backed masking operation could not run.

A pass does not mean that no personal information remains: it is bounded by what detection found. The one-call masking HTTP service also has different return behavior from the CLI; a successful HTTP response must not be used as the sole acceptance check.

6. Run the Agreed Input Set

After the sample meets the agreed criteria, process the remaining inputs with the reviewed configuration. Revisit the evaluation when file structures, languages, scan quality, treatment rules, or downstream requirements change.

Define retention for source files, intermediate data, mappings, and outputs. Confirm the selected interface’s cleanup and error behavior rather than assuming that completion removes every sensitive artifact.

Related Pages