Unstructured Data

Unstructured data processing extracts content from files, identifies sensitive information, applies a treatment policy, and writes transformed outputs. File processing retains location and context information so replacements can be applied to the correct text, field, or document region.

This page describes the processing model and its boundaries. For the operational sequence, see Process Unstructured Files.

Availability and interfaces depend on the selected workflow. Document masking and synthetic document generation are evaluation-stage capabilities. Processing related files across different formats requires an integration configured for those inputs; it is not implied by the document-masking command.

Processing Modes

The input and required output determine the workflow:

Mode	Input	Result
File de-identification	Existing business files, communication exports, or packaged file collections	Content transformed according to a category-level policy, with consistent references where identities are resolved within the processing scope
Document masking	Existing documents handled by the document-masking workflow	Detected personal values replaced in supported document formats
Synthetic document generation	A blank or dummy-filled form and field generation rules	New documents populated with synthetic values, with expected field values for evaluation

Mode

Input

Result

File de-identification

Existing business files, communication exports, or packaged file collections

Content transformed according to a category-level policy, with consistent references where identities are resolved within the processing scope

Document masking

Existing documents handled by the document-masking workflow

Detected personal values replaced in supported document formats

Synthetic document generation

A blank or dummy-filled form and field generation rules

New documents populated with synthetic values, with expected field values for evaluation

File de-identification and document masking operate on existing content. Generation creates new field values and places them into a template. These modes do not have identical input adapters, output formats, or execution interfaces.

Input Handling

Inputs can include documents, spreadsheets, text-based content, email and messaging exports, and archives such as ZIP files. These are examples of input categories, not a guarantee that every file or embedded object can be processed by every workflow.

An archive is a container for files. Processing its contents requires adapters for the individual file types inside it. The input scope should identify included files, attachments, nested content, and any exclusions. An archive being readable does not by itself establish that all of its contents have been transformed.

The extraction method depends on how the content is stored:

Text-bearing files expose text together with structural information such as paragraphs, messages, headers, or cells.
Scanned documents require optical character recognition (OCR) to recover text and page coordinates.
File collections require a record of each file’s location so transformed content can be associated with its source.

The dedicated document-masking command accepts PDF, DOCX, and PPTX files, with one format per batch. Broader input categories require the corresponding file-processing integration.

Processing Flow

Extract: read content and retain its source location and surrounding context.
Detect: identify sensitive spans or fields using configured recognizers, rules, and local model-based detection where applicable.
Resolve: associate recognized references with the same entity within the selected files.
Transform: apply the configured treatment to each finding.
Reconstruct: write transformed content through the appropriate output writer.
Validate: inspect the delivered outputs for residual values and structural or consistency problems.

The processing representation separates three concerns: what was detected, where it was found, and what treatment should be applied. For example, a finding can refer to a name in a spreadsheet cell, a span in an email body, or text at a page location. This is a processing model, not a public API schema.

Detection and Treatment Policies

Detection identifies candidate sensitive information. A treatment policy determines what happens to a finding. Pattern recognition can identify values with recognizable syntax; contextual detection can identify information whose meaning depends on surrounding text or a field’s role.

Depending on the workflow, treatments include:

tokenize: replace a recognized value with a reference token. Related tokens can retain an association to the same entity.
replace: insert a synthetic value, with consistency or format constraints supported by the configured transformer.
redact: remove the selected content.
retain: leave a category unchanged when the policy permits it.

These are treatment concepts, not configuration keys. Available categories and treatment options are defined by the selected integration. Database-column PII scanning is documented separately in PII Detection & Protection; its UI settings should not be assumed to configure every file-processing detector.

Consistency Across Related Files

Consistent replacement and identity resolution are related but different operations. Consistent replacement maps a recognized input value to the same replacement within a defined scope. Identity resolution determines whether different references, such as a full name and an email address, refer to the same person.

File-processing integrations can use context and known identifiers to resolve related references before applying replacements. Ambiguous references must not be treated as resolved solely because their text is similar.

Table 1. Illustrative example; all input values are fictional
Location	Input reference	Logical association
Document	Alex Morgan	Person 001
Email	alex.morgan@example.com	Person 001, when the identity link is established
Spreadsheet	A. Morgan	Person 001 only when context resolves the abbreviation

The association is used when applying the configured treatment. It is not a promise that ambiguous abbreviations will be matched automatically.

The dedicated document-masking workflow applies one batch-wide transformation to detected values. A broader file-processing integration may maintain a shared entity registry across its selected files. Neither mechanism establishes automatic identity linkage to unrelated batches, separate organizations, or live database records. Those boundaries must be configured and verified separately.

Outputs and Format Behavior

Output behavior is workflow-specific:

The document-masking PDF path reconstructs image-based PDFs. The output is not a searchable-text copy of the source.
Supported Word and PowerPoint processing rewrites document content and returns the corresponding file format. Unsupported embedded structures can cause a refusal.
Other file-processing integrations can reconstruct native files where supported, but may deliver extracted PDF content or fallback outputs as transformed text.
Synthetic generation produces searchable PDFs, generated rows, field-level expected values, and placement results.

Review the actual delivered files, not only intermediate extracted text. File readability, retained structure, replacement consistency, and removal of detected original values are separate checks.

Data Handling and Deployment

Local processing keeps extraction, detection, and reconstruction in the configured processing environment. Connections to other components depend on the workflow.

In the documented Governor-backed masking workflow, detected real values are replaced with tokens before the transformation table is sent to Governor. The mapping from tokens to real values remains local. This data boundary is specific to that workflow; it is not a blanket statement that every integration has no network dependencies.

Source files, working data, identity mappings, caches, and output reports require appropriate access and retention controls. Do not assume that a report or batch manifest is safe to distribute merely because the transformed document has passed a check.

Validation Boundaries

A known-value check searches for original values identified by detection or supplied as review criteria. It cannot establish that detection found every sensitive value. Detection coverage should be evaluated using representative, labeled examples, including relevant file structures and scan conditions.

Validation should distinguish:

Sensitive information that was not detected.
Detected information that was not successfully transformed.
Incorrect or inconsistent identity associations.
Output files that are unreadable, incomplete, or unsuitable for the consuming application.

Treat completion of processing and acceptance for downstream use as separate decisions.

Synthetic Document Generation

For template-based evaluation, Synthesized generates structured rows and places each row’s values into a document template. Fillable forms provide field geometry; other forms require calibration. The outputs include the generated document and field-level ground truth, so a downstream reader’s extracted values can be compared with the values that were placed.

Use blank or dummy-filled templates. This workflow creates synthetic field values; it is not an alternative way to remove sensitive content from an already completed document.