Best Practices

Production recommendations, performance optimization, and best practices for using TDK effectively and securely.

What You’ll Find Here

This subsection provides best practices for:

  • Performance - Optimizing transformation speed and resource usage

  • Security - Secure configuration and credential management

  • Workflow Design - Organizing and maintaining workflow configurations

  • Testing - Validating workflows before production deployment

  • Monitoring - Tracking transformation progress and health

  • Error Handling - Dealing with failures and edge cases

Key Recommendations

Performance Best Practices

  • Disable indexes during large transformations, rebuild after

  • Use incremental masking for regular updates

  • Leverage TDK agents for parallel processing

  • Tune batch sizes for your database

See: Improving Performance

Security Best Practices

  • Use secret managers (AWS, GCP, Vault) - never hardcode credentials

  • Enable read-only mode on production sources

  • Implement RBAC for multi-user environments

  • Audit workflow executions regularly

See: Security & Compliance

Configuration Best Practices

  • Store workflow configs in version control

  • Use environment variables for environment-specific settings

  • Test workflows on small datasets first

  • Document complex transformations with comments

  • Use reusable scripts for common logic

Data Quality Best Practices

  • Verify referential integrity after transformations

  • Validate data distributions match expectations

  • Test edge cases and null handling

  • Use consistent seeds for reproducible results

Where to Start

Related Topics