Reading and Analyzing Logs

Understand TDK logs to diagnose and troubleshoot issues.

Log Locations

Docker Compose

# View backend logs
docker compose logs backend

# Follow logs in real-time
docker compose logs -f backend

# View last 100 lines
docker compose logs --tail=100 backend

# View agent logs
docker compose logs agent

Kubernetes

# View backend logs
kubectl logs -n tdk deployment/tdk-backend

# Follow logs
kubectl logs -n tdk deployment/tdk-backend -f

# View specific pod
kubectl logs -n tdk tdk-backend-abc123-xyz

# View agent logs
kubectl logs -n tdk deployment/tdk-agent

CLI Mode

Logs output to stdout/stderr:

tdk --config-file workflow.yaml 2>&1 | tee tdk.log

Log Levels

TDK uses standard log levels:

  • ERROR: Critical failures requiring attention

  • WARN: Potential issues or deprecated features

  • INFO: General informational messages

  • DEBUG: Detailed diagnostic information

  • TRACE: Very detailed execution traces

Enabling Debug Logging

Docker Compose

services:
  backend:
    environment:
      LOGGING_LEVEL_IO_SYNTHESIZED: DEBUG

Application Properties

logging.level.io.synthesized=DEBUG
logging.level.org.springframework=INFO

Understanding Log Messages

Successful Workflow Execution

INFO  Starting workflow execution: workflow_id=123
INFO  Processing table: public.customers
INFO  Rows processed: 10000/10000 (100%)
INFO  Workflow completed successfully in 45.2s

Common Error Patterns

Connection Error:

ERROR Failed to connect to database: java.sql.SQLException: Connection refused

Foreign Key Violation:

ERROR Referential integrity error: FK constraint violated
  Table: orders, Column: customer_id
  Parent: customers, Parent Column: id

Memory Error:

ERROR OutOfMemoryError: Java heap space
  at io.synthesized.tdk.processor.DataProcessor.process

Searching Logs

Find Errors

docker compose logs backend | grep -i error

# With context (5 lines before/after)
docker compose logs backend | grep -i -C 5 error

Find Specific Workflow

docker compose logs backend | grep "workflow_id=123"

Time-Based Search

# Logs since specific time
docker compose logs backend --since 2024-01-15T10:00:00

Exporting Logs

Save logs for support:

# Docker Compose
docker compose logs backend > tdk-backend.log
docker compose logs agent > tdk-agent.log

# Kubernetes
kubectl logs -n tdk deployment/tdk-backend > tdk-backend.log
kubectl logs -n tdk deployment/tdk-agent > tdk-agent.log

Log Rotation

Docker Compose

Configure in docker-compose.yml:

services:
  backend:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

See Also