Incremental Model Strategy Selector

Incremental Model Strategy Selector

Overview

Selects and configures optimal incremental model strategies. This skill optimizes data transformation efficiency through proper incremental processing patterns.

Capabilities

• Incremental strategy selection (append, merge, delete+insert)
• Partition pruning optimization
• Unique key configuration
• On_schema_change handling
• Full refresh scheduling
• Lookback window optimization
• Late-arriving data handling

Input Schema

{
  "modelCharacteristics": {
    "sourceType": "string",
    "updatePattern": "append|update|delete",
    "volumeGB": "number",
    "updateFrequency": "string"
  },
  "platform": "snowflake|bigquery|redshift",
  "existingModel": "object"
}

Output Schema

{
  "strategy": "append|merge|delete+insert",
  "config": "object",
  "partitionStrategy": "object",
  "refreshSchedule": "object",
  "dbtConfig": "object"
}

Target Processes

• Incremental Model Setup
• dbt Model Development
• Pipeline Migration

Usage Guidelines

1. Analyze source data update patterns
2. Measure data volume and update frequency
3. Select strategy based on characteristics
4. Configure appropriate lookback windows

Best Practices

• Use append for insert-only sources
• Use merge for sources with updates
• Configure partition pruning for large tables
• Schedule periodic full refreshes for data correction
• Handle late-arriving data with appropriate lookback