Lakeflow Jobs란?

[생성] → [트리거 대기] → [실행 시작] → [Task 순차/병렬 실행] → [완료/실패] → [알림/후처리]

{
  "retry_on_timeout": true,
  "max_retries": 3,
  "min_retry_interval_millis": 60000,
  "unlimited_retries": false
}

# Python SDK로 크론 스케줄 설정
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

job = w.jobs.create(
    name="daily-sales-pipeline",
    tasks=[...],
    schedule={
        "quartz_cron_expression": "0 0 2 * * ?",  # 매일 새벽 2시
        "timezone_id": "Asia/Seoul",
        "pause_status": "UNPAUSED"
    },
    email_notifications={
        "on_failure": ["team@company.com"],
        "on_success": ["team@company.com"]
    }
)

[수집] → [정제] → [집계] → [보고서]

[대상 목록 조회] → [For Each: 테이블별 처리] → [완료 알림]

# databricks.yml
bundle:
  name: sales-pipeline

resources:
  jobs:
    daily_sales:
      name: "daily-sales-pipeline"
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: "Asia/Seoul"

      tasks:
        - task_key: ingest
          pipeline_task:
            pipeline_id: ${resources.pipelines.bronze.id}

        - task_key: transform
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ./notebooks/silver_transform.py

        - task_key: aggregate
          depends_on:
            - task_key: transform
          sql_task:
            query: ./sql/gold_aggregate.sql
            warehouse_id: ${var.warehouse_id}