DummyOperator with BranchPythonOperator

Example using DummyOperator with BranchPythonOperator in a DAG. In this scenario, the DAG decides between two branches based on a condition, with DummyOperator marking the end of each branch.

Example: Conditional Workflow with Branching and Dummy Operators

Let’s say we want the DAG to check if a number is even or odd. Based on this condition, it will follow one of two paths, with each path ending in a DummyOperator.

python
from airflow import DAG
from airflow.operators.python import PythonOperator, BranchPythonOperator
from airflow.operators.dummy import DummyOperator
from datetime import datetime
import random

# Define the DAG
with DAG(
    dag_id="branching_with_dummy_operator_dag",
    start_date=datetime(2023, 10, 1),
    schedule_interval="@daily",
    catchup=False,
) as dag:

    # Branching function to decide path based on even or odd check
    def decide_even_or_odd():
        random_number = random.randint(1, 100)
        print(f"Generated random number: {random_number}")
        if random_number % 2 == 0:
            return "even_path"
        else:
            return "odd_path"

    # BranchPythonOperator to choose the path
    branching = BranchPythonOperator(
        task_id="branching",
        python_callable=decide_even_or_odd
    )

    # Define the tasks for each path
    def process_even():
        print("Processing even number...")

    def process_odd():
        print("Processing odd number...")

    even_task = PythonOperator(
        task_id="even_task",
        python_callable=process_even
    )

    odd_task = PythonOperator(
        task_id="odd_task",
        python_callable=process_odd
    )

    # Dummy operators to mark the end of each branch
    even_end = DummyOperator(task_id="even_end")
    odd_end = DummyOperator(task_id="odd_end")

    # Define dependencies
    branching >> [even_task, odd_task]
    even_task >> even_end
    odd_task >> odd_end

Explanation of the DAG

1. Branching Logic:

   • The decide_even_or_odd function generates a random number between 1 and 100.
   • If the number is even, it returns "even_path", directing the workflow to the even_task.
   • If the number is odd, it returns "odd_path", directing the workflow to the odd_task.

2. BranchPythonOperator:

   • The branching task uses decide_even_or_odd to choose between the two paths, even_task and odd_task.

3. Dummy Operators as End Points:

   • Each branch ends with a DummyOperator, even_end or odd_end, serving as placeholders to mark the end of each path.

4. Skipped Tasks:

   • The branch not selected (either even_task or odd_task) will be marked as skipped, along with its corresponding DummyOperator end marker (even_end or odd_end).

Benefits

Using DummyOperator in combination with BranchPythonOperator clearly defines the end of each conditional path, making the workflow easier to manage and read. It also allows easy extension if you want to add more steps after each branch later.

Post Reference: Vikram Aristocratic Elfin Share