Conceptual Material: Introduction to Orchestration and Kestra

Workflow Orchestration Introduction

https://youtu.be/Np6QmmcgLCs

What is Orchestration?

Orchestration in software development refers to the automated arrangement, coordination, and management of complex systems, services, or workflows to achieve specific goals. It involves organizing and controlling interactions between various components, ensuring that they work together seamlessly to deliver the desired functionality or outcome.

What is Kestra?

• All-in-one automation and orchestration platform
  • ETL/ELT
  • API orchestration
  • Scheduled & event-driven workflows
  • Batch data pipelines
  • Interactive conditional inputs
• Flexibility on how you want to control your workflows: no code, low code, full code.
• Multi-language support
• Monitoring workflows and executions
• 600+ plugins & 250+ blueprints

Learn Kestra

https://youtu.be/o79n-EVpics

<aside> <img src="/icons/info-alternate_green.svg" alt="/icons/info-alternate_green.svg" width="40px" />

Kestra Video Tutorials: https://kestra.io/tutorial-videos/all

Kestra Tutorial: https://kestra.io/docs/tutorial ← This one is straightforward and very good.

Documentation: https://kestra.io/docs

</aside>

• Flows: Workflows declared using YAML syntax (to be language agnostic).
  • Required fields: id (name of the flow), namespace (used to separate environments, e.g., dev and prod), and tasks (list of task to execute sequentially in the defined order). https://kestra.io/docs/tutorial/fundamentals
• Tasks: Atomic actions in flows. There are many task types predefined.
  • Can be small and granular, a whole script, or large and complex processes.
  • Are defined in the form of a list and executed sequentially in the order of definition.
  • Can be run in parallel (Flowable tasks).
  • Required fields: id (name of the task), and type (a task type). Other required fields depend on the task type. https://kestra.io/docs/tutorial/fundamentals#supported-task-types
• Inputs: Allow parameters for workflows so they can be more dynamic and reusable.
  • Defined in flows with a sequence of mappings called inputs.
  • Required fields for each element of the sequence: id and type. https://kestra.io/docs/tutorial/inputs
    • With defaults we can define a default value.
• Outputs: Allow to pass data between tasks and flows.
  • Can be variables or files stored in the internal storage. https://kestra.io/docs/tutorial/outputs
• Triggers: Allow automatic execution of flows based on events.
  • Defined in flows with a sequence of mappings called triggers.
  • Required fields: id and type; other required fields depend on the type of trigger. https://kestra.io/docs/tutorial/triggers
• Flowable tasks: Run tasks or subflows in parallel, create loops and conditional branching. https://kestra.io/docs/tutorial/flowable