Flyte

Flyte is an open-source workflow orchestration platform built specifically for AI, ML, and data pipelines. You author workflows in plain Python (no proprietary DSL), test them locally, then run them at scale with durable execution, automatic retries, dynamic runtime branching, and infrastructure-aware autoscaling. The platform handles agent-based workflows, generative AI inference, distributed model training (via Spark, PyTorch, Ray), and classic ETL with built-in data lineage, versioning, and resource observability.

Flyte sits in the same conceptual space as Airflow, Prefect, and Dagster but is markedly more opinionated about ML and GPU workloads, with strong typing, container-per-task isolation, and first-class support for long-running model training. The OSS version (6k+ GitHub stars) is genuinely free and self-hostable; the commercial cousin Union.ai layers on enterprise scale (50k+ actions per run, sub-100ms task startup, support contracts). It's used in production by Mistral, NVIDIA, Tesla, Adobe, and Shopify, which is a meaningful signal for a workflow engine.

Best for teams that already live in Kubernetes and Python and want one orchestrator covering both pre-training pipelines and inference-time agent workflows. Less appropriate if you just need a quick agent loop or a no-code automation tool.