Ressources
Retour

Rejoignez l'AI + Data Tour pour bénéficier de formations pratiques, découvrir des témoignages clients concrets et échanger avec les experts produits Domo près de chez vous.

Inscrivez-vous maintenant
À propos
Retour
Récompenses
Reconnu comme Leader pendant
34 trimestres consécutifs
Leader de l'été 2026 en BI embarquée, plateformes d'analyse, BI, outils ETL, préparation de données et gouvernance des données
Tarifs

Machine Learning Pipeline Orchestration Guide: Tools and Patterns

3
min read
Friday, October 16, 2026
Table of contents
Carrot arrow icon

Training a machine learning model is only the beginning. The hard part? Keeping it running reliably with fresh data, clear approvals, and results that actually reach the right people and systems. This guide explains what ML pipeline orchestration is, walks through architecture patterns like batch, micro-batch, and real-time, covers governance and monitoring practices that prevent surprises, and provides a starter blueprint for small teams ready to ship their first orchestrated pipeline this month.

Key takeaways

Here are the main points to keep in mind.

  • ML pipeline orchestration automates the coordination of data ingestion, feature engineering, model training, deployment, and monitoring so predictions reach business workflows reliably.
  • Choose architecture patterns (batch, micro-batch, or real-time) based on decision latency requirements and cost constraints, not technical complexity.
  • Tool selection depends on team size, existing infrastructure, and whether you need general workflow orchestration or ML-specific capabilities.
  • Governance and observability built into pipelines accelerate delivery rather than slow it down by catching issues before they reach production.
  • Start with one use case tied to a clear key performance indicator (KPI), automate the riskiest handoffs first, and expand only after proving value.

Foundations

What is ML pipeline orchestration?

ML pipeline orchestration is the automated coordination of machine learning lifecycle stages, including data preparation, feature engineering, model training, evaluation, deployment, and monitoring, with dependency management, scheduling, retries, and observability built in.

Machine learning pipelines look straightforward on a whiteboard: data is collected, features are engineered, then a model is trained, evaluated, and deployed to make predictions. Reality is messier. Data schemas change. Jobs fail in the middle of the night. Models drift, and handoffs across tools and teams create delaysnobody budgeted for.

Orchestration is the connective tissue that keeps all of these steps running in the right order, at the right time, with the right dependencies. At a minimum, an orchestration system must provide these capabilities:

  • Dependency resolution and scheduling (time-based or event-driven) so tasks run in the correct order
  • Idempotent retries and backfills that recover from failures without corrupting state
  • Artifact versioning with lineage tracking for reproducibility and auditability
  • Triggers that respond to data arrival, drift thresholds, or external events
  • Evaluation gates that block promotion until quality thresholds pass
  • Observability through logs, metrics, and alerts tied to clear service-level objectives (SLOs)
  • Access control and approval workflows for safe promotion to production

What orchestration is not: writing the model code itself, designing features, storing raw data, or providing the inference runtime. Those responsibilities belong to adjacent layers (data engineering, model development, and serving infrastructure) that orchestration coordinates but does not replace.

Think of orchestration as traffic control for your ML pipeline. It lightens the manual work, follows best practices, and keeps models fresh as the business changes. If you're new to core concepts, see AI vs machine learning and common AI models.

Upstream reliability matters. Orchestration usually starts by making data sources trustworthy and timely with modern connectors and change-data-capture. This is where cloud data integration and application programming interface (API) integration earn their keep.

{{custom-cta-1}}

How orchestration differs from related concepts

The term "orchestration" gets used loosely, so it helps to draw boundaries. The following table clarifies where ML pipeline orchestration fits relative to adjacent concepts:

ConceptPrimary goalTypical artifactsCommon triggersExample tools
ML pipeline orchestrationCoordinate the full ML lifecycle from data prep through deployment and monitoringDatasets, features, trained models, evaluation reports, deployed endpointsSchedule, data arrival, drift threshold, manual approvalAirflow, Prefect, Kubeflow Pipelines, SageMaker Pipelines
Data orchestrationMove and transform data between systems (extract, transform, load [ETL] / extract, load, transform [ELT])Raw files, transformed tables, data quality reportsSchedule, change data capture (CDC) events, file arrivalFivetran, Airbyte, dbt, Dagster
Workflow orchestrationAutomate any multi-step business or technical processTask outputs, notifications, approvalsSchedule, API calls, actions from peopleTemporal, n8n, Zapier
MLOps (discipline)Operationalize ML across the entire lifecycleExperiment logs, model registries, monitoring dashboardsVaries by practice areaMLflow, Weights & Biases, Vertex AI

ML pipeline orchestration is a subset of MLOps and often builds on top of data orchestration.

Orchestration also integrates with several adjacent systems that handle specialized responsibilities. The following table clarifies what the orchestrator hands off to each system and what it receives in return:

SystemPrimary jobWhat orchestrator sendsWhat orchestrator receivesCommon failure mode
Experiment trackingLog parameters, metrics, and artifacts from training runsRun ID, hyperparameters, code versionMetric values, artifact URIsOrphaned runs when orchestrator fails mid-job
Feature storeServe consistent features for training and inferenceFeature view version, entity keysFeature vectors, freshness metadataTraining/serving skew from version mismatch
Model registryVersion and stage models through approval workflowsTrained model artifact, evaluation metricsApproved model URI, deployment stagePromotion blocked by missing evaluation data
CI/CD pipelineTest and deploy code changesCode SHA, environment specificationBuild status, deployment confirmationOrchestrator and CI/CD both trying to deploy
Serving infrastructureHandle inference requests at scaleModel URI, resource requirementsEndpoint URL, health statusOrchestrator unaware of serving failures

If your team already runs data pipelines with a tool like Airflow or Dagster, ML orchestration extends that foundation to handle training schedules, evaluation gates, model registries, and deployment automation. Treating ML orchestration as a separate system rather than an extension of existing data workflows creates duplicate infrastructure and inconsistent data lineage. Teams fall into this trap more often than you'd expect.

A common question: is MLflow an orchestration tool? MLflow is primarily an experiment tracking and model registry platform. It excels at logging parameters, metrics, and artifacts from training runs, and it provides a model registry for versioning and staging models through approval workflows. However, it does not handle job scheduling, dependency management, or pipeline coordination on its own. Teams typically pair MLflow with a dedicated orchestrator like Airflow or Prefect to get end-to-end pipeline automation.

Why orchestration matters right now

AI has moved from experiments to operations. In 2026, a McKinsey Global Survey reported that nearly nine in ten organizations are using AI in at least one business function, with 44 percent now scaling AI across the enterprise. The 2026 Stanford AI Index highlights record investment, with US private AI investment surging to $285.9 billion in 2025.

That adoption figure matters because it signals a shift: the question is no longer whether to adopt AI, but how to operationalize it reliably. As adoption and investment rise, the bottleneck shifts from "Can we build a model?" to "Can we run it reliably and connect it to decisions?"

{{custom-cta-2}}

What parts of the ML pipeline get orchestrated?

A minimal production pipeline includes seven stages, each producing typed artifacts that flow to the next:

  1. Data ingestion and validation: Pull sources, check schema and freshness, quarantine bad batches. Outputs validated datasets with quality metadata.
  2. Feature engineering and reuse: Transform data; reuse features across training and serving. Outputs feature vectors with version identifiers.
  3. Training and experimentation: Schedule retrains, tune hyperparameters, track metrics. Outputs trained model artifacts with parameter logs.
  4. Evaluation and approval gates: Compare to the current model; promote only if thresholds pass. Outputs evaluation reports and promotion decisions.
  5. Packaging and versioning: Containerize, register artifacts with lineage for reproducibility. Outputs registered model URIs with full provenance.
  6. Deployment: Batch, micro-batch, or real-time; use canary or blue-green to reduce risk. Outputs deployed endpoints or batch prediction tables.
  7. Monitoring and feedback: Watch performance, drift, and cost; feed outcomes back to retrain. Outputs alerts, drift reports, and retraining triggers.

Each stage can fail in predictable ways. Data ingestion fails when schemas change upstream or sources go offline. Feature engineering fails on null values or type mismatches. Training fails on resource exhaustion or diverging loss. Evaluation fails when baseline metrics are missing. Packaging fails on dependency conflicts. Deployment fails on resource limits or health check timeouts. Monitoring fails when alert thresholds are misconfigured.

The orchestrator's job is to detect these failures, retry safely, and block downstream stages from running on bad inputs.

The following pseudo-code illustrates a minimal six-task directed acyclic graph (DAG) with retry policies and a gating task:

dag mltrainingpipeline:
schedule: daily at 2 am UTC

task ingestdata:
retry: 3 times with exponential backoff
outputs: validateddataseturi

task computefeatures:
dependson: ingestdata
inputs: validateddataseturi
outputs: featureviewversion

task trainmodel:
dependson: computefeatures
inputs: featureviewversion
retry: 2 times
checkpoint: every 30 minutes
outputs: modelartifacturi

task evaluatemodel:
dependson: trainmodel
inputs: modelartifacturi, championmodeluri
outputs: evaluationreport

task approvalgate:
dependson: evaluatemodel
condition: evaluationreport.accuracy > champion.accuracy - 0.01
onfailure: notify team, block promotion

task registeranddeploy:
dependson: approvalgate
inputs: modelartifacturi
outputs: deployedendpointuri



This structure applies to both batch retraining (where the entire DAG runs on a schedule) and streaming-feature architectures (where ingestion and feature computation run continuously while training and evaluation run periodically).

Technical patterns and tooling

Batch, micro-batch, and real-time architecture patterns

Choosing between batch and stream processing is a business decision first. Pick the simplest design that meets your decision service-level agreement (SLA) and budget.

Batch

How it works: Process large volumes on a schedule (hourly or nightly) to compute features, train or score, and publish results.

Strengths: Simple, cheap at scale, reproducible and backfillable.

Trade-offs: Latency in hours; not for on-the-spot decisions.

Orchestrator role: Manages job scheduling, partition-based backfills, and checkpoint recovery for long-running jobs.

Micro-batch

How it works: Run small windows every few minutes; often change data capture (CDC)-triggered plus frequent schedules.

Strengths: Near-real-time freshness without always-on streaming.

Trade-offs: More moving parts and job overhead than batch. Teams sometimes underestimate the operational complexity here. Micro-batch requires careful handling of late-arriving data and window boundaries that batch pipelines simply do not face.

Orchestrator role: Manages window boundaries, late-data handling, and deduplication across overlapping runs.

Real-time

How it works: Event-driven ingestion with online inference from an endpoint and an onlinefeature cache.

Strengths: Second-level decisions (fraud, pricing, in-session personalization).

Trade-offs: Highest complexity, strict contracts and observability, careful cost control.

Orchestrator role: Triggers model updates based on drift signals, manages canary deployments, and coordinates fallback logic.

Quick chooser

If your SLA is...Choose...Because...
Hours or longerBatchSimplest to operate, cheapest at scale
MinutesMicro-batchNear-real-time without streaming complexity
SecondsReal-timeOnly when delayed decisions have measurable cost

Pragmatic path: start in batch, move critical steps to micro-batch, and reserve real-time for the narrow slice of decisions that truly need it. Consider creating real-time data pipelines for guidance on the transition.

Pattern fit by scenario

Retail restocking (micro-batch first): A regional retailer updates store stock-keeping unit (SKU) replenishment lists as sales come in. Trigger: point-of-sale (POS) deltas every five minutes. Cadence: micro-batch feature refresh, nightly model retrain. Safeguard: weekly approval gates for new model versions. KPI: stockout rate reduction.

Business-to-business (B2B) lead routing (micro-batch to selective real-time): Marketing routes hot leads to sales within minutes. Trigger: customer relationship management (CRM) webhook on new lead. Cadence: two-minute enrichment batches, real-time scoring for chat escalation only. Safeguard: 95th-percentile latency service-level objective (SLO) on the real-time path. KPI: lead response time.

Fraud detection (real-time with batch backstops): Payments team needs near-instant decisions. Trigger: transaction event stream. Cadence: real-time scoring, nightly batch for label reconciliation and retraining. Safeguard: circuit breakers, dead-letter queues, canary deployments. KPI: fraud loss rate.

Comparing ML orchestration tools

Most teams use a general tool for managing data and workflows with machine learning-specific components (for tracking experiments, maintaining registries, and deployment) and a central data storage system in an enterprise data warehouse. The right combination depends on your team's size, existing infrastructure, and whether you need general workflow orchestration or ML-specific capabilities.

The following table compares major orchestration tools across key dimensions:

ToolTypeStrengthsConsiderationsBest fit
Apache AirflowGeneral-purposeMature ecosystem, extensive integrations, large communitySteeper learning curve, requires infrastructure managementTeams with dedicated platform engineers and complex DAG requirements
PrefectGeneral-purposePython-native, modern API, hybrid execution modelNewer ecosystem, fewer pre-built integrations than AirflowPython-first teams wanting simpler setup than Airflow
DagsterGeneral-purpose with data focusStrong data asset concepts, built-in testing, observabilitySmaller community than AirflowTeams prioritizing data lineage and software engineering practices
Kubeflow PipelinesML-specificNative Kubernetes, end-to-end ML workflow supportRequires Kubernetes expertise, complex setupOrganizations already running Kubernetes at scale
SageMaker PipelinesML-specific (AWS)Tight AWS integration, managed infrastructureAWS lock-in, less flexibility outside AWS ecosystemAWS-native teams wanting minimal infrastructure management
KedroML-specificModular pipeline design, strong reproducibility, integrates with multiple orchestratorsRequires pairing with an orchestrator for schedulingTeams prioritizing clean code structure and experiment reproducibility
MetaflowML-specificNetflix-proven, simple Python API, built-in versioningLess mature ecosystem than Airflow, AWS-oriented defaultsData science teams wanting minimal infrastructure overhead
FlyteML-specificStrong typing, Kubernetes-native, built-in caching and versioningSmaller community, steeper learning curveTeams needing strict reproducibility and type safety

Scenario-based selection

The following recommendations match common team situations to tool choices:

ScenarioRecommendedWhyAvoid
3-person team on AWS, first production pipelineMetaflow or SageMaker PipelinesMinimal infrastructure overhead, quick startKubeflow (too complex for team size)
Platform team on Kubernetes, multiple ML teamsKubeflow Pipelines or FlyteNative Kubernetes, multi-tenant supportPrefect Cloud (adds another control plane)
Regulated enterprise, audit requirementsDagster or Airflow with lineage pluginsStrong lineage, mature governance patternsTools without OpenLineage support
Streaming-heavy, event-driven architecturePrefect or DagsterBetter event-trigger support than AirflowBatch-oriented tools without event handling
Data science team, notebook-heavy workflowKedro + Airflow or MetaflowClean notebook-to-production pathPure infrastructure tools without DS ergonomics

When to migrate

Consider moving from simpler tooling when you hit these triggers:

  • Move from cron scripts to Airflow or Prefect when you have more than 10 cross-dependent jobs or need backfill capabilities
  • Move from Airflow to Kubeflow when Kubernetes is already your deployment target and you need tighter ML-specific integrations
  • Move from single-tool to hybrid (orchestrator plus MLflow) when experiment tracking and model registry become bottlenecks

Selection criteria

When evaluating tools, weigh these factors against your team's situation:

  • Team size and expertise: Smaller teams often benefit from managed services or simpler tools like Prefect; larger teams can justify the overhead of Airflow or Kubeflow.
  • Cloud preference: If you're committed to a single cloud provider, native tools (SageMaker for Amazon Web Services (AWS), Vertex AI Pipelines for Google Cloud Platform (GCP)) reduce integration friction.
  • ML-specific vs general-purpose: General-purpose orchestrators handle broader workloads but may require additional ML tooling; ML-specific platforms bundle more capabilities but may not fit non-ML workflows.
  • Existing infrastructure: Teams already running Kubernetes may find Kubeflow a natural fit; teams without container orchestration expertise should consider managed cloud orchestration platforms.
  • Statefulness and retries: ML jobs often run for hours and need checkpointing. Evaluate how each tool handles long-running tasks, idempotency, and automatic retries.
  • Metadata and lineage: If auditability matters (and it usually does), prioritize tools that track which data, code, and parameters produced each artifact.
  • Graphics processing unit (GPU) scheduling: Training workloads may need gang scheduling, node affinity, or preemption policies. Not all orchestrators handle GPU allocation gracefully.

Orchestration building blocks

The core components that make orchestration work include:

  • Scheduling and triggers: Combine scheduled tasks and events; support retries and backfills.
  • Dependencies and data contracts: Define readiness (columns, types, ranges) and block on violations.Vague contracts are worse than none. If your contract does not specify acceptable null rates (e.g., less than 2 percent) or value ranges, it won't catch the data quality issues that actually break models.
  • Packaging and secrets: Containerize steps, version artifacts, and securely manage credentials.
  • Observability and service-level objectives (SLOs): Monitor logs, metrics, and lineage with clear goals for freshness, accuracy, latency, and cost.
  • Governance and security: Manage approvals, version updates, and rollbacks; align with AI data governance and data governance benefits.

Reliability engineering: making pipelines safe to re-run

Reliability is where orchestration earns its keep. A pipeline that works once is a demo; a pipeline that works every time is production infrastructure.

Core concepts

Understanding these terms helps teams communicate about reliability:

  • Idempotent task: A task that produces the same output whether run once or multiple times with the same inputs. Safe to retry without side effects.
  • Backfill: Re-running a pipeline for historical time periods, typically to recover from failures or incorporate late-arriving data.
  • Checkpoint: A saved intermediate state that allows a long-running job to resume from where it left off rather than starting over.
  • Deterministic re-run: The ability to reproduce any historical pipeline execution with identical results given the same inputs.

Patterns and anti-patterns

The following table captures common reliability decisions and their consequences:

PatternAnti-patternWhy it matters
Write to partitioned, immutable paths (e.g., /data/2026/01/15/)Overwrite a "latest.csv" fileImmutable paths enable safe backfills and auditing
Pin data snapshot IDs in pipeline metadataQuery "current" tables without versioningPinned snapshots ensure reproducibility
Retry with exponential backoff and a cap (e.g., 3 retries, max 1 hour)Unbounded retries that run foreverCapped retries prevent runaway costs and resource exhaustion
Store code SHA, data version, and parameters per runRely on "whatever was in main"Full metadata enables deterministic re-runs
Use partition-based backfills with concurrency limitsBackfill entire history in one jobSmaller windows reduce blast radius and resource contention

Backfill playbook

When backfilling historical data, follow these steps:

  1. Select the window: Identify the date range that needs reprocessing. Start with the smallest window that addresses the issue.
  2. Check for overlap: Verify that no in-flight runs cover the same partitions. Overlapping runs cause data corruption.
  3. Set concurrency limits: Run backfill jobs with lower parallelism than normal to avoid starving production workloads.
  4. Validate incrementally: Check outputs after each partition before proceeding to the next.
  5. Update downstream consumers: Notify systems that depend on the backfilled data so they can refresh their caches.

Reproducibility requirements

To re-run any model within 30 minutes (a reasonable operational standard), each pipeline run must capture:

  • Code commit hash (SHA): The exact commit that produced the run
  • Data snapshot ID: Version identifiers for all input datasets
  • Environment hash: Container image digest or environment specification
  • Random seed: For any stochastic operations
  • Hyperparameters: All configuration values used in training
  • Upstream run IDs: Links to the pipeline runs that produced input artifacts

Training checkpoints for long GPU jobs

For training jobs that run for hours, checkpoint frequency involves a trade-off. Checkpointing every 10 minutes adds storage cost and I/O overhead but limits lost work to 10 minutes on failure. Checkpointing every hour reduces overhead but risks losing significant progress. A reasonable default: checkpoint every 30 minutes for jobs under four hours, every 15 minutes for longer jobs.

Evaluation orchestration and promotion gates

Evaluation gates are the quality control checkpoint that prevents bad models from reaching production. Without them, a model that performs worse than the current champion can silently replace it.

The evaluation DAG

A complete evaluation workflow includes these stages:

  1. Eval dataset selection: Choose a held-out dataset that represents production traffic. Version it alongside the model.
  2. Metric computation: Calculate primary metrics (accuracy, precision, recall, F1) and business metrics (revenue impact, false positive cost).
  3. Slice checks: Evaluate performance across customer segments, geographies, or other meaningful subgroups. A model that improves overall but degrades for a key segment may not be acceptable.
  4. Regression vs baseline: Compare against the current champion model, not just absolute thresholds.
  5. Fairness and bias checks: Verify that performance is equitable across protected groups where applicable.
  6. Approval gate: Block or allow promotion based on all preceding checks.
  7. Registry promotion: Move the approved model to the production stage in the model registry.

Sample gate policy

The following table shows how to structure promotion criteria:

MetricThreshold typeValueAction on failure
Overall accuracyRelative to championWithin 1%Block promotion
Worst-slice accuracyAbsoluteAbove 85%Block promotion
99th-percentile latencyAbsoluteUnder 200 msBlock promotion
Memory usageAbsoluteUnder 4GBWarn, allow with approval
Fairness gapAbsoluteUnder 5% difference across groupsBlock promotion

Statistical rigor

Not every metric improvement is meaningful. When comparing a new model to the champion:

  • Use confidence intervals rather than point estimates
  • Require a minimum evaluation set size (typically thousands of examples) before trusting small differences
  • Consider whether a 0.1 percent accuracy improvement justifies the operational cost of deployment

Deployment patterns

After passing evaluation gates, models move to production through controlled rollouts:

  • Champion/challenger: Run the new model alongside the current champion, comparing predictions without affecting people
  • Canary deployment: Route a small percentage of traffic (e.g., five percent) to the new model and monitor for issues
  • Shadow deployment: Run the new model on production traffic but discard its predictions, useful for latency and error testing

Human-in-the-loop approval

Automated gates handle routine promotions, but some changes warrant human review:

  • New data sources added to the pipeline
  • Architecture changes (new model type, different feature set)
  • Material metric shifts (more than five percent improvement or any degradation)
  • Changes affecting regulated domains or sensitive customer segments

The goal is specific approval triggers, not blanket review requirements that slow everything down.

Governance and risk management without slowing teams down

Think of governance as the guardrails that make changes safe and keep customer data private. Good guardrails are built into the workflow, so teams move with less friction, not more.

Security controls checklist

Production orchestration systems should implement these controls:

  • Secrets manager integration: No credentials in DAG code; use a secrets manager (HashiCorp Vault, AWS Secrets Manager, or similar)
  • Short-lived credentials: Use OpenID Connect (OIDC) or similar for temporary tokens rather than long-lived API keys
  • Least-privilege service accounts: Each task runs with only the permissions it needs, not a shared admin account
  • Data-access boundaries: Tasks inherit access controls from the data they touch; orchestrator cannot bypass data governance
  • Artifact signing: Model artifacts are signed and verified before deployment to prevent tampering
  • Audit log retention: All pipeline runs, approvals, and promotions logged with timestamps and actor identity

Role-based access model

A simple three-role model covers most teams:

  • Pipeline editors: Can modify DAG definitions and task code. Typically ML engineers and data engineers.
  • Run triggers: Can start pipeline runs and backfills but cannot change definitions. Typically data scientists and analysts.
  • Model promoters: Can approve models for production deployment. Typically senior engineers or ML leads.

Approval gate triggers

Rather than requiring approval for every change, trigger human review when:

  • A new data source is added to the pipeline
  • Model architecture changes (not just hyperparameter tuning)
  • Evaluation metrics shift by more than a defined threshold
  • The model will be deployed to a new environment or customer segment

Audit trail requirements

Each pipeline run should capture:

  • Data version: Which snapshot of each input dataset was used
  • Code SHA: The exact commit that produced the run
  • Parameters: All configuration values
  • Approver: Who approved promotion (if applicable)
  • Timestamp: When each stage completed

This information supports compliance requirements in regulated industries and enables debugging when issues arise.

Observability and continuous monitoring

Strong data observability tells you that everything is healthy,and what to fix when it is not. Aim for a small set of clear signals and clear responses.

What to watch

Start with these signals so you can catch issues early:

  • Data health: Is data late or missing? Did a column change in a way that breaks things? Are values way outside normal ranges?
  • Model health: How accurate are the results? Are some customer groups doing much better or worse than others? How often does the system fall back to a simple rule?
  • Change over time (drift): Has the incoming data changed, or has the relationship between inputs and outcomes shifted?
  • Speed and cost: How long do runs take? How often do they fail? How much do you spend per run or per 1,000 predictions?
  • Business impact: Are key performance indicators (KPIs) like conversion, revenue, and cases sent for review moving the right way,and which model version is live?

Set simple targets

Pick targets (sometimes called service-level objectives, or "SLOs") for freshness, speed, accuracy, and cost. Example: "Daily data ready by 2 am", "Dashboard updates within five minutes", "Prediction service responds under 300 ms", "Stay under $X per 1,000 predictions."

Trigger catalog

The following table connects monitoring signals to orchestrator actions:

SignalExample thresholdAutomated actionHuman approval required?Owner
Data freshness breachMore than 2 hours lateBlock downstream tasks, alert on-callNoData engineering
Population stability index (PSI) driftPopulation stability index (PSI) greater than 0.2Queue retraining jobYes, if drift persists 3 daysML engineering
Performance decayAccuracy drops more than 3% week-over-weekQueue retraining job, alert ML leadYesML engineering
Schema changeNew or removed columnsQuarantine batch, block pipelineYesData engineering
Cost overrunRun exceeds 150% of budgetTerminate job, alert financeNoPlatform engineering
Latency spikep99 exceeds 500ms for 10 minutesScale up resources, alert on-callNoPlatform engineering

Alert-fatigue controls

Too many alerts cause teams to ignore warnings. Manage alert volume with:

  • Deduplication windows: Don't fire the same alert repeatedly within a 30-minute window
  • Severity tiers: Page on-call for critical issues; send Slack for warnings; log informational alerts
  • Symptom-based alerts: Alert on outcomes people would notice (late data, failed runs, slow responses) rather than every internal metric fluctuation

Retraining decision rules

Drift alone doesn't always justify retraining. A model can drift without degrading if the relationship between features and outcomes remains stable. Retrain when:

  • Drift exceeds threshold AND performance metrics decline
  • Drift persists for multiple measurement periods (not just a single spike)
  • Business context changes (new product launch, seasonal shift) that the current model cannot capture

Avoid retraining on noise. Frequent unnecessary retraining wastes GPU budget and introduces deployment risk.

Scheduling patterns

Most pipelines combine multiple trigger types:

  • Cron-based schedules work well for batch jobs with predictable timing (nightly retraining, hourly feature refreshes).
  • Data-availability triggers fire when upstream datasets land, preventing jobs from running on stale or missing inputs.
  • Event-driven triggers respond to external signals (new file in storage, webhook from a source system, drift threshold crossed).

For each pattern, define what happens when a trigger fires but the previous run hasn't finished. Options include queuing, skipping, or failing loudly.

Monitoring checklist

A production-ready pipeline tracks these signals at minimum:

  • Pipeline latency: How long from trigger to completion? Track p50, p90, and p99.
  • Success rate: What percentage of runs complete without error? Set a floor (e.g., 95 percent) and alert below it.
  • Queue time: How long do jobs wait before starting? Long queues signal resource contention.
  • Resource utilization: GPU and CPU usage during training; memory pressure during feature computation.
  • Cost per run: Track compute spend per pipeline execution and per 1,000 predictions.
  • Data drift signals: Statistical distance between training and serving distributions (PSI, KL divergence, or simpler threshold checks).

Incident playbooks

When alerts fire, responders need a clear path forward. Keep short playbooks for common scenarios:

  • Stuck run: Check for resource exhaustion, upstream dependency failures, or deadlocks. If the job has been running longer than twice its normal duration, consider terminating and restarting.
  • Data drift spike: Investigate whether the drift reflects a genuine change in the world (seasonal shift, new customer segment) or a data quality issue (schema change, missing values). Pause promotion until root cause is clear.
  • Failed backfill: Verify that historical data is available and that the backfill window does not overlap with in-progress runs. Retry with smaller windows if resource limits are the issue.
  • Resource exhaustion: Scale up temporarily or reduce concurrency. For GPU jobs, check whether other workloads are competing for the same node pool.

Sample service-level indicators (SLIs) and service-level objectives (SLOs)

Service-level indicators (SLIs) measure what matters; SLOs set the bar. Here are examples for common pipeline types:

Pipeline typeSLISLO
Nightly batch trainingTime from trigger to model registeredLess than 4 hours, 95 percent of runs
Micro-batch feature refreshLatency from source event to feature store updateLess than 5 minutes, 99 percent of events
Real-time inferenceEnd-to-end latency (request to response)Less than 200 ms at p99
All pipelinesRun success rateGreater than 98 percent over rolling 7 days

What to focus on by pattern

Each pattern needs a different monitoring focus:

  • Batch: Watch that each day's data arrives and is complete, that backfills are correct, and that outputs look reasonable. Alert before the business SLA so there's time to fix things.
  • Micro-batch: Keep an eye on short delays between windows, duplicates, and API rate limits. Make sure "real-time BI" tiles match the pipeline's latest timestamp.
  • Real-time: Track queue size, slow responses, and error rates. Add "circuit breakers" (temporarily switch to a safe fallback) and hold bad events to replay later instead of losing them.

Cost and performance orchestration

Cost management is often an afterthought until the bill arrives. Building cost awareness into orchestration prevents surprises and helps justify continued investment.

Cost levers

The following table summarizes the main ways to reduce pipeline costs:

LeverSavings mechanismRisk or dependency
Spot or preemptible instances60-80% discount on computeRequires checkpointing; jobs may be interrupted
Step-level cachingSkip unchanged upstream stepsCache invalidation bugs can cause stale outputs
Intelligent retraining triggersRetrain on drift, not scheduleMay miss gradual degradation if thresholds are too loose
Right-sized resourcesMatch instance type to workloadUnder-provisioning causes failures; over-provisioning wastes money
Retry capsPrevent runaway retriesMay abandon recoverable jobs too early
Batch non-urgent workRun during off-peak hoursIncreases latency for affected workloads

Cost policies

Set explicit policies to prevent budget overruns:

  • Maximum cost per run: Terminate jobs that exceed 150 percent of expected cost
  • Per-team GPU quota: Allocate GPU hours by team to prevent one project from starving others
  • Alert at 80 percent of budget: Warn before hitting limits, not after
  • Weekly cost review: Track cost-per-1,000-predictions and cost-per-training-run as standard metrics

Retraining frequency trade-offs

Daily retraining on barely-changed data wastes GPU budget. Weekly retraining on rapidly-shifting data causes accuracy decay. The right frequency depends on how fast the underlying data distribution changes, how sensitive business outcomes are to model accuracy, and the cost of each retraining run.

Start with weekly retraining and adjust based on drift monitoring. If drift signals rarely trigger and accuracy remains stable, consider moving to bi-weekly. If drift spikes frequently, consider more frequent retraining or event-triggered retraining.

From models to outcomes: the last mile to business workflows

An orchestrated pipeline is only useful if it changes decisions. Close the loop by putting predictions where people already work: on the same data dashboards as core KPIs. Trigger simple actions automatically (create a ticket, send a discount, pause a risky transaction) and start with one action you trust before expanding. Notify the right people when thresholds are crossed or a new model goes live. Measure impact so you can prove value: revenue lift, fewer stockouts, shorter response times.

A starter blueprint for small teams

You don't need a big MLOps setup to see value. Start small. Ship.

Phase 0 (1-2 weeks): get the basics in place

Start with these setup steps:

  • Choose one use case tied to a clear key performance indicator (KPI).
  • List your data sources and owners; add simple checks so bad or late data does not flow downstream.
  • Pick an orchestration tool and set up version control (e.g., Git) plus a basic continuous delivery pipeline to run checks when code changes.
  • Decide your first pattern (often micro-batch) and how often to retrain.

Phase 1 (2-4 weeks): build the first end-to-end pipeline

These are the core steps for the first release:

  • Ingest and validate data; reuse features where you can.
  • Train the model and track experiments; agree on the bar it must beat to go live.
  • Package and deploy; log how fast it runs, how accurate it is, and how much it costs.
  • Deliver predictions to the dashboard your team already uses.

Minimum viable guardrails for this phase: idempotent tasks, one evaluation gate, one alert on pipeline failure, one documented rollback path.

How you'll know you're ready

Use this checklist to confirm the pipeline is ready to run in production:

  • Freshness: data lands on time (e.g., micro-batch is no more than five minutes behind).
  • Quality: key inputs pass basic checks; failures stop the pipeline instead of shipping bad outputs.
  • Performance: the model meets the accuracy level you agreed on; speed is within target.
  • Visibility: a health view shows freshness, basic accuracy, speed, and cost; alerts include links to what broke.
  • Last mile: at least one dashboard tile and one small automation use the predictions.
  • Safety: you can roll back quickly if the new version misbehaves.
  • Ownership: someone is on point when alerts fire, with a short runbook.

First 30-60 days: how to track value

Track these signs first so you can prove impact early:

  • Shorter response time to hot leads; fewer out-of-stocks; fewer chargebacks. Pick one leading indicator.
  • Less manual work (fewer handoffs), shorter recovery from failures, and shorter rollback time.
  • Spend stays within plan; track cost per 1,000 predictions.

Phase 2 (then iterate)

Harden and scale. Safer rollouts (canary or blue-green), a small feature store for reuse, simple drift checks, and basic cost tuning. Typical timelines vary by team size and complexity, but most teams complete Phase 2 within two to three months of finishing Phase 1.

Common pitfalls and how to avoid them

Even experienced teams hit the same issues. The following table summarizes common pitfalls and their fixes:

PitfallSymptomFix
Duplicate pipelines ("shadow IT")Different teams get different answers from the same dataShare templates and a common data dictionary; assign owners for core tables
Manual handoffsEmail approvals and "run this script" steps slow everything downPut approvals in your code process; turn notebook steps into versioned jobs
No easy rollbackRecovery is slow when a model slipsKeep versioned models and data; use canary or blue-green releases; practice rollbacks
Costs creeping upSurprise bills from frequent micro-batches or always-on servicesSet budgets and alerts; cache hot features; batch non-urgent work; right-size servers
Models going staleAccuracy fades as behavior changesSet drift checks and retrain rules; watch key customer segments
Training/serving skewModel sees different features in each placeShare feature definitions; test for parity between offline and online features; canary checks
Weak data contractsVague contracts fail to catch quality issuesSpecify acceptable null rates, value ranges, and schema expectations explicitly
Too many noisy alertsAlert fatigue causes teams to ignore warningsAlert on symptoms people would notice; tune thresholds based on actual incidents
Over-complex designs too earlyTeams build for scale they don't have yetStart with the simplest pattern that meets your SLA; add complexity only when proven necessary
Secrets in codeCredentials leak into version controlUse a secrets manager; scan commits for exposed credentials
Unclear ownershipNobody knows who to call when alerts fireAssign on-call owners; keep short runbooks for common issues
Too many external dependenciesPipelines break when third-party services changePin versions; add health checks; have fallback logic for critical dependencies

When a pipeline breaks, triage in this order: first, check whether the data is fresh and valid; second, check what changed last; third, look for drift or traffic-mix changes; fourth, review access and permissions; fifth, roll back or use a safe fallback; sixth, add a note to dashboards and open a ticket.

Bringing ML pipeline orchestration together

ML pipeline orchestration is the automated coordination of ML lifecycle stages with dependency management, scheduling, retries, and observability. The highest-impact actions for most teams: make tasks idempotent so retries are safe, add evaluation gates so bad models cannot reach production, and build a trigger catalog that connects monitoring signals to orchestrator actions.

Start small, automate the risky handoffs, and focus on the last mile. Dashboards, alerts, and simple automations that change day-to-day decisions. As you grow, tighten monitoring and governance and move only the high-value use cases to real-time.

Take the next step with Domo

Ready to turn orchestration into business results? Domo helps teams move from raw data to measurable outcomes through three connected layers:

  • Foundation: Connect and govern your data with modern pipelines and freshness checks, starting with cloud data integration. Make your data AI-ready so orchestration has a reliable base to build on.
  • Activation: Orchestrate your workflows with versioning, approvals, lineage, and alerts built in. Turn AI into action through agents and apps on governed data.
  • Distribution: Deliver outcomes into the workflows people already use, whether dashboards, automated actions, or AI assistants.

Start with one use case, one KPI, and one decision speed. If you'd like help scoping a focused pilot, talk to Domo about a path to scale.

See how Domo orchestrates ML pipelines end to end

Watch demo

Build reliable ML workflows with governance and alerts—fast

Try free
See Domo in action
Watch Demos
Start Domo for free
Free Trial

Frequently asked questions

No items found.
No items found.
Explore all
No items found.
AI & Data Science