What Is Streaming ETL? Architecture, Benefits, and Use Cases

Streaming extract, transform, load (ETL) processes data continuously as it arrives, cutting latency from hours to milliseconds and enabling immediate action on insights. This article covers the architecture behind streaming pipelines, compares them to batch and extract, load, transform (ELT) approaches, and walks through implementation challenges and industry use cases that show where continuous data processing delivers the most value.

Key takeaways

Here are the main points to keep in mind:

• Streaming ETL continuously extracts, transforms, and loads data as it is generated, enabling immediate insights rather than waiting for scheduled batch processing.
• The core components of streaming ETL include data ingestion, stream processing engines, transformation logic, and output destinations working together in a continuous pipeline.
• Organizations use streaming ETL for fraud detection, IoT monitoring, personalized customer experiences, and other scenarios where decisions made in seconds drive business value.
• Implementing streaming ETL requires addressing challenges like handling high-velocity data, ensuring data correctness, managing schema evolution, and integrating with existing systems.
• Platforms like Domo simplify streaming ETL by providing flexible data pipelines, automation, and integration across data sources.

What is streaming ETL?

Streaming ETL captures, transforms, and loads data in real time. That means milliseconds to seconds after events occur. Traditional ETL waits for scheduled batches. Streaming ETL doesn't wait.

The process follows three core steps:

• Extract: The pipeline captures data continuously from sources such as Internet of Things (IoT) devices, application programming interfaces (APIs), application logs, clickstream events, or database transaction logs using Change Data Capture (CDC).
• Transform: Raw data undergoes validation, filtering, masking, enrichment, and aggregation while still in motion.
• Load: Transformed data flows to its destination, whether a cloud data warehouse, analytics platform, search index, or alerting system.

Traditional ETL follows a batch processing model, where teams extract data at scheduled intervals, transform it, and then load it into a destination system. Effective for historical reporting. Best suited for historical reporting rather than immediate insights.

Streaming ETL, by contrast, continuously ingests and processes data. You can act on changes as they happen. This proves particularly useful when seconds matter: financial fraud detection, predictive maintenance in manufacturing, personalized customer experiences in retail. The result? Reduced latency, improved operational efficiency, and a more agile approach to data management.

How streaming ETL differs from traditional ETL

ETL isn't being replaced. It's being complemented by newer approaches that fit different use cases. The modern data stack includes several patterns working alongside each other:

• Batch ETL remains the right choice for periodic analytics reporting, compliance transformations, and curated dimensional models where data freshness measured in hours is acceptable.
• ELT pushes raw data into cloud warehouses or data lakes first, then transforms it using tools like dbt or Spark. This pattern works well when transformation logic is complex and the warehouse has the compute power to handle it.
• Streaming ETL handles scenarios where data must be transformed in flight and delivered within seconds, such as fraud detection, operational dashboards, or triggering automated actions.
• CDC-based replication captures database changes continuously and streams them downstream, often feeding into streaming ETL pipelines or directly into analytics systems.

Which combination fits your latency requirements, transformation complexity, and operational constraints? That's the real question.

Key components of streaming ETL architecture

A streaming ETL pipeline consists of several interconnected components that work together to move data from source to destination with minimal delay. Two primary ingestion patterns feed these pipelines: event-stream publishing (where applications emit events directly to a message broker) and Change Data Capture (where tools read database transaction logs to capture every insert, update, and delete as a continuous stream).

Data ingestion

Data ingestion is the entry point. Two distinct patterns dominate this layer:

Event-stream publishing involves applications, IoT sensors, or user interfaces emitting events directly to a message broker. Clickstream data from a website, sensor readings from manufacturing equipment, and transaction events from a payment gateway all follow this pattern. The application publishes events as they occur.

Change Data Capture reads from database transaction logs, such as PostgreSQL's write-ahead log or MySQL's binary log, to capture row-level changes without requiring application modifications. CDC proves particularly valuable when you need to stream data from existing transactional databases without adding instrumentation to application code.

Both patterns feed into message brokers like Apache Kafka, Amazon Kinesis, or Google Pub/Sub. These provide durable, scalable queuing and ensure smooth data transfer even during traffic spikes.

Change data capture as an ingestion pattern

CDC deserves special attention because it solves a common challenge: how do you stream data from an existing database without modifying the applications that write to it?

CDC tools connect directly to a database's transaction log and capture every row insertion, update, and deletion as a live event stream. This approach provides several advantages:

• No application changes required, since CDC reads from the database layer rather than requiring instrumentation in application code
• Full-fidelity change history, capturing not just current state but the sequence of changes over time
• Low overhead on the source database, as reading transaction logs is typically less resource-intensive than polling queries

When is CDC the right choice? When the data source is an online transaction processing (OLTP) database. When application-level event publishing is not feasible. When you need complete change history for audit or replay purposes. One important detail: if you enable CDC without configuring adequate log retention on the source database, you'll have gaps in your change stream. Logs get purged before the CDC tool reads them. Recovering from that is painful. Tools in this space include Debezium, AWS Database Migration Service, and Google Datastream.

Stream processing

Stream processing engines consume data from message brokers and apply transformations in real time. These engines handle the computational work of filtering, aggregating, joining, and enriching data while it flows through the pipeline.

Unlike batch processing (where you operate on complete datasets), stream processing works with unbounded data that arrives continuously. This requires different computational models, including windowing strategies to group events by time and state management to track information across events.

Common stream processing engines include Apache Flink, Apache Spark Structured Streaming, and cloud-native services like Amazon Kinesis Data Analytics and Google Cloud Dataflow.

Transformation and enrichment

Streaming transformations differ from batch transformations in important ways. Because data arrives continuously and unboundedly, you cannot simply scan an entire dataset. Instead, transformations must account for:

State management: Operations like counting unique users or calculating running averages require maintaining state across events. This state must be persisted and managed carefully, often with time-to-live settings to prevent unbounded growth. Teams underestimate how quickly state can grow. Without TTL policies, a pipeline running for weeks can exhaust memory or storage, causing silent failures or degraded performance.

Window types: Tumbling windows divide time into fixed, non-overlapping intervals (every 5 minutes). Sliding windows overlap and update more frequently. Session windows group events by activity periods with gaps between them. The right window type affects both aggregation semantics and resource requirements.

Temporal joins: Joining a stream of events against reference data (like enriching transactions with customer information) requires strategies for handling updates to that reference data and deciding how long to wait for matching records.

Late and out-of-order events: Events don't always arrive in the order they occurred. Watermarks track the progress of event time through the system and define how long to wait for late-arriving data before finalizing results.

Deduplication: At-least-once delivery semantics mean duplicate events can occur. Deduplication logic using unique event identifiers and state with TTL settings prevents duplicates from affecting downstream results.

Output and storage

The final component delivers transformed data to its destination. The choice depends on how the data will be consumed:

Real-time online analytical processing (OLAP) stores like ClickHouse, Apache Druid, or Apache Pinot provide sub-second query latency for dashboards and interactive analytics. Ideal when people need to slice and dice data immediately after it arrives.

Cloud data warehouses like Snowflake, BigQuery, or Amazon Redshift can ingest streaming data through micro-batch loading or streaming ingestion APIs. This works well when you need to combine streaming data with historical data for analysis, though query latency is typically higher than dedicated OLAP stores.

Time-series databases like InfluxDB or TimescaleDB are optimized for sensor data, metrics, and other time-stamped measurements where queries focus on recent time ranges and downsampling.

How streaming ETL works

The streaming ETL process follows a structured sequence of steps, ensuring that data is continuously extracted, transformed, and loaded with minimal delay. Here's how each stage unfolds:

1. Data extraction: The process begins with capturing data from sources such as IoT devices, APIs, application logs, and event streams. Unlike batch processing, where data is extracted at scheduled intervals, streaming ETL continuously pulls data as it is generated. For database sources, CDC tools read transaction logs to capture changes within milliseconds of when they occur.
2. Data ingestion: Once extracted, the data flows into a message broker like Apache Kafka or AWS Kinesis. These tools ensure smooth data transfer and provide fault tolerance by managing high-velocity data streams. Message brokers also enable replay capabilities, allowing you to reprocess historical events when needed.
3. Data transformation: The raw data undergoes transformations within milliseconds to seconds of arrival. These transformations may include validation to ensure accuracy, filtering to remove irrelevant or redundant information, aggregation using windowed computations to consolidate multiple data points, enrichment by joining with reference data for enhanced insights, and deduplication to handle duplicate events from at-least-once delivery.
4. Data loading: The pipeline then delivers the transformed data to its destination system within seconds of the original event. Destinations could include a cloud data warehouse for analytics, a real-time OLAP store for dashboards, or an alerting system that triggers automated responses based on streaming analytics.
5. Continuous monitoring and optimization: The final step involves monitoring the ETL pipeline for errors, bottlenecks, or data inconsistencies. Key metrics include consumer lag (the delay between event generation and processing), watermark delay (how far behind event time the system is running), and error rates. Performance tuning and scaling adjustments ensure that the pipeline remains efficient and responsive to changes in data volume.

Streaming ETL architecture patterns

Two primary architecture patterns dominate, each with distinct tradeoffs.

Event-stream architecture involves applications publishing events directly to a message broker. A stream processor consumes, transforms, and routes those events to downstream systems. This pattern excels when you need to enrich events with external data in flight, when multiple consumers need the same event stream, or when the source systems are already instrumented to emit events.

CDC-based architecture uses tools that read database transaction logs to capture every row change as a live event stream. This pattern is preferred when you need guaranteed database state consistency, when modifying source applications is not feasible, or when you need complete change history for audit purposes.

The following decision framework helps choose between patterns:

Many production systems use a hybrid approach. They combine CDC from transactional databases with event streams from user-facing applications, then join these streams in a processing engine to build comprehensive views.

The core components across both patterns include:

Data sources: IoT sensors, application logs, financial transactions, and cloud services that generate data needing processing.

Message brokers: Tools like Apache Kafka, Amazon Kinesis, or Google Pub/Sub that capture and queue data, ensuring smooth and scalable ingestion into the pipeline.

Processing engines: Technologies like Apache Flink or Spark Structured Streaming that transform data in motion, filtering, aggregating, and enriching it before delivery.

Storage solutions: Cloud-based data warehouses such as Snowflake or Amazon Redshift that store transformed data, making it accessible for analytics and reporting.

Analytics and visualization: Data platforms like Domo provide dashboards and alerts, enabling immediate action on streaming data insights.

Event-driven architecture considerations

Event-driven pipelines introduce operational considerations that batch systems don't face. Schema evolution is among the most significant.

As business requirements change, event schemas evolve. A new field gets added. A field type changes. A field becomes deprecated. Without careful management, schema changes break downstream consumers.

Schema registry patterns address this challenge by maintaining a central repository of schemas and enforcing compatibility rules. Backward compatibility ensures old consumers can read new events. Forward compatibility ensures new consumers can read old events. Full compatibility provides both guarantees.

Formats like Avro and Protobuf support schema evolution natively, encoding schema information with the data and enabling automatic compatibility checking. Implementing a schema registry early in your streaming ETL journey prevents painful migrations later.

Batch ETL vs streaming ETL

Batch ETL remains useful for historical data analysis. Streaming ETL, however, is becoming the standard for organizations needing immediate insights. Batch ETL operates on a scheduled basis, meaning data is processed in intervals ranging from minutes to hours. This creates delays in decision-making, as insights are only available after the batch process completes.

Streaming ETL processes data continuously, significantly reducing latency. Businesses can act on the latest data without waiting for scheduled updates. A retail company using batch ETL might only update inventory levels once a day, whereas a company using streaming ETL can adjust stock levels based on live sales data.

A practical decision rubric helps determine which approach fits your use case:

• If your required data freshness service-level agreement (SLA) is under five minutes, streaming ETL is likely the appropriate choice.
• If transformations are compute-heavy and time-sensitivity is low (hourly or daily updates are acceptable), batch ETL or ELT is more cost-effective.
• If data volume is highly variable with unpredictable spikes, streaming with auto-scaling handles this more effectively than scheduled batch jobs.
• If you need both immediate operational insights and deep historical analysis, a hybrid approach using streaming for dashboards and batch for complex analytics often works best.

Micro-batching represents a middle ground between true streaming and traditional batch processing. Systems like Spark Structured Streaming process small batches at short intervals, typically 30 to 120 seconds. This approach provides near-real-time latency with simpler exactly-once semantics than pure streaming, though it can't match the sub-second latency of true stream processing.

ETL vs ELT vs streaming ETL

The terminology around data integration patterns can be confusing. Here's a clear taxonomy:

Streaming ETL transforms data in flight, before it lands in storage. Data is validated, filtered, enriched, and aggregated as it flows through the pipeline. The destination receives clean, transformed data ready for consumption. This pattern is ideal when transformations are straightforward, when you need to mask personally identifiable information (PII) before data reaches storage, or when downstream systems expect a specific format.

ELT (extract, load, transform) ingests raw data first, then transforms it at rest in the data warehouse or data lake. Tools like dbt or warehouse-native SQL handle transformations after data lands. This pattern works well when transformations are complex, when you want to preserve raw data for flexibility, or when your warehouse has abundant compute capacity.

Streaming ELT combines elements of both: raw event streams are ingested to a staging layer with minimal transformation, then downstream processes (often scheduled or triggered) transform the data for consumption.

Benefits of streaming ETL

Streaming ETL provides several advantages that make it an essential component of modern data strategies. By processing data as soon as it arrives, organizations can use analytics and make decisions sooner.

Real-time analytics: Streaming ETL ensures insights are always current. Particularly useful in supply chain logistics, where adjustments made in seconds can prevent bottlenecks and optimize operations.

Consistent data integrity: Continuous monitoring and processing help identify and correct inconsistencies as they occur. This reduces errors and ensures organizations always have clean, reliable data for decision-making.

Adaptability to data volume: Streaming ETL platforms can scale horizontally, distributing workloads efficiently to handle large volumes of data. Some systems also use in-memory processing to manage data surges without overwhelming storage infrastructure.

Cost efficiency: Because streaming ETL processes each data event as it happens, organizations can avoid the high costs associated with running frequent batch operations on large-scale infrastructure. Compute resources are used continuously rather than in expensive periodic spikes.

Integration across platforms: Streaming ETL can ingest and process data from multiple sources, including cloud platforms, IoT devices, and traditional databases. Consolidating data from different systems into a unified view becomes much easier.

Deeper insights: Streaming ETL enables data enrichment by integrating incoming data with historical records or external sources. This supports predictive analytics, anomaly detection, and trend forecasting.

Streaming ETL tools and technologies

The streaming ETL landscape includes tools across several pipeline layers.

Ingestion and message brokers handle the capture and queuing of streaming data. Look for tools that offer durable message storage, horizontal scalability, and replay capabilities. Apache Kafka dominates this layer, with cloud-native alternatives including Amazon Kinesis, Google Pub/Sub, and Azure Event Hubs.

Stream processing engines perform the transformation work. Key selection criteria include support for event-time processing, state management capabilities, exactly-once semantics, and SQL interfaces for accessibility. Apache Flink provides the most complete feature set for complex streaming workloads. Apache Spark Structured Streaming offers a unified batch and streaming API. Cloud-managed options include Amazon Kinesis Data Analytics, Google Cloud Dataflow, and Azure Stream Analytics.

Storage and serving destinations receive transformed data. The choice depends on query latency requirements and consumption patterns. Cloud data warehouses like Snowflake, BigQuery, and Amazon Redshift handle analytical queries with moderate latency. Real-time OLAP stores like ClickHouse, Apache Druid, and Apache Pinot provide sub-second query performance for dashboards.

Orchestration and observability tools manage pipeline operations. Apache Airflow, Prefect, and Dagster handle workflow orchestration. Monitoring stacks built on Prometheus, Datadog, or cloud-native tools track pipeline health metrics.

Domo connects to streaming pipeline outputs and transforms them into business-ready analytics and dashboards. Rather than replacing your streaming infrastructure, Domo provides the visualization, alerting, and collaboration layer that makes streaming data actionable for people across the organization.

Challenges of implementing streaming ETL

Streaming ETL introduces operational complexity that batch processing doesn't face.

Handling high-velocity data requires infrastructure that can scale with demand. Message brokers must handle throughput spikes without dropping events. Processing engines need sufficient parallelism to keep up with ingestion rates. Backpressure mechanisms prevent fast producers from overwhelming slow consumers.

Ensuring data consistency across distributed systems is more complex than in batch processing. Network partitions, node failures, and message redelivery can all introduce inconsistencies that require careful handling.

Integrating with existing systems often means bridging different data formats, schemas, and protocols. Legacy databases may not support CDC natively. Downstream systems may expect batch-style updates rather than continuous streams.

Managing schema evolution requires planning from the start. As business requirements change, event schemas evolve. Without schema management practices, changes can break downstream consumers and require coordinated deployments.

Monitoring and observability become critical when data flows continuously. You need visibility into consumer lag, end-to-end latency, watermark delay, and error rates. Alerting thresholds must be tuned to distinguish normal variation from actual problems.

Data correctness and exactly-once semantics

Data correctness is a first-class design concern in streaming ETL. Unlike batch processing, where you can rerun a job from scratch, streaming systems must handle failures gracefully while maintaining accurate results.

Delivery semantics define how the system handles failures and retries:

At-least-once delivery guarantees that every event is processed at least once, but duplicates may occur. This is the default for most streaming systems because it is simpler to implement. Downstream systems must handle duplicates through deduplication logic.

Exactly-once delivery guarantees that every event is processed exactly once, even in the presence of failures. Achieving this requires coordination between the processing engine and the output destination, typically through distributed transactions or idempotent writes. You'll notice that many teams assume their processing engine provides exactly-once guarantees end-to-end, but these guarantees often only apply within the engine itself. Writes to external systems may still produce duplicates unless you implement idempotent output logic.

Practical strategies for ensuring correctness include:

Idempotent writes: Design your output operations so that processing the same event multiple times produces the same result. Using a unique event identifier as a primary key causes duplicate writes to update rather than insert.

Deduplication keys: Maintain state tracking which event identifiers have been processed within a time window. Reject events with identifiers already seen.

Watermarking for late events: Configure how long the system waits for late-arriving events before finalizing window results. Events arriving after the watermark may be dropped or routed to a late-data handler.

Reconciliation jobs: Run periodic batch jobs that compare streaming results against source-of-truth data to catch and correct any discrepancies.

For a payment processing pipeline, these strategies might combine: use exactly-once semantics where supported, implement idempotent writes using transaction IDs as primary keys, and run nightly reconciliation against the source database to verify accuracy.

Streaming ETL use cases by industry

Streaming ETL can have a significant impact on companies across different industries, boosting how they use data to benefit their business.

Manufacturing

Traeger Grills, renowned for pioneering wood pellet grill technology, faced challenges with fragmented data across various systems, hindering decision-making. By integrating Domo's platform, Traeger consolidated data from sources like Salesforce and Google Analytics into a unified view. This transformation enabled executives to access up-to-date insights instantly, facilitating proactive management and rapid response to emerging trends.

For other companies in the manufacturing sector, streaming ETL can further enhance such transformations by continuously processing data from production lines, supply chains, and quality control systems. Manufacturers can promptly address equipment malfunctions, optimize inventory levels, and improve product quality.

Healthcare

GE Healthcare, a leader in medical technology and diagnostics, sought to improve data sharing and analytics across its finance division. By adopting Domo, GE Healthcare established a single source of truth, enabling rapid data sharing and reporting on a global scale. This shift empowered people across finance and operations to use data for strategic decision-making.

Implementing streaming ETL in healthcare can further enhance such initiatives by enabling continuous processing of patient monitoring data, electronic health records, and medical device outputs. Early detection of patient deterioration. Efficient management of healthcare resources. Improved patient outcomes.

Professional services

CAE USA, a prominent aviation training company, sought to increase data awareness and operational efficiency. Utilizing Domo, CAE USA integrated data across departments, enhancing project management, training programs, and data security. The platform's capabilities shortened ETL processing time, reducing data flow times from hours to minutes and enabling timely insights and decision-making.

In the professional services industry, streaming ETL can further optimize operations by continuously integrating data from client interactions, project management tools, and financial systems. This supports immediate resource allocation, rapid response to client needs, and dynamic project adjustments.

Additional industries

Finance: Fraud detection systems analyze transactional data as it happens, identifying anomalies and blocking suspicious activities before they escalate. High-frequency trading algorithms use streaming ETL to analyze market fluctuations and execute trades within milliseconds.

E-commerce: Customer interactions are analyzed instantly to provide personalized recommendations, optimize pricing, and adjust inventory based on purchasing patterns.

Internet of Things: Smart cities, connected vehicles, and industrial automation rely on streaming ETL to process sensor data and trigger automated actions based on thresholds or patterns.

Precision agriculture: IoT sensors in farming continuously monitor soil moisture and weather conditions, automating irrigation for maximum efficiency and crop yield.

Getting started with streaming ETL

Getting started with streaming ETL can be done even without extensive data expertise or IT know-how. You need to understand how you want to use your data and make sure you get the right tools in place.

Before selecting tools or architecture, assess whether streaming ETL fits your use case. A simple decision prompt: if your required data freshness SLA is under five minutes and you need to act on data as it arrives, streaming ETL is worth evaluating. If hourly or daily updates are acceptable and your transformations are complex, batch ETL or ELT may be more cost-effective.

Your company can get started by:

• Assessing your data needs and identifying key use cases where immediate insights drive business value
• Choosing technologies that support streaming ETL, like Domo's flexible data platform
• Designing a scalable ETL architecture that integrates with existing data systems, including data governance basics like schema contracts, PII handling, and data lineage
• Testing and deploying streaming ETL pipelines while continuously monitoring performance and optimizing workflows

If you're in the market for a streaming ETL solution, Domo is a solid option for teams that want an intuitive and scalable way to integrate processing into business operations. With powerful automation, flexible data pipelines, and integration across platforms, Domo empowers organizations to connect data from multiple sources, process and transform data instantly, and gain actionable insights through AI-driven analytics and visualization.

Ready to use streaming ETL for insights and decision-making? Contact Domo today to learn how the platform can help you integrate, process, and analyze data.