Batch vs streaming vs reverse ETL: a decision checklist
Most data teams pick their pipeline architecture based on what's trendy, not what the problem actually needs. Someone reads a blog post about streaming and suddenly every pipeline needs Kafka. Someone else gets burned by a late batch job and decides everything must be real-time. Neither instinct is right. The question isn't which pattern is best — it's which pattern matches your latency requirements, your data volume, and what happens downstream when the data lands.
Here's a working checklist for actually deciding, based on what breaks in production when teams get this wrong.
Start with the decision, not the data
Before picking a pipeline pattern, answer one question: what decision or action depends on this data, and how fast does that decision need to happen?
- If a human looks at a dashboard once a day to plan tomorrow's ad spend, batch is fine.
- If a system needs to suppress a customer from an ad audience the moment they purchase, you need something closer to real-time.
- If the data needs to land inside a CRM or ad platform so a rep or algorithm can act on it, you're talking about reverse ETL, not just data movement.
Write down the actual use case in one sentence before you touch infrastructure. "Marketing wants ROAS by channel every morning" is a different problem than "support needs real-time order status in Zendesk" is a different problem than "sales needs lead scores pushed into Salesforce within minutes of form fill." Each of those points to a different architecture, and conflating them is how teams end up with three overlapping systems doing the same job badly.
When batch is the right call
Batch processing — pulling data on a schedule, transforming it, loading it somewhere — is still the default for most analytics and reporting workloads. Don't apologize for using it.
- Latency tolerance is measured in hours, not seconds. Daily ROAS reports, weekly cohort analysis, monthly inventory reconciliation — none of this needs sub-minute freshness.
- Data volume is large but predictable. Nightly syncs of order history, product catalogs, or warehouse inventory counts are exactly what batch was built for.
- Your transformations are complex. Multi-step joins across orders, returns, ad spend, and inventory are easier to debug and test in a batch model where you can run the whole pipeline end to end and inspect every stage.
- Cost matters more than speed. Batch jobs are cheaper to run and easier to schedule around off-peak compute windows. Streaming infrastructure runs constantly whether or not there's meaningful data moving through it.
- Your team is small. Batch tools (dbt, Airflow, scheduled scripts) have a much lower operational burden than managing a streaming cluster. If you don't have someone who can debug a stuck Kafka consumer at 2am, don't build a system that needs one.
The failure mode with batch is pretending it's faster than it is. If your "daily" sync actually takes six hours to run and often fails partway through, you don't have a daily pipeline — you have an unreliable one. Fix reliability before you consider upgrading to streaming.
When streaming actually pays off
Streaming pipelines process data as events happen, not on a schedule. They're more expensive to build and run, so the bar for justifying them should be high.
- The action has to happen in near real-time. Fraud detection, inventory holds during flash sales, or triggering an abandoned cart flow within minutes — these need event-driven architecture because the value of the data decays fast.
- Volume is continuous, not batchable. High-frequency clickstream data or webhook events from a dozen marketplaces arriving all day don't compress well into a single nightly job.
- You need to react to sequences of events, not snapshots. Detecting that a customer viewed a product three times and abandoned checkout twice requires tracking state across events, which batch jobs handle awkwardly.
- Downstream systems expect events, not tables. If you're feeding a real-time personalization engine or a live ops dashboard that ops staff are watching during a product launch, streaming is the only pattern that fits.
The failure mode with streaming is building it when nobody actually needs the speed. If your "real-time" dashboard is checked twice a day, you paid for infrastructure complexity you're not using. Ask who looks at this data and how often before committing to an event-driven system. A lot of streaming projects get justified by hypothetical future use cases that never materialize, while the team pays the operational tax every day in the meantime.
When reverse ETL is the missing piece
Reverse ETL gets lumped in with batch and streaming conversations, but it solves a different problem: getting clean data out of your warehouse and back into the operational tools your team actually works in.
- The data already lives in your warehouse, correctly modeled. Reverse ETL only works well when you've done the transformation work upstream. If your warehouse data is messy, you're just piping garbage into Salesforce or Klaviyo faster.
- Business teams need warehouse-level logic in tools they already use. Customer lifetime value, churn risk scores, or true ROAS by SKU — calculated once in the warehouse — often need to show up in a CRM, ad platform, or support tool where non-technical teams make decisions.
- You're tired of logic living in two places. If marketing ops has built a parallel version of your customer segments directly inside an ad platform because nobody could get warehouse data into it, reverse ETL closes that gap and keeps one source of truth.
- The sync frequency needed is minutes to hours, not seconds. Most reverse ETL tools sync on a schedule or trigger — this is fine for updating audience lists or CRM fields, but it's not a substitute for true streaming if you need sub-second reactions.
The failure mode with reverse ETL is treating it as a fix for bad data modeling. It's a delivery mechanism, not a transformation layer. If your warehouse tables are wrong, reverse ETL will just distribute that wrongness into every tool your team touches, faster and more consistently than manual exports ever could.
Mixing patterns without creating chaos
Most real systems use all three. A typical ecommerce data stack might batch-load order and inventory data overnight, stream webhook events from marketplaces for near-real-time stock updates, and run reverse ETL to push customer segments into an email platform every hour. That's not architectural indecision — that's matching each data flow to its actual requirement.
The mistake is letting these patterns blur into each other without clear ownership. If three different syncs are writing to the same customer record on three different schedules, you'll get race conditions and conflicting data, and nobody will be able to explain why the numbers don't match. Document which pipeline owns which field, and make sure there's one source of truth per data type, even when multiple pipelines touch it.
The actual checklist
Before building anything, answer these in order:
- What decision or action depends on this data, and who makes it?
- How much latency can that decision tolerate — hours, minutes, or seconds?
- Is the data arriving as discrete events or as bulk extracts?
- Does this data need to live in an operational tool, or just in a dashboard or warehouse?
- Can your team realistically operate and debug the infrastructure this choice requires?
Pick the simplest pattern that satisfies the actual latency requirement — not the one that sounds most impressive in a planning doc. Batch is boring and it works for most reporting needs. Streaming is powerful and expensive, and it should be reserved for problems where speed genuinely changes the outcome. Reverse ETL isn't a pipeline pattern so much as a distribution layer, and it only works if the data feeding it is already correct. Get the requirement right first, and the architecture choice gets a lot easier.