PE / L6-L7 Trade-Off Analysis

Spark vs Flink vs Hadoop vs Storm

Four data processing engines, eight tables, zero hand-waving. Trade-offs grounded in operational reality, post-mortems, and what an L7 interviewer actually probes for.

As of 2026-06 · Reflects Spark 4.0, Flink 2.0 (disaggregated state via ForStDB), Hadoop 3.x, Storm 2.x

PE Verdict

For new systems in 2026: Spark for batch + ML, Flink for stateful streaming, and that covers 95% of workloads. Hadoop (MapReduce) survives only on inherited clusters and Hive-on-HDFS legacy. Storm is a maintenance burden everywhere it still runs. The choice between Spark and Flink is not "batch vs stream"; it is "latency budget vs state-size profile, and how much operational complexity you can absorb."

Best default choices

SparkDefault for batch ETL, lakehouse SQL, ML feature engineering, broad ecosystem, and micro-batch streaming FlinkDefault for low-latency, stateful, exactly-once streaming with large keyed state and precise backpressure HadoopKeep for inherited MapReduce/HDFS estates; avoid for greenfield unless legacy constraints dominate StormMaintain existing topologies carefully; prefer Flink or modern stream processors for new systems

Section 01

Trade-Offs

What each engine gives you, what it costs, and where it bites. One table per technology, minimum seven rows. Sort by clicking any column header.

Spark

in-memory DAG, micro-batch streaming

Trade-Off	What You Gain	What You Give Up	When It Bites You	PE Nuance
In-memory RDD/DataFrame execution over disk-shuffle MapReduce	10x to 100x speedup on iterative and multi-stage workloads, sub-minute jobs that took an hour on MR	RAM becomes the binding constraint, executor sizing is a constant tuning problem	A skewed join blows executor memory and the job dies after 90% completion, three times in a row	Most "Spark is slow" complaints trace to default spark.sql.shuffle.partitions of 200 left untouched on TB-scale data. AQE since 3.2 makes this less catastrophic but does not eliminate the need to think about partition sizing.
Micro-batch streaming over true event-at-a-time	Single API and runtime for batch and stream, lower mental overhead for teams already on Spark	p99 latency floor near the trigger interval, typically 100ms minimum, 1 to 5s realistic	Use case shifts to require sub-100ms reactions (fraud, real-time bidding) and the architecture cannot follow without engine swap	Continuous Processing mode exists since 2.3 and gives near-1ms latency, but supports a narrow operator set and degraded exactly-once. Treat it as experimental, not as parity with Flink.
Lineage-based fault tolerance over checkpointing	No checkpoint storage cost, recompute from source data on failure	Long lineage chains become expensive to recompute, must manually checkpoint at chain breakpoints	An ML pipeline with 40 transformations loses an executor near the end, recomputation takes longer than the original job	For Structured Streaming, the lineage model is augmented with a write-ahead log and state store. For batch jobs with deep lineage, call checkpoint() or localCheckpoint() at logical breakpoints. Most teams discover this only after a multi-hour recompute.
Catalyst optimizer with Adaptive Query Execution	Skew detection at runtime, dynamic partition coalescing, broadcast join promotion based on actual sizes	Optimizer is a black box, plans change between runs, performance regressions hard to debug	A nightly job that ran in 12 minutes for six months suddenly takes 45 minutes and no code or data changed	The fix is rarely "tune AQE"; it is partition stats drift in Hive Metastore or Iceberg. ANALYZE TABLE COMPUTE STATISTICS on a schedule is non-negotiable for cost-based optimization.
Driver-centric coordination over decentralized control	Simple mental model, single point for monitoring, deterministic task scheduling	Driver is a single point of failure, OOM on the driver kills the entire job	A developer calls .collect() on a 50GB DataFrame and brings down the production scheduler	Driver memory should be sized for coordination overhead plus broadcast variables, never for result collection. Use .show(), .write(), or limit pulls. Lint for .collect() in CI.
Cluster manager pluggability (YARN, k8s, standalone)	Run on existing Hadoop clusters or move to k8s without rewriting jobs	Each manager has its own bugs and resource model, leaky abstraction	Move from YARN to k8s and discover dynamic allocation behavior differs, autoscaling tuning starts from scratch	Spark-on-k8s reached production parity around 3.1 but still trails YARN on dynamic allocation responsiveness. For new deployments in 2026, k8s is the right bet; for stable YARN workloads, the migration ROI is rarely there.
Wide API surface (RDD, DataFrame, Dataset, SQL, MLlib, GraphX, Streaming)	One engine covers ETL, SQL analytics, ML training, graph processing	API fragmentation, MLlib lags scikit/PyTorch, GraphX abandoned, RDD vs DataFrame perf gap	Hiring engineers who know all the surfaces is hard, teams converge on DataFrame and let other APIs rot	The honest answer in 2026 is: Spark SQL + DataFrame for compute, delegate ML to PyTorch/JAX on the same data, and accept that GraphX is dead. MLlib is fine for feature engineering, weak for modern model training.
Photon (Databricks) and Comet (open source) acceleration	2x to 10x speedup on SQL workloads via vectorized C++ execution	Photon is Databricks-only and proprietary, Comet is early-stage with operator coverage gaps	Migrate workload off Databricks and lose 50% of throughput, cost model breaks	The vectorization gap is the real Databricks moat in 2026. Open-source Spark on Comet narrows it but not yet for the full operator set. Bake this into TCO models, not just license cost.

Flink

true streaming, first-class state

Trade-Off	What You Gain	What You Give Up	When It Bites You	PE Nuance
Event-at-a-time processing over micro-batching	Sub-100ms p99 end-to-end, no batch boundary artifacts in event-time aggregations	Higher per-record overhead, harder to amortize across batches	A simple stateless map+filter benchmarks slower than Spark Structured Streaming at very high throughput on bounded inputs	The latency win only materializes when state and event-time semantics matter. For stateless transforms, Spark with a tight micro-batch trigger is operationally simpler and the throughput edge is real.
Keyed state as a first-class primitive with RocksDB backend	Terabyte-scale keyed state per operator, incremental checkpoints, predictable access	RocksDB tuning becomes part of the ops surface (block cache, write buffer, compaction)	State grows past local SSD, checkpoint duration p99 exceeds checkpoint interval, recovery becomes minutes	Flink 2.0 (March 2025) solves this with disaggregated state via ForStDB on S3, but adoption is still early. For 1.x in production, set state TTL on every keyed state object, monitor RocksDB SST file count.
Chandy-Lamport checkpoints over lineage replay	Bounded recovery time independent of source backlog, exactly-once with transactional sinks	Checkpoint storage cost (typically S3), checkpoint duration grows with state size	Checkpoint duration exceeds interval, the system enters "checkpoint backpressure" and you cannot ship checkpoints fast enough to recover	Unaligned checkpoints (since 1.11) decouple checkpoint progress from operator processing, mitigating but not eliminating this. Alert on lastCheckpointDuration and on numberOfFailedCheckpoints, not just on liveness.
Event-time semantics with watermarks over processing-time	Correct aggregations even with out-of-order arrival, allowed lateness as a knob	Watermark generation is a heuristic, late events past the bound are dropped or routed to side outputs	An idle Kafka partition stalls the global watermark, downstream window aggregations stop firing while everything looks healthy	The silent failure mode here is the worst kind: aggregations stop firing without alarms because watermarks are stuck. Enable withIdleness() on every watermark strategy in production; monitor currentOutputWatermark per operator.
Savepoints for operational job migration	Schema evolution, parallelism changes, version upgrades without data loss	Schema migration is brittle, breaking changes require careful state mapping via State Processor API	Major operator graph change between savepoints, state cannot be remapped, you must reprocess from source	The State Processor API is underused. Every team that runs Flink at scale eventually needs to surgically modify keyed state during an upgrade. Build the tooling before you need it, not during the incident.
JVM-based runtime with off-heap memory management	Predictable memory usage, fewer GC pauses than naive heap-only designs	JVM still has tail-latency GC events, memory configuration is multi-layer and complex	A 99.9th-percentile GC pause on a single TaskManager triggers checkpoint timeout, cascades to job restart	Flink's managed memory model is one of the best in the JVM ecosystem but the configuration surface (framework, task, managed, network) is notoriously hard to reason about. Set up dashboards for each pool before tuning.
Two coexisting APIs: DataStream and Table/SQL	SQL for analyst-accessible streaming, DataStream for fine-grained operator control	Feature parity gaps, SQL planner can choose plans the DataStream user would never write	A SQL streaming job has unexpected state retention behavior because the planner inserts a sink materialization the user did not see	The Table API is the right default for new use cases in 2026 but inspect generated execution plans on critical paths. EXPLAIN PLAN should be a CI artifact, not a one-time check.
Cluster topology with separate JobManager and TaskManagers	Clean separation of coordination and execution, supports HA JobManager via ZK or k8s	More moving parts than Spark's driver model, JobManager HA setup is its own project	A JobManager failure during checkpoint completion leaves checkpoints in an indeterminate state, manual cleanup required	HA JobManager via k8s leader election (Flink 1.12+) is the modern path. Avoid ZK-based HA for new deployments unless you already run a battle-tested ZK ensemble.

Hadoop (MapReduce)

disk-shuffle batch, HDFS coupling

Trade-Off	What You Gain	What You Give Up	When It Bites You	PE Nuance
Disk-based shuffle over in-memory pipelining	Handles datasets larger than aggregate cluster RAM without OOM	Every stage writes intermediate output to HDFS, dominant cost in I/O time	A 5-stage pipeline becomes 5x the I/O cost of the equivalent Spark DAG, runtime dominated by shuffle reads	This was the right trade in 2008 with 8GB nodes and spinning disks. In 2026 with terabyte-class RAM and NVMe, the trade is structurally obsolete. Migration to Spark almost always wins on TCO within 18 months.
Data locality via co-located HDFS and compute	Reduced network traffic when storage and compute share nodes	Cannot scale storage and compute independently, hardware refresh is all-or-nothing	You need more compute for a quarter-end batch but the cluster is sized for steady-state storage, must over-provision permanently	Disaggregated storage (S3/GCS/ADLS) plus elastic compute is the cloud-era pattern that broke Hadoop's value prop. The locality advantage rarely materializes on cloud where network is fast enough that S3 reads are competitive with HDFS local reads.
Rigid map then reduce stage model	Simple programming model, predictable execution, easy to reason about	Multi-step computations require chaining MR jobs, each chain step pays the full shuffle cost	Iterative algorithm (k-means, PageRank) takes hundreds of MR jobs, runtime is hours instead of minutes	This is exactly why Tez and Spark won. The map-reduce primitive is too coarse. Even within the Hadoop ecosystem, Hive on Tez and Hive on Spark dominate over Hive on MR for any non-trivial query.
Free-tier replication on HDFS (3x default)	Strong durability, locality for reads, simple operator mental model	3x storage cost vs erasure-coded alternatives, write amplification on hot blocks	A petabyte-scale data lake has 3PB of disk after replication, hardware refresh budgets balloon	HDFS erasure coding (since 3.0) reduces this to 1.5x but has read-path overhead and is incompatible with many older clients. Cloud object stores quietly do EC under the hood at 11 nines for ~$0.023/GB-mo, breaking the HDFS cost argument.
YARN as the resource manager	Multi-tenant cluster, capacity scheduler with queues, integration with Kerberos	YARN is its own operational subsystem, hard to share with k8s-native workloads	The org standardizes on k8s for everything else, the Hadoop team becomes an island	Running YARN in 2026 is increasingly a hiring problem. Engineers fresh out of school have k8s muscle memory, not YARN. The institutional knowledge premium on YARN ops is real, plan for it.
NameNode-based metadata over distributed metadata	Simple single-source-of-truth model, fast metadata operations within memory limits	NameNode memory bounds total file count (~hundreds of millions of files practical ceiling)	Many small files (the "small files problem") exhaust NameNode heap, cluster becomes unstable	HDFS Federation and Router-based federation (3.x) mitigate but never fully fix this. Object stores have no such limit by design. If your workload generates lots of small files, this alone may force the migration.
On-prem deployment as the default assumption	Predictable cost at very high steady-state utilization, data sovereignty for regulated workloads	Capacity planning is a discrete-time problem, no elastic scale, hardware refresh cycles	Workload becomes seasonal or bursty, average utilization drops below 40%, cloud TCO wins by 2-3x	The few remaining justified Hadoop deployments are extreme data gravity (multi-PB datasets where egress alone justifies on-prem) or regulatory air-gap. Outside those, the on-prem assumption is the original sin.
Java-only programming model (with streaming exception)	JVM ecosystem maturity, deep integration with Hadoop ecosystem libraries	Hadoop Streaming for non-Java is awkward (stdin/stdout), no first-class Python or Scala	Data science teams want pandas-like APIs, MR forces them through PySpark anyway	The MR API is artifact-of-its-era. In 2026 the right Hadoop usage pattern is "HDFS as a storage layer accessed via Hive/Spark/Trino," not "write MR jobs." If you're writing new MR code, stop.

Storm

tuple-at-a-time, spouts and bolts

Trade-Off	What You Gain	What You Give Up	When It Bites You	PE Nuance
Tuple-at-a-time processing over batched stream model	Genuinely low per-event latency, no batch boundary artifacts	No native state, no event-time, no exactly-once without Trident	Every realistic streaming requirement (windows, aggregations, joins) forces you onto Trident, which is micro-batched anyway	This was Storm's defining insight in 2011 and is now its core limitation. Flink kept the tuple-at-a-time model and added state, watermarks, and exactly-once. Storm did not catch up.
XOR-based ack tree for at-least-once delivery	Elegant low-overhead mechanism for tracking tuple lineage, no per-tuple commit log	At-least-once only, duplicate handling pushed to application code	Aggregation logic in bolts produces double-counting under failure, correctness bugs in production discovered weeks later	The XOR ack mechanism is genuinely clever and Twitter-engineered for its scale. But the cost of "duplicates are your problem" is enormous in correctness-sensitive workloads. Most teams discover this only after a Nimbus restart causes silent overcounting.
Spout-bolt topology over dataflow operator graph	Explicit control over parallelism per component, fine-grained tuning	Programming model is lower-level than Flink/Spark, more boilerplate per operator	A complex topology becomes hundreds of lines of Java glue, refactoring is high-risk	Storm SQL exists but lags Flink SQL by years. The Java/Clojure topology API was novel in 2011, now feels primitive. Hiring engineers excited to write Storm bolts in 2026 is hard.
Nimbus as a centralized topology controller	Simple deployment model, clear single point of authority	Nimbus failure was historically catastrophic (now HA-able via ZK)	Nimbus crash mid-topology-deploy leaves the cluster in a half-deployed state, manual recovery	Storm 2.x added Nimbus HA but adoption in legacy installations is patchy. If you're running Storm at all in 2026, verify HA is configured; many installations never upgraded past 1.x.
ZooKeeper for cluster coordination	Battle-tested distributed coordination, well-understood ops	ZK is another system to operate, watches don't scale to large topologies	Large topology with thousands of executors creates ZK watch storm, coordination latency spikes	This is a fixable problem in modern ZK but the architectural choice predates Raft-based alternatives. New systems would not pick ZK as the coordination layer. KRaft (Kafka) and etcd showed the path.
Trident for exactly-once and state	Exactly-once semantics, state primitives, batched processing	Different programming model than core Storm, latency reverts to micro-batch	Team adopts Trident for exactly-once, discovers they've recreated Spark Streaming with worse tooling	Trident is the moment Storm conceded the streaming wars. Once you're on Trident, the case for Storm over Flink (or even Spark Structured Streaming) evaporates. If you need Trident, you needed Flink.
Worker-process-per-topology over containerized model	Process isolation, JVM-level resource control	Heavyweight worker startup, slow topology deployment	Rolling deploys of frequently-updated topologies take 20+ minutes, CI/CD friction	This is fixable on Heron (the Twitter-internal successor) which moved to process-per-instance. Apache Storm itself never fully closed this gap. In 2026, container-native engines (Flink on k8s) have a structural advantage here.
Maintenance-mode ecosystem trajectory	Stable API, no breaking changes, what works keeps working	Few new features, declining contributor base, security CVE response slower	A CVE in a transitive Storm dependency requires upgrading Storm itself, breaking changes in adjacent libraries	This is the existential trade-off for Storm in 2026. You're betting the system stays in maintenance for the duration of your need. For legacy maintenance: fine. For new builds: do not start here.

Section 02

Use Cases

Real workloads where each engine earned its slot. Driving Property is the single dimension that ruled out alternatives, not a feature list.

Spark

Use Case	Company / Scenario	Driving Property	Scale Dimension	Why Not Alternative
Lakehouse ETL with Delta/Iceberg tables	Netflix, Apple, most mid-to-large data orgs	Single engine for batch ETL, SQL analytics, and ACID table operations	Hundreds of PB across thousands of tables, daily ingest of 10s of TB	Flink lacks the SQL maturity for analyst workloads, Hadoop MR cannot do ACID, Trino is read-only for transformations
ML feature engineering pipelines	Most ML platforms (Uber Michelangelo, Airbnb Zipline)	Same engine produces training features (batch) and inference features (stream), no train-serve skew	1000+ feature pipelines, PB-scale historical training data	Flink can do streaming features but batch backfill is awkward, pandas does not scale to TB historical data
Periodic large-scale aggregations (daily/hourly)	Ad analytics, BI rollups, finance reconciliation	Cost-efficient batch over "real-time" when latency budget is hours	10s of billions of events per day, 100+ aggregation jobs	Streaming engines are 3-10x more expensive for the same workload when latency is not a hard requirement
Graph and iterative analytics	LinkedIn (historically), recommender systems	In-memory caching of intermediate results across iterations, 10-100x faster than MR	Trillion-edge graphs, 100s of iterations of PageRank/connected components	MapReduce reads from HDFS every iteration, unusable for graph workloads at scale
Spark Structured Streaming for moderate-latency pipelines	Teams already on Spark needing minutes-fresh data	Reuse of existing batch code, no new operational surface	100K to 1M events/sec, 30s to 5min latency tolerance	Flink requires a new ops capability, justifiable only when latency budget is sub-second
Notebook-driven exploratory analytics	Data science teams on Databricks/Jupyter	Interactive sub-second response on TB-scale data via cached DataFrames	10s to 100s of TB working sets, dozens of concurrent users	Hadoop is too slow for interactive use, Flink is operator-oriented not exploration-oriented

Flink

Use Case	Company / Scenario	Driving Property	Scale Dimension	Why Not Alternative
Real-time fraud detection	Stripe, Capital One, Alibaba	Sub-100ms decision latency with stateful per-card rules across multiple time windows	100K+ TPS, billions of cards in keyed state	Spark micro-batch p99 cannot meet decision SLA, Storm lacks the stateful primitives without Trident
Ad click aggregation with event-time correctness	Pinterest, large ad platforms	Event-time windows tolerant to 10-minute out-of-order arrival, exactly-once to billing	100K+ clicks/sec, minute-granular aggregation	Spark Structured Streaming watermark model is less expressive, Storm requires DIY watermarking
CDC-driven materialized views	Uber, Netflix, modern data platforms	Low-latency join of multiple CDC streams to keep derived tables fresh	10s of source tables joined, sub-minute freshness on derived views	Spark Structured Streaming joins are harder to express, Storm has no native stream-stream join with time semantics
Real-time metrics and observability pipelines	Lyft, Netflix Mantis, observability vendors	Sub-second p99 from event to dashboard with high-cardinality aggregation	Millions of events/sec, millions of unique series	Spark's batch boundary makes p99 unpredictable, Storm cannot do windowed aggregation efficiently
Streaming SQL on Kafka topics	Confluent Cloud customers, Alibaba Cloud streaming	SQL-first interface for analyst-friendly streaming, materialized as Kafka or Paimon tables	1000s of streaming queries, 100s of topics, mixed retention	ksqlDB has narrower SQL coverage, Spark Streaming SQL is less mature for windowed joins
Long-running stateful applications (sessionization)	Gaming companies, IoT platforms, telematics	Session windows with multi-hour gaps, keyed state per user/device with TTL	100M+ active keys, GB to TB of state per TaskManager	No other engine handles large keyed state with incremental checkpoints at this scale

Hadoop (MapReduce)

Use Case	Company / Scenario	Driving Property	Scale Dimension	Why Not Alternative
Legacy Hive-on-MR warehouse queries	Enterprise data warehouses circa 2012-2018	SQL over very large datasets with stable, well-known performance	10s to 100s of TB tables, nightly aggregation jobs	In 2026 this should be Hive-on-Tez or Spark SQL; the MR backend persists only via inertia
HDFS as a storage layer (compute via Spark/Trino)	On-prem data lakes, regulated industries	POSIX-like API, multi-tenant access control via Ranger, no egress fees	10s of PB of storage, 1000s of users	S3-class object stores have egress costs and are off-limits in air-gapped environments
Large-scale log archival processing	Compliance and audit pipelines	Cost-efficient compute on rarely-accessed data, batch latency irrelevant	Years of historical logs, occasional ad-hoc scans	Spark and Flink work too but the operational inertia for archival workloads is real, no reason to migrate
ETL jobs predating Spark adoption	Banks, telecoms, insurance carriers	Existing code, validated outputs, regulatory documentation	1000s of MR jobs, decade of accumulated business logic	Migration to Spark is technically straightforward but the validation burden and audit trail rewrite is the actual blocker
Bulk data movement and format conversion	DistCp-style cross-cluster transfers	Bandwidth-saturating parallel copy with retry semantics	PB-scale transfers across datacenters	DistCp is good enough and embedded; switching to Spark-based alternatives is rarely worth it for pure copy workloads

Storm

Use Case	Company / Scenario	Driving Property	Scale Dimension	Why Not Alternative
Legacy real-time trending and personalization	Twitter (historically), Spotify, Yahoo (legacy)	Low per-event latency for stateless or lightly-stateful enrichment	100K to 1M tuples/sec, sub-100ms processing per tuple	Predated Flink at scale; new builds in 2026 should not use Storm here
Real-time ETL pipelines (Kafka to OLAP)	Enterprises that adopted Storm 2014-2017	Stateless event reformatting and routing with at-least-once delivery	10K to 100K events/sec, simple per-tuple transforms	Kafka Streams or Flink with Table API gives the same outcome with less ops overhead
Operational telemetry pipelines	Telco network monitoring, infrastructure observability	Tuple-at-a-time processing with at-least-once for alert generation	100K events/sec, alert-on-pattern logic	Flink or even simple Kafka consumers handle this with better state and exactly-once if needed
Distributed RPC for offline scoring	Storm DRPC pattern, ML scoring service prototypes	Stream-style execution of synchronous request/response with parallelism	1000s of QPS, multi-stage scoring	Modern alternatives (Ray Serve, BentoML) handle this with first-class HTTP semantics
Maintenance of pre-existing Storm topologies	Any org with a Storm investment from 2014-2018 era	Cost of migration exceeds cost of continued operation	Per-topology basis, often 10s of long-running topologies	The only honest 2026 use case. Plan the exit, prioritize by failure rate.

Section 03

Limitations

Multi-tech matrix: rows are limitation categories, columns are engines. Cells describe how each tech is constrained on that axis. Severity badges reflect production impact.

Toggle

Limitation Category	Spark	Flink	Hadoop	Storm
Tail latency floor	Medium ~100ms minimum in Structured Streaming, 1-5s realistic at moderate throughput	Medium Single-digit ms achievable but JVM GC pauses contribute to p99 tail	Critical Minutes to hours, not designed for low latency	Medium Sub-100ms but tail spikes during topology rebalance
State size ceiling per worker	High 10s of GB per executor in Structured Streaming, RocksDB backend since 3.2 helps	Medium TB per slot with RocksDB local; 2.0 disaggregated state via ForStDB removes the local disk ceiling	Critical Stateless MR model, no built-in state primitive	Critical No native state; Trident adds it with substantial latency cost
Exactly-once at sink	Medium Exactly-once state; sink must be idempotent or transactional for end-to-end	Medium Same constraint; broader transactional sink coverage (Kafka, JDBC XA, Iceberg)	Medium Exactly-once is trivial for bounded jobs, not a streaming concern	High At-least-once by default; exactly-once requires Trident which loses latency edge
Schema evolution mid-job	Medium State store schema changes require migration job, no online evolution	High Savepoint + State Processor API workflow; brittle for non-trivial schema changes	Medium Re-run the job with new schema; bounded by job duration	High Topology redeployment, manual state migration if Trident used
Multi-language support	Medium Python/Scala/Java/R; PySpark has perf overhead via Arrow serialization	High Java/Scala first-class; PyFlink lags in feature parity and perf	High Java-first; Hadoop Streaming for other languages is stdin/stdout, awkward	High Java/Clojure first-class; multilang protocol exists but rarely used
SQL maturity	Medium Spark SQL is mature, Catalyst optimizer with AQE; gaps in streaming SQL semantics	Medium Flink SQL strong for streaming, fewer batch optimizations vs Spark	High SQL only via Hive layer; Hive-on-MR is slow vs Hive-on-Tez/Spark	Critical Storm SQL is minimal and rarely production-used
Operational complexity (ops headcount)	Medium Manageable; Databricks/EMR Serverless reduce burden substantially	High State size, checkpoint tuning, watermark monitoring all require expertise	Critical YARN + HDFS + ZK + Hive Metastore; multi-person ops team minimum	Critical Nimbus, Supervisors, ZK; declining institutional knowledge
Backpressure handling	Medium Coarse-grained via batch sizing; rate limiting on the input side	Medium Credit-based per-channel backpressure, finest-grained of the four	Medium N/A for batch jobs; not a streaming concern	High Manual spout pause/resume; backpressure was bolted on in 1.x
ML/AI integration	Medium MLlib + integrations with PyTorch/TF; weakest at modern LLM training	High FlinkML is minimal; 2.2 added ML_PREDICT/VECTOR_SEARCH for inference only	Critical Mahout is dead; no realistic ML story on MR	Critical No ML story; not designed for it
Cross-region / multi-DC story	Medium Built on cluster manager + storage layer; multi-DC is at storage level (cross-region S3)	High No native multi-region job; user designs Kafka MirrorMaker + per-region Flink	High HDFS Federation for metadata; cross-DC is DistCp + manual coordination	High Per-cluster; cross-region is application-level

Section 04

Fault Tolerance

Replication and recovery dimensions for compute engines (storage replication is the underlying FS, not the engine). RTO/RPO numbers are typical production observations, not vendor specs.

Toggle

Dimension	Spark	Flink	Hadoop	Storm
Replication model	RDD lineage recomputation; Structured Streaming uses write-ahead log + state store	Chandy-Lamport distributed snapshots to durable store (S3/HDFS)	HDFS 3x replication for data; task re-execution for compute	XOR ack tree for at-least-once; Trident layer for exactly-once via batches
Failure detection	Heartbeat to driver, executor lost after configurable timeout (default 60s)	JobManager heartbeat, TaskManager loss detected in ~30s	YARN ResourceManager heartbeats with NodeManagers, ~10s detection	Worker heartbeats to Supervisor and ZK, ~30s detection
Failover mechanism	Driver re-issues failed tasks; if driver dies, entire job dies (use cluster mode for auto-restart)	Full job restart from last checkpoint by default; region failover (1.9+) recovers only affected pipeline regions	Failed map/reduce tasks re-executed on different nodes, idempotent by design	Supervisor restarts worker; Nimbus reassigns if node lost
RTO (typical)	15-30s for executor loss; minutes for driver loss (restart from last checkpoint)	10-30s for TaskManager loss with checkpointing enabled; minutes for very large state	Minutes (task re-execution); job restart on AppMaster failure	10-30s for worker; topology continues processing on remaining workers (degraded throughput)
RPO (typical)	Zero with exactly-once Structured Streaming and idempotent sink; otherwise duplicates possible	Zero with exactly-once and transactional sink; replay from last checkpoint, no data loss	Zero for batch jobs (idempotent re-execution)	Duplicates possible (at-least-once); zero only with Trident + idempotent operations
Split-brain behavior	Driver is authoritative; no split-brain in single-driver model	JobManager HA via ZK or k8s leader election; only one active leader	NameNode HA with ZK fencing; YARN RM HA with ZK	Nimbus HA in 2.x via ZK; only one active Nimbus at a time
Blast radius of single-node failure	Affected tasks rescheduled; data lineage replayed if RDD partition lost	Without region failover: entire job restarts. With it: only affected pipeline region	Single task slot affected; HDFS 3x replication means no data loss	Affected executors restarted; in-flight tuples replayed via XOR ack
Cross-region failover	Application-level; rerun job in second region against replicated data	Application-level; active/standby clusters with Kafka MirrorMaker is the common pattern	DistCp + manual cluster cutover; cross-region HDFS is not a thing	Application-level; deploy topology in second region
Data loss scenarios	State store loss with non-replicated checkpoint location; in-flight micro-batch on dead executor without WAL	Checkpoint store loss (S3 region outage); state corruption from buggy operator code	Simultaneous loss of all 3 HDFS replicas (rare); NameNode metadata loss without HA	Nimbus + ZK loss without HA; at-least-once duplicates as the more common correctness issue

Section 05

Partitioning

For data engines, "sharding" means partitioning data across workers. The shape of the partitioning model determines parallelism, hot-key behavior, and rebalancing cost.

Toggle

Dimension	Spark	Flink	Hadoop	Storm
Partitioning model	Hash partitioning by default on shuffle; range partitioning available; custom partitioner supported	Hash partitioning on keyBy; rebalance, rescale, broadcast, shuffle as alternatives	Hash partitioning on reducer (HashPartitioner default); custom partitioner via API	Stream grouping: shuffle, fields (hash), all, global, direct, custom
Partition key constraints	Any column or expression; non-deterministic keys cause shuffle correctness issues	Any KeySelector returning hashable key; key extracted from record at runtime	Key emitted from mapper; partitioner receives key+value	Tuple field used in fields grouping; arbitrary logic in custom grouping
Rebalancing mechanism	AQE coalesces partitions at runtime; manual repartition() for explicit reshuffling	Key groups (default 128) assigned to slots; rescaling reassigns key groups across new parallelism	Static at job submission; no rebalancing within a job	Topology rebalance command; reassigns executors to workers, restarts affected components
Rebalancing cost / impact	Free at stage boundaries via AQE; explicit repartition triggers full shuffle (expensive)	Requires savepoint + restart; downtime in seconds to minutes; Flink 2.0 disaggregated state reduces this to seconds at any state size	Not applicable mid-job; resubmit with new partitioner	Topology pauses during rebalance, in-flight tuples may be replayed; seconds to minutes
Hot-shard / hot-key behavior	AQE skew join handling splits skewed partitions; pre-AQE, salting was the manual fix	Single key always lands on single slot; mitigation is pre-aggregation, key salting, or two-stage aggregation	Skewed reducer becomes the long-tail straggler; speculative execution does not help	Bolt instance for hot key becomes bottleneck; manual rekeying or pre-aggregation
Maximum partitions (practical)	10s of thousands per stage; driver memory bounds task tracking	Parallelism limited by available slots; key groups default 128, configurable up to thousands	10s of thousands of reducers practical; AppMaster bottleneck at very high counts	Thousands of executors per topology; Nimbus + ZK bottleneck beyond that
Rescaling without job restart	Dynamic allocation adds/removes executors; partition count fixed per stage	Reactive mode (1.13+) or adaptive scheduler (1.15+) rescale within job; otherwise savepoint + restart	No, MR jobs have fixed parallelism at submission	Rebalance command resizes topology without redeploy
Cross-partition operations	Joins trigger shuffle; broadcast join avoids it for small tables (under ~10MB)	keyBy + co-process for stream-stream joins; broadcast state for slow-changing dimensions	Reduce-side joins via shuffle; map-side join for sorted bucketed inputs	Join requires custom bolt with state, error-prone; Trident has join primitives

Section 06

Replication

Engines do not replicate user data, they replicate state and recovery artifacts. The underlying storage layer (HDFS, S3, RocksDB on EBS) provides data durability. This table covers state and checkpoint replication.

Toggle

Dimension	Spark	Flink	Hadoop	Storm
Replication topology	N/A in-process; RDD persist with REPLICATE_2 stores cache on 2 executors for failover speed	Checkpoint stored once in durable FS (replication is FS-level); local task state not replicated	HDFS 3x replication for data; computation is replicated only via speculative execution	Tuple lineage tracked via XOR ack; topology metadata in ZK is replicated by ZK quorum
Sync vs async	Checkpoint writes are synchronous within micro-batch; cache replication is async	Aligned checkpoints are synchronous barrier-based; unaligned (1.11+) allow async progress	HDFS writes sync to first replica, pipeline async to remaining	Ack tree is async; Trident commit is synchronous per batch
Replication factor	Configurable per cached RDD (NONE, 1, 2); checkpoint inherits FS replication factor	Single checkpoint copy in FS; FS provides durability (S3 11 nines, HDFS 3x)	HDFS default 3x; configurable per file via dfs.replication	N/A engine-level; ZK ensemble size (3 or 5) provides metadata replication
Consistency level options	Exactly-once via checkpoint+WAL coordination, at-least-once via simpler config	EXACTLY_ONCE or AT_LEAST_ONCE per checkpoint config; choice impacts barrier alignment	HDFS strong consistency for reads; eventual within replication pipeline	At-most-once (disable acks), at-least-once (default), exactly-once (Trident)
Replication lag (typical)	Checkpoint interval determines lag (default 10s); state freshness bounded by it	Checkpoint interval is the recovery lag bound (typically 10s to 5min)	HDFS pipeline write sub-second for small blocks, seconds for larger	Sub-second for tuple ack; Trident batch interval is the recovery lag (~seconds)
Conflict resolution	N/A; single writer per partition, last-write-wins on retry	N/A; single writer per key group, deterministic replay from checkpoint	N/A; idempotent task re-execution, no conflict by design	At-least-once means application must be idempotent or handle duplicates
Cross-region replication	Use cross-region S3 for checkpoints (high cost); typically active/standby clusters	Cross-region S3 for checkpoints; common pattern is active-active with Kafka MirrorMaker	DistCp scheduled jobs; cross-region HDFS not native	Per-cluster; application-level cross-region patterns
Behavior during partition (network split)	Driver-side: lost executor treated as failed; isolated partition's writes are abandoned	JobManager loses TaskManager heartbeat, declares failure, restarts from checkpoint	YARN times out lost containers, reassigns tasks; HDFS quorum-based for writes	Worker isolated from Nimbus/ZK becomes orphan; Nimbus assigns replacement

Section 07

Better Usage Patterns

Where most teams stumble vs where PE-level teams land. These are the patterns that show up in code review as the difference between "works" and "works at scale."

Spark

Pattern	What Most Teams Do Wrong	The Better Way	Why It Matters
Partition sizing	Leave spark.sql.shuffle.partitions at default 200; observe 30GB shuffles dying on executors	Enable AQE (default in 3.2+), set initial partitions high (1000-5000), let AQE coalesce post-shuffle	The single most common cause of "Spark is slow" tickets. AQE removes the manual tuning need but only if enabled and given a high starting count.
Broadcast join hints	Rely on Catalyst to detect small-side tables; fail when stats are stale or data grows	Explicit broadcast() hint for dimension tables under 100MB; reject if stats indicate otherwise	One stale stat turning a broadcast join into a shuffle join can 10x runtime. Explicit hints are documentation that stat changes do not silently break performance.
Caching strategy	Cache everything with default MEMORY_AND_DISK; thrash memory and spill constantly	Cache only what is reused 3+ times, choose MEMORY_ONLY_SER or DISK_ONLY based on access pattern	Caching is not free. Wrong cache choice creates GC pressure that slows everything, not just the cached job.
Schema management with Iceberg/Delta	Read Parquet files directly without a table format; lose schema evolution and ACID	Always go through Iceberg or Delta tables; never read raw Parquet in production pipelines	Table formats solve schema evolution, time travel, and concurrent writers. The migration cost from raw Parquet is real but pays back within months at any non-trivial scale.
Structured Streaming checkpoint location	Put checkpoints in the same path as output; deleting old outputs breaks streaming	Dedicated checkpoint location in object storage, separate lifecycle from data outputs	Conflated checkpoint and output paths is a recipe for accidental state loss. Treat the checkpoint dir as a database, not a temp folder.
Driver vs executor responsibility	.collect() or .toPandas() on large DataFrames; driver OOM kills the job	Aggregate to bounded size before collecting; use .foreachPartition() for distributed side effects	Driver memory is for coordination, not data. Any "let me just look at this" call that downloads a multi-GB DataFrame is a production incident waiting to happen.

Flink

Pattern	What Most Teams Do Wrong	The Better Way	Why It Matters
State TTL on every keyed state	Never set TTL; state grows unboundedly until OOM or checkpoint timeout in production	StateTtlConfig on every keyed value/list/map state; cleanup in background or on read	This is the number one cause of Flink production incidents. Sessions, user contexts, dedup sets all need TTL. The default is "forever" and forever is a long time.
Watermark idle detection	Configure watermark strategy without withIdleness(); single quiet partition stalls aggregations globally	Always set withIdleness(Duration.ofMinutes(N)); pick N based on slowest expected partition	An empty Kafka partition with no events looks healthy and silently breaks event-time windows. This is the textbook silent-trust-degradation failure mode.
Checkpoint interval and timeout	Use defaults (no checkpoint timeout, 10s interval); long checkpoints stack up on slow state backends	Set explicit timeout (1-5 min based on state size); set min-pause-between-checkpoints to prevent stacking	Without timeout, a slow checkpoint blocks the next one; without min-pause, you can checkpoint continuously and never make progress on actual work.
Savepoint discipline	Take savepoints only when needed for upgrades; lose state to a bug-induced restart	Scheduled savepoints (daily or weekly) to durable storage; named, versioned, retained per policy	Savepoints are insurance. The first time you need one and don't have it, you understand why this should be automated like database backups.
State backend choice	Default to HashMapStateBackend; hit JVM heap limits in production	RocksDB backend (or ForSt in 2.0) for any state larger than 1GB; configure block cache appropriately	HashMap is fine for stateless or tiny state. Real production streaming state is RocksDB. Switching mid-flight requires savepoint migration, so choose early.
Operator chaining and parallelism	Set high parallelism uniformly; pay shuffle cost for stages that don't need parallelism	Set parallelism per operator based on actual work; use .disableChaining() selectively for debugging only	Parallelism is not free. Each parallel slot has GC, network, and coordination overhead. Skewed parallelism profiles outperform uniform high parallelism for most pipelines.

Hadoop (MapReduce)

Pattern	What Most Teams Do Wrong	The Better Way	Why It Matters
Combiner usage	Skip combiner; let all map output traverse the shuffle to reducers	Implement combiner for associative+commutative reductions; reduce shuffle volume 10-100x	The combiner is MR's biggest free performance win. Most legacy MR code does not use it because it was treated as advanced. It is not.
Small files problem	Write millions of small output files; NameNode heap exhaustion and slow reads	Use CombineFileInputFormat on read; concatenate outputs via Hive/Spark; SequenceFile or Parquet containers	NameNode memory is finite (~150 bytes per file). A pipeline that generates 10M small files takes 1.5GB of NameNode heap by itself. This is how clusters die slowly.
Custom partitioner for known skew	Use HashPartitioner with skewed key; one reducer takes 80% of work	Custom partitioner with key-aware distribution; salt highly skewed keys	A skewed reducer is a job that "finishes" except for that one task running for hours. The pattern of pre-aggregating skewed keys is older than Spark and still works.
Speculative execution settings	Leave defaults on; both copies of a task race, wasting cluster capacity	Disable for jobs with side effects; keep enabled for pure transformations	Speculative execution helps stragglers but doubles I/O for everyone. Know what your jobs do before leaving it on.
HDFS block size for read-heavy workloads	Use 128MB default for all data; mismatch with workload access pattern	256MB or 512MB blocks for sequential analytical scans; 64MB for many-small-file access	Block size determines parallelism and seek overhead. Default is a compromise; analytical workloads benefit from larger blocks, ad-hoc reads from smaller.
Migrating off MR to Tez or Spark	Treat MR as a permanent fixture; accept 10x slowdowns vs alternatives	Migrate Hive workloads to Hive-on-Tez or Hive-on-Spark; net-new code in Spark only	This is the most consequential pattern. New MR code in 2026 should be approved only with explicit justification. Hive-on-MR alone often runs 5-10x faster on Tez with zero code change.

Storm

Pattern	What Most Teams Do Wrong	The Better Way	Why It Matters
Acker bolt count	Leave acker count at 1; acker becomes bottleneck at high tuple rates	Set ackers proportional to worker count (1 per worker as starting point)	The acker is the at-least-once mechanism. One acker for hundreds of bolts is a guaranteed bottleneck. Documented but routinely missed.
Stream grouping choice	Use shuffleGrouping for everything; pay random distribution cost when fields grouping would work	fieldsGrouping for stateful bolts (locality), shuffleGrouping for stateless balance	Stateful bolts with shuffleGrouping forces cross-bolt state coordination. Right grouping is local affinity. Wrong grouping is correctness or perf bug.
Topology rebalancing for capacity	Redeploy topology to change parallelism; minutes of downtime	Use storm rebalance command; reassigns executors without full redeploy	Rebalance is not zero-downtime but is faster than redeploy. Most teams discover this only after the third capacity-driven redeploy.
Tuple timeout configuration	Leave at 30s default; slow downstream bolt causes replay storms	Tune timeout to match expected processing time + p99 margin; lower if at-most-once acceptable	Tuple replay on timeout is the most common cause of Storm "duplicates." Tighten or loosen timeout based on actual processing distribution, not the default.
Backpressure tuning	Disable backpressure (1.x default off); spouts overrun bolts, OOM workers	Enable backpressure with appropriate high/low water marks for receive queues	Storm without backpressure under sustained load is a workout in tuple loss and OOM. The defaults predate backpressure; turn it on explicitly.
Migration to Flink or Heron	Stay on Storm indefinitely; ecosystem rot, declining support	Plan migration to Flink (or Heron for Storm-API compat); prioritize topologies by failure rate	Storm is in maintenance, not death, but the trajectory is clear. Plan the exit on your terms, not on a critical CVE that forces it.

Section 08

Advanced / Next-Gen Alternatives

Successors, adjacent technologies, and architectural patterns that change the decision. Maturity reflects production-readiness for serious workloads, not GitHub stars.

Spark

Successor / Alternative	What It Improves	Maturity	Migration Cost	When To Consider
Photon (Databricks) / Comet (Apache, open source)	Vectorized C++ execution, 2-10x speedup on SQL workloads	Photon: Prod Comet: Emerging	Low (Photon is transparent on Databricks; Comet plugs into open-source Spark)	SQL-heavy workloads where compute cost is the top line item; LLM/AI workloads where memory bandwidth matters
DuckDB / Polars	Single-node analytical queries on TB-scale data without cluster overhead	Prod	Medium (different APIs, but compatible with Parquet/Iceberg)	Workloads that fit on a large single node (256GB+ RAM); save 10x on cluster overhead
Trino / Athena / BigQuery	SQL-only interactive analytics over object storage with autoscaling	Prod	Medium (SQL-compatible but ETL needs to run somewhere)	BI workloads and ad-hoc analytics; pair with Spark for ETL, separate read and write engines
Ray (Anyscale)	Distributed Python-native compute, especially strong for ML training and serving	Prod	Medium-High (different programming model; Ray Data narrows the gap for data workloads)	ML training pipelines, hyperparameter tuning, reinforcement learning; deep Python integration

Flink

Successor / Alternative	What It Improves	Maturity	Migration Cost	When To Consider
Flink 2.0 with disaggregated state (ForStDB)	Removes local disk ceiling on state size; checkpoints in seconds at any scale; faster rescaling	Emerging	Low (in-place upgrade; new state backend opt-in)	Workloads with state larger than local SSD; cloud-native deployments where compute is ephemeral
Kafka Streams	Library-based streaming, no separate cluster; tight Kafka integration	Prod	Medium (different model, but for Kafka-native pipelines, simpler ops)	Microservices needing streaming logic with deployment alongside business services; avoid for cross-cluster jobs
Materialize / RisingWave	Streaming SQL with materialized views over Postgres-compatible interface	Emerging (Materialize Prod-ready 2023+)	Medium-High (different operational model, smaller ecosystem)	Streaming use cases where SQL is the entire programming model; analyst-driven streaming
Confluent Cloud for Apache Flink	Fully-managed Flink with Kafka-native auth, scaling, and billing	Prod	Low (managed; primary trade-off is vendor lock)	Teams that don't want to operate Flink themselves; standard Kafka + Flink stack

Hadoop (MapReduce)

Successor / Alternative	What It Improves	Maturity	Migration Cost	When To Consider
Spark (replacing MR as compute engine)	10-100x faster on multi-stage jobs, modern APIs, active development	Prod	Medium (most MR can be rewritten in Spark; Hive-on-MR to Hive-on-Spark often trivial)	Anywhere you still run MR; the migration is the default 2026 answer
Apache Tez (DAG-based MR replacement)	Multi-stage execution without intermediate HDFS writes; 5-10x faster Hive queries	Mature but waning	Low (drop-in for Hive)	If you're committed to Hadoop stack but need to speed up Hive without going to Spark
Object storage + Iceberg/Delta/Hudi (replacing HDFS)	Decoupled storage and compute, elastic scaling, lower TCO, ACID over object stores	Prod	High (data migration; query engine retooling)	Cloud migration; multi-engine access (Spark + Trino + Flink) on same data
Kubernetes (replacing YARN)	Unified scheduler across all workloads, better ecosystem alignment	Prod	Medium-High (rebuild deployment, monitoring, security model)	When YARN is the last Hadoop component holding back full k8s adoption

Storm

Successor / Alternative	What It Improves	Maturity	Migration Cost	When To Consider
Apache Flink	True streaming with state, exactly-once, event-time, watermarks, active ecosystem	Prod	High (rewrite required; topology to DataStream/Table API)	Any new build, and most legacy Storm where state/correctness matters
Apache Heron (Twitter's Storm successor)	Storm topology API compatibility, better resource isolation, container-native	Maintenance / dormant	Low (API-compatible drop-in)	Mostly historical; consider only if you must keep Storm-style topology code and Flink migration is blocked
Kafka Streams	Streaming as a library inside Kafka clients, no separate cluster, Kafka-native	Prod	Medium (rewrite; different programming model)	Kafka-centric pipelines; service-embedded streaming without operating a Flink cluster
Plain Kafka consumers with custom logic	Simplest possible streaming, no framework overhead	Prod	Low to Medium (depends on logic complexity)	Stateless or trivially-stateful pipelines; the "do we even need a framework?" question