Zaharia et al (2012) - Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Key Insights
- Apache Spark introduces RDDs (Resilient Distributed Datasets) enabling in-memory computation 10-100x faster than MapReduce for iterative algorithms, with lineage-based fault recovery.
Edit on GitHub — registry.json
Background
By 2012, MapReduce had made petabyte-scale processing routine, but its disk-centric execution model punished the workloads that mattered next: iterative algorithms and interactive exploration. Machine learning, graph analysis, and iterative data mining repeatedly reuse intermediate results across stages, and each pass forced a write to stable storage. Zaharia and the Berkeley AMPLab designed Spark around a single question: what if the fault-tolerant abstraction itself lived in memory?
The RDD Abstraction
The paper introduces Resilient Distributed Datasets (RDDs): immutable, partitioned collections of records that support coarse-grained transformations such as map, filter, and join, plus actions like count and collect. Instead of replicating data for fault tolerance, an RDD records its own lineage — the dependency graph of transformations that produced it — so any lost partition can be recomputed from its parents. Lineage replaces replication as the recovery mechanism, which is what makes in-memory caching practical.
Deep Dive
Users can mark RDDs for persistence with persist(), choosing between memory-only, memory-and-disk, or serialized layouts. Lazy evaluation defers computation until an action is invoked, letting the scheduler pipeline transformations and recover only the partitions actually lost. The paper reports up to 20x speedups over Hadoop on iterative workloads such as logistic regression and k-means clustering, with recovery an order of magnitude faster than replication-based approaches.
Why It Matters
RDDs were the fault-tolerance core that later became Spark SQL, DataFrames, and Structured Streaming. The lineage-based recovery idea now underpins every modern lakehouse engine and structured streaming system, and remains the mental model for debugging recomputation and caching behavior in production Spark clusters.
Key Takeaways
- Lineage beats replication for recomputable transformations — recovery cost trades off against memory pressure.
- Cache deliberately: memory-only persistence avoids disk I/O but drops partitions under pressure; plan for spill.
- Coarse-grained transformations are the price of lineage: RDDs support batch updates well, point updates poorly.