midApache Spark
How does Spark handle Memory Management?
Updated May 5, 2026
Short answer
Spark divides executor memory into Storage (for caching) and Execution (for shuffles/joins).
Deep explanation
Since Spark 1.6+, Unified Memory Management allows Storage and Execution to 'borrow' space from each other if needed.
Real-world example
Tuning a job that has many joins to have more Execution memory to avoid spilling to disk.
Common mistakes
- Setting `spark.executor.memory` too high without accounting for overhead and OS memory, leading to YARN killing the container.
Follow-up questions
- What is Off-heap memory?