juniorHadoop

How does MapReduce process data, and what are its limitations?

Updated Feb 20, 2026

Short answer

MapReduce runs in two phases with a shuffle between them: map transforms each input record into key-value pairs, the shuffle groups all values for a key onto one reducer, and reduce aggregates each group. It scales to petabytes and tolerates node failure by re-running tasks. Its limitations are that it writes to disk between every stage, forces every problem into two phases, and is unsuited to iterative or low-latency work.

Deep explanation

TypeScript
input splits          map                shuffle & sort            reduce
--------------   ----------------   ---------------------   ----------------
"the cat sat" -> (the,1)(cat,1)  \
                 (sat,1)          >  (cat,[1,1]) ---------> (cat, 2)
"the cat ran" -> (the,1)(cat,1)  /   (the,[1,1]) ---------> (the, 2)
                 (ran,1)             (ran,[1])  ---------> (ran, 1)
                                     (sat,[1])  ---------> (sat, 1)

The shuffle is where the cost lives. Map output is partitioned by key hash, sorted, and transferred across the network so that every value for a key reaches the same reducer. It is the only all-to-all communication in the model and usually dominates runtime.

A combiner is a mini-reducer run on the map side that pre-aggregates before the shuffle:

Java
job.setCombinerClass(SumReducer.class);   // reduces (the,1)(the,1) -> (the,2) locally

For a word count over a large corpus this can cut shuffle volume by orders of magnitude. The constraint is that the operation must be commutative and associative — sum and max qualify, average does not, which is a classic source of silently wrong results.

Fault tolerance is simple and effective. Each task is deterministic and its input is replicated, so a failed task is just re-run elsewhere. Speculative execution additionally launches duplicates of unusually slow tasks and takes whichever finishes first, which handles stragglers caused by failing hardware.

The limitations, in order of importance:

  • Disk between every stage. Map output is written to disk, and multi-job pipelines write to HDFS with three-way replication between them. An iterative algorithm running 20 passes does 20 rounds of that. Spark keeps intermediates in memory and is commonly 10–100× faster on such workloads.
  • Everything must be expressed as map-then-reduce. Joins, sorts, and multi-stage aggregations become chains of jobs with all the intermediate I/O that implies.
  • Batch only. Job startup is tens of seconds, so it cannot serve interactive queries or streaming.
  • Data skew. One key holding a disproportionate share of records sends all of it to one reducer, which becomes the whole job's critical path.

What survives is the model rather than the implementation: partition the data, move computation to the data, make tasks deterministic and re-runnable. Spark, Flink, and BigQuery all rest on those ideas while discarding the two-phase structure and the disk round trips.

Real-world example

Skew, and the standard fix, in one example:

TypeScript
Log analysis by user_id. One bot account produced 40% of all events.


Without salting:   reducer for bot_id processes 400M records
                   all other reducers process ~1M each
                   job runtime = the bot reducer's runtime


With salting:      key becomes user_id + "_" + random(0..99)
                   -> 100 partial counts for the bot, spread across reducers
                   -> a second pass sums the 100 partials back together

The extra job is cheap; the first version's runtime is set entirely by one straggling reducer. This is the same problem Spark practitioners meet as data skew, and the same fix — the framework changed but the physics did not.

Common mistakes

  • - Using a combiner for a non-associative operation such as average, which produces silently wrong results.
  • - Ignoring data skew, so one hot key makes a single reducer the critical path for the whole job.
  • - Running iterative algorithms on MapReduce, where inter-stage disk I/O dominates the actual computation.
  • - Emitting very large values from the mapper, inflating shuffle traffic that could have been pre-aggregated.
  • - Assuming reducers see keys in globally sorted order
  • sorting is guaranteed within a partition, not across them.
  • - Building a pipeline of many chained jobs without noticing each boundary costs a full replicated HDFS write and read.

Follow-up questions

  • What is a combiner and when can you use one?
  • Why is the shuffle usually the bottleneck?
  • What is speculative execution?
  • How does Spark avoid MapReduce's disk overhead?

More Hadoop interview questions

View all →